1
|
Colas JT, O’Doherty JP, Grafton ST. Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts. PLoS Comput Biol 2024; 20:e1011950. [PMID: 38552190 PMCID: PMC10980507 DOI: 10.1371/journal.pcbi.1011950] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Accepted: 02/26/2024] [Indexed: 04/01/2024] Open
Abstract
Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.
Collapse
Affiliation(s)
- Jaron T. Colas
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, California, United States of America
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, California, United States of America
| | - John P. O’Doherty
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, California, United States of America
| | - Scott T. Grafton
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, California, United States of America
| |
Collapse
|
2
|
Li JJ, Shi C, Li L, Collins AGE. Dynamic noise estimation: A generalized method for modeling noise fluctuations in decision-making. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.06.19.545524. [PMID: 38328176 PMCID: PMC10849494 DOI: 10.1101/2023.06.19.545524] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/09/2024]
Abstract
Computational cognitive modeling is an important tool for understanding the processes supporting human and animal decision-making. Choice data in decision-making tasks are inherently noisy, and separating noise from signal can improve the quality of computational modeling. Common approaches to model decision noise often assume constant levels of noise or exploration throughout learning (e.g., the ϵ -softmax policy). However, this assumption is not guaranteed to hold - for example, a subject might disengage and lapse into an inattentive phase for a series of trials in the middle of otherwise low-noise performance. Here, we introduce a new, computationally inexpensive method to dynamically infer the levels of noise in choice behavior, under a model assumption that agents can transition between two discrete latent states (e.g., fully engaged and random). Using simulations, we show that modeling noise levels dynamically instead of statically can substantially improve model fit and parameter estimation, especially in the presence of long periods of noisy behavior, such as prolonged attentional lapses. We further demonstrate the empirical benefits of dynamic noise estimation at the individual and group levels by validating it on four published datasets featuring diverse populations, tasks, and models. Based on the theoretical and empirical evaluation of the method reported in the current work, we expect that dynamic noise estimation will improve modeling in many decision-making paradigms over the static noise estimation method currently used in the modeling literature, while keeping additional model complexity and assumptions minimal.
Collapse
Affiliation(s)
- Jing-Jing Li
- Helen Wills Neuroscience Institute, University of California, Berkeley, 175 Li Ka Shing Center, Berkeley, 94720, CA, United States
| | - Chengchun Shi
- Department of Statistics, London School of Economics and Political Science, 69 Aldwych, London, WC2B 4RR, United Kingdom
| | - Lexin Li
- Helen Wills Neuroscience Institute, University of California, Berkeley, 175 Li Ka Shing Center, Berkeley, 94720, CA, United States
- Department of Biostatistics and Epidemiology, University of California, Berkeley, 2121 Berkeley Way, Berkeley, 94720, CA, United States
| | - Anne G E Collins
- Helen Wills Neuroscience Institute, University of California, Berkeley, 175 Li Ka Shing Center, Berkeley, 94720, CA, United States
- Department of Psychology, University of California, Berkeley, Berkeley, 94720, CA, United States
| |
Collapse
|
3
|
Howlett JR, Paulus MP. Out of control: computational dynamic control dysfunction in stress- and anxiety-related disorders. DISCOVER MENTAL HEALTH 2024; 4:5. [PMID: 38236488 PMCID: PMC10796870 DOI: 10.1007/s44192-023-00058-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 12/28/2023] [Indexed: 01/19/2024]
Abstract
Control theory, which has played a central role in technological progress over the last 150 years, has also yielded critical insights into biology and neuroscience. Recently, there has been a surging interest in integrating control theory with computational psychiatry. Here, we review the state of the field of using control theory approaches in computational psychiatry and show that recent research has mapped a neural control circuit consisting of frontal cortex, parietal cortex, and the cerebellum. This basic feedback control circuit is modulated by estimates of reward and cost via the basal ganglia as well as by arousal states coordinated by the insula, dorsal anterior cingulate cortex, amygdala, and locus coeruleus. One major approach within the broader field of control theory, known as proportion-integral-derivative (PID) control, has shown promise as a model of human behavior which enables precise and reliable estimates of underlying control parameters at the individual level. These control parameters correlate with self-reported fear and with both structural and functional variation in affect-related brain regions. This suggests that dysfunctional engagement of stress and arousal systems may suboptimally modulate parameters of domain-general goal-directed control algorithms, impairing performance in complex tasks involving movement, cognition, and affect. Future directions include clarifying the causal role of control deficits in stress- and anxiety-related disorders and developing clinically useful tools based on insights from control theory.
Collapse
Affiliation(s)
- Jonathon R Howlett
- VA San Diego Healthcare System, 3350 La Jolla Village Dr, San Diego, CA, 92161, USA.
- Department of Psychiatry, University of California San Diego, La Jolla, CA, USA.
| | | |
Collapse
|
4
|
Sato Y, Sakai Y, Hirata S. State-transition-free reinforcement learning in chimpanzees (Pan troglodytes). Learn Behav 2023; 51:413-427. [PMID: 37369920 DOI: 10.3758/s13420-023-00591-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/07/2023] [Indexed: 06/29/2023]
Abstract
The outcome of an action often occurs after a delay. One solution for learning appropriate actions from delayed outcomes is to rely on a chain of state transitions. Another solution, which does not rest on state transitions, is to use an eligibility trace (ET) that directly bridges a current outcome and multiple past actions via transient memories. Previous studies revealed that humans (Homo sapiens) learned appropriate actions in a behavioral task in which solutions based on the ET were effective but transition-based solutions were ineffective. This suggests that ET may be used in human learning systems. However, no studies have examined nonhuman animals with an equivalent behavioral task. We designed a task for nonhuman animals following a previous human study. In each trial, participants chose one of two stimuli that were randomly selected from three stimulus types: a stimulus associated with a food reward delivered immediately, a stimulus associated with a reward delivered after a few trials, and a stimulus associated with no reward. The presented stimuli did not vary according to the participants' choices. To maximize the total reward, participants had to learn the value of the stimulus associated with a delayed reward. Five chimpanzees (Pan troglodytes) performed the task using a touchscreen. Two chimpanzees were able to learn successfully, indicating that learning mechanisms that do not depend on state transitions were involved in the learning processes. The current study extends previous ET research by proposing a behavioral task and providing empirical data from chimpanzees.
Collapse
Grants
- 16H06283 Ministry of Education, Culture, Sports, Science, Japan Society for the Promotion of Science
- 18H05524 Ministry of Education, Culture, Sports, Science, Japan Society for the Promotion of Science
- 19J22889 Ministry of Education, Culture, Sports, Science, Japan Society for the Promotion of Science
- 26245069 Ministry of Education, Culture, Sports, Science, Japan Society for the Promotion of Science
- U04 Program for Leading Graduate Schools
Collapse
Affiliation(s)
- Yutaro Sato
- Wildlife Research Center, Kyoto University, Kyoto, Japan.
- University Administration Office, Headquarters for Management Strategy, Niigata University, Niigata, Japan.
| | - Yutaka Sakai
- Brain Science Institute, Tamagawa University, Tokyo, Japan
| | - Satoshi Hirata
- Wildlife Research Center, Kyoto University, Kyoto, Japan
| |
Collapse
|
5
|
Park H, Doh H, Lee E, Park H, Ahn WY. The neurocognitive role of working memory load when Pavlovian motivational control affects instrumental learning. PLoS Comput Biol 2023; 19:e1011692. [PMID: 38064498 PMCID: PMC10732416 DOI: 10.1371/journal.pcbi.1011692] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Revised: 12/20/2023] [Accepted: 11/15/2023] [Indexed: 12/21/2023] Open
Abstract
Research suggests that a fast, capacity-limited working memory (WM) system and a slow, incremental reinforcement learning (RL) system jointly contribute to instrumental learning. Thus, situations that strain WM resources alter instrumental learning: under WM loads, learning becomes slow and incremental, the reliance on computationally efficient learning increases, and action selection becomes more random. It is also suggested that Pavlovian learning influences people's behavior during instrumental learning by providing hard-wired instinctive responses including approach to reward predictors and avoidance of punishment predictors. However, it remains unknown how constraints on WM resources affect instrumental learning under Pavlovian influence. Thus, we conducted a functional magnetic resonance imaging (fMRI) study (N = 49) in which participants completed an instrumental learning task with Pavlovian-instrumental conflict (the orthogonalized go/no-go task) both with and without extra WM load. Behavioral and computational modeling analyses revealed that WM load reduced the learning rate and increased random choice, without affecting Pavlovian bias. Model-based fMRI analysis revealed that WM load strengthened RPE signaling in the striatum. Moreover, under WM load, the striatum showed weakened connectivity with the ventromedial and dorsolateral prefrontal cortex when computing reward expectations. These results suggest that the limitation of cognitive resources by WM load promotes slow and incremental learning through the weakened cooperation between WM and RL; such limitation also makes action selection more random, but it does not directly affect the balance between instrumental and Pavlovian systems.
Collapse
Affiliation(s)
- Heesun Park
- Department of Psychology, Seoul National University, Seoul, Korea
| | - Hoyoung Doh
- Department of Psychology, Seoul National University, Seoul, Korea
| | - Eunhwi Lee
- Department of Psychology, Seoul National University, Seoul, Korea
| | - Harhim Park
- Department of Psychology, Seoul National University, Seoul, Korea
| | - Woo-Young Ahn
- Department of Psychology, Seoul National University, Seoul, Korea
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, Korea
| |
Collapse
|
6
|
Topel S, Ma I, Sleutels J, van Steenbergen H, de Bruijn ERA, van Duijvenvoorde ACK. Expecting the unexpected: a review of learning under uncertainty across development. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2023:10.3758/s13415-023-01098-0. [PMID: 37237092 PMCID: PMC10390612 DOI: 10.3758/s13415-023-01098-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 03/28/2023] [Indexed: 05/28/2023]
Abstract
Many of our decisions take place under uncertainty. To successfully navigate the environment, individuals need to estimate the degree of uncertainty and adapt their behaviors accordingly by learning from experiences. However, uncertainty is a broad construct and distinct types of uncertainty may differentially influence our learning. We provide a semi-systematic review to illustrate cognitive and neurobiological processes involved in learning under two types of uncertainty: learning in environments with stochastic outcomes, and with volatile outcomes. We specifically reviewed studies (N = 26 studies) that included an adolescent population, because adolescence is a period in life characterized by heightened exploration and learning, as well as heightened uncertainty due to experiencing many new, often social, environments. Until now, reviews have not comprehensively compared learning under distinct types of uncertainties in this age range. Our main findings show that although the overall developmental patterns were mixed, most studies indicate that learning from stochastic outcomes, as indicated by increased accuracy in performance, improved with age. We also found that adolescents tended to have an advantage compared with adults and children when learning from volatile outcomes. We discuss potential mechanisms explaining these age-related differences and conclude by outlining future research directions.
Collapse
Affiliation(s)
- Selin Topel
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands.
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands.
| | - Ili Ma
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Jan Sleutels
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden University, Institute for Philosophy, Leiden, The Netherlands
| | - Henk van Steenbergen
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Ellen R A de Bruijn
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Anna C K van Duijvenvoorde
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| |
Collapse
|
7
|
Self-judgment dissected: A computational modeling analysis of self-referential processing and its relationship to trait mindfulness facets and depression symptoms. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2023; 23:171-189. [PMID: 36168080 PMCID: PMC9931629 DOI: 10.3758/s13415-022-01033-9] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 08/29/2022] [Indexed: 11/08/2022]
Abstract
Cognitive theories of depression, and mindfulness theories of well-being, converge on the notion that self-judgment plays a critical role in mental health. However, these theories have rarely been tested via tasks and computational modeling analyses that can disentangle the information processes operative in self-judgments. We applied a drift-diffusion computational model to the self-referential encoding task (SRET) collected before and after an 8-week mindfulness intervention (n = 96). A drift-rate regression parameter representing positive-relative to negative-self-referential judgment strength positively related to mindful awareness and inversely related to depression, both at baseline and over time; however, this parameter did not significantly relate to the interaction between mindful awareness and nonjudgmentalness. At the level of individual depression symptoms, at baseline, a spectrum of symptoms (inversely) correlated with the drift-rate regression parameter, suggesting that many distinct depression symptoms relate to valenced self-judgment between subjects. By contrast, over the intervention, changes in only a smaller subset of anhedonia-related depression symptoms showed substantial relationships with this parameter. Both behavioral and model-derived measures showed modest split-half and test-retest correlations. Results support cognitive theories that implicate self-judgment in depression and mindfulness theories, which imply that mindful awareness should lead to more positive self-views.
Collapse
|
8
|
Abstract
In reinforcement learning (RL) experiments, participants learn to make rewarding choices in response to different stimuli; RL models use outcomes to estimate stimulus-response values that change incrementally. RL models consider any response type indiscriminately, ranging from more concretely defined motor choices (pressing a key with the index finger), to more general choices that can be executed in a number of ways (selecting dinner at the restaurant). However, does the learning process vary as a function of the choice type? In Experiment 1, we show that it does: Participants were slower and less accurate in learning correct choices of a general format compared with learning more concrete motor actions. Using computational modeling, we show that two mechanisms contribute to this. First, there was evidence of irrelevant credit assignment: The values of motor actions interfered with the values of other choice dimensions, resulting in more incorrect choices when the correct response was not defined by a single motor action; second, information integration for relevant general choices was slower. In Experiment 2, we replicated and further extended the findings from Experiment 1 by showing that slowed learning was attributable to weaker working memory use, rather than slowed RL. In both experiments, we ruled out the explanation that the difference in performance between two condition types was driven by difficulty/different levels of complexity. We conclude that defining a more abstract choice space used by multiple learning systems for credit assignment recruits executive resources, limiting how much such processes then contribute to fast learning.
Collapse
Affiliation(s)
| | - Amy Zou
- University of California, Berkeley
| | - Anne G E Collins
- University of California, Berkeley
- Helen Wills Neuroscience Institute Berkeley, CA
| |
Collapse
|
9
|
Geana A, Barch DM, Gold JM, Carter CS, MacDonald AW, Ragland JD, Silverstein SM, Frank MJ. Using Computational Modeling to Capture Schizophrenia-Specific Reinforcement Learning Differences and Their Implications on Patient Classification. BIOLOGICAL PSYCHIATRY. COGNITIVE NEUROSCIENCE AND NEUROIMAGING 2022; 7:1035-1046. [PMID: 33878489 PMCID: PMC9272137 DOI: 10.1016/j.bpsc.2021.03.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/25/2020] [Revised: 03/24/2021] [Accepted: 03/24/2021] [Indexed: 10/21/2022]
Abstract
BACKGROUND Psychiatric diagnosis and treatment have historically taken a symptom-based approach, with less attention on identifying underlying symptom-producing mechanisms. Recent efforts have illuminated the extent to which different underlying circuitry can produce phenotypically similar symptomatology (e.g., psychosis in bipolar disorder vs. schizophrenia). Computational modeling makes it possible to identify and mathematically differentiate behaviorally unobservable, specific reinforcement learning differences in patients with schizophrenia versus other disorders, likely owing to a higher reliance on prediction error-driven learning associated with basal ganglia and underreliance on explicit value representations associated with orbitofrontal cortex. METHODS We used a well-established probabilistic reinforcement learning task to replicate those findings in individuals with schizophrenia both on (n = 120) and off (n = 44) antipsychotic medications and included a patient comparison group of bipolar patients with psychosis (n = 60) and healthy control subjects (n = 72). RESULTS Using accuracy, there was a main effect of group (F3,279 = 7.87, p < .001), such that all patient groups were less accurate than control subjects. Using computationally derived parameters, both medicated and unmediated individuals with schizophrenia, but not patients with bipolar disorder, demonstrated a reduced mixing parameter (F3,295 = 13.91, p < .001), indicating less dependence on learning explicit value representations as well as greater learning decay between training and test (F1,289 = 12.81, p < .001). Unmedicated patients with schizophrenia also showed greater decision noise (F3,295 = 2.67, p = .04). CONCLUSIONS Both medicated and unmedicated patients showed overreliance on prediction error-driven learning as well as significantly higher noise and value-related memory decay, compared with the healthy control subjects and the patients with bipolar disorder. Additionally, the computational model parameters capturing these processes can significantly improve patient/control classification, potentially providing useful diagnosis insight.
Collapse
Affiliation(s)
- Andra Geana
- Department of Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, Rhode Island.
| | - Deanna M Barch
- Department of Psychological & Brain Sciences, Washington University in St. Louis, St. Louis, Missouri; Department of Radiology, Mallinckrodt Institute of Radiology, Washington University School of Medicine, St. Louis, Missouri; Department of Psychiatry, Washington University School of Medicine, St. Louis, Missouri
| | - James M Gold
- Department of Psychiatry, Maryland Psychiatric Research Center, Baltimore, Maryland
| | - Cameron S Carter
- Department of Psychiatry, University of California, Davis, California
| | - Angus W MacDonald
- Department of Psychology, University of Minnesota, Minneapolis, Minnesota
| | - J Daniel Ragland
- Department of Psychiatry, University of California, Davis, California
| | | | - Michael J Frank
- Department of Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, Rhode Island
| |
Collapse
|
10
|
Hitchcock P, Forman E, Rothstein N, Zhang F, Kounios J, Niv Y, Sims C. Rumination Derails Reinforcement Learning with Possible Implications for Ineffective Behavior. Clin Psychol Sci 2022; 10:714-733. [PMID: 35935262 PMCID: PMC9354806 DOI: 10.1177/21677026211051324] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/24/2023]
Abstract
How does rumination affect reinforcement learning-the ubiquitous process by which we adjust behavior after error in order to behave more effectively in the future? In a within-subject design (n=49), we tested whether experimentally manipulated rumination disrupts reinforcement learning in a multidimensional learning task previously shown to rely on selective attention. Rumination impaired performance, yet unexpectedly this impairment could not be attributed to decreased attentional breadth (quantified using a "decay" parameter in a computational model). Instead, trait rumination (between subjects) was associated with higher decay rates (implying narrower attention), yet not with impaired performance. Our task-performance results accord with the possibility that state rumination promotes stress-generating behavior in part by disrupting reinforcement learning. The trait-rumination finding accords with the predictions of a prominent model of trait rumination (the attentional-scope model). More work is needed to understand the specific mechanisms by which state rumination disrupts reinforcement learning.
Collapse
Affiliation(s)
- Peter Hitchcock
- Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI
| | - Evan Forman
- Psychology Department, Drexel University, Philadelphia, PA
| | - Nina Rothstein
- Applied Cognitive & Brain Sciences, Drexel University, Philadelphia, PA
| | - Fengqing Zhang
- Psychology Department, Drexel University, Philadelphia, PA
| | - John Kounios
- Applied Cognitive & Brain Sciences, Drexel University, Philadelphia, PA
| | - Yael Niv
- Princeton Neuroscience Institute & Psychology Department, Princeton University, Princeton, NJ
| | - Chris Sims
- Cognitive Science Department, Rensselaer Polytechnic Institute, Troy, NY
| |
Collapse
|
11
|
Ho MK, Abel D, Correa CG, Littman ML, Cohen JD, Griffiths TL. People construct simplified mental representations to plan. Nature 2022; 606:129-136. [PMID: 35589843 DOI: 10.1038/s41586-022-04743-9] [Citation(s) in RCA: 18] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2021] [Accepted: 04/07/2022] [Indexed: 11/09/2022]
Abstract
One of the most striking features of human cognition is the ability to plan. Two aspects of human planning stand out-its efficiency and flexibility. Efficiency is especially impressive because plans must often be made in complex environments, and yet people successfully plan solutions to many everyday problems despite having limited cognitive resources1-3. Standard accounts in psychology, economics and artificial intelligence have suggested that human planning succeeds because people have a complete representation of a task and then use heuristics to plan future actions in that representation4-11. However, this approach generally assumes that task representations are fixed. Here we propose that task representations can be controlled and that such control provides opportunities to quickly simplify problems and more easily reason about them. We propose a computational account of this simplification process and, in a series of preregistered behavioural experiments, show that it is subject to online cognitive control12-14 and that people optimally balance the complexity of a task representation and its utility for planning and acting. These results demonstrate how strategically perceiving and conceiving problems facilitates the effective use of limited cognitive resources.
Collapse
Affiliation(s)
- Mark K Ho
- Department of Psychology, Princeton University, Princeton, NJ, USA. .,Department of Computer Science, Princeton University, Princeton, NJ, USA.
| | - David Abel
- Department of Computer Science, Brown University, Providence, RI, USA.,DeepMind, London, UK
| | - Carlos G Correa
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Michael L Littman
- Department of Computer Science, Brown University, Providence, RI, USA
| | - Jonathan D Cohen
- Department of Psychology, Princeton University, Princeton, NJ, USA.,Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Thomas L Griffiths
- Department of Psychology, Princeton University, Princeton, NJ, USA.,Department of Computer Science, Princeton University, Princeton, NJ, USA
| |
Collapse
|
12
|
Eckstein MK, Master SL, Xia L, Dahl RE, Wilbrecht L, Collins AGE. The interpretation of computational model parameters depends on the context. eLife 2022; 11:75474. [PMID: 36331872 PMCID: PMC9635876 DOI: 10.7554/elife.75474] [Citation(s) in RCA: 18] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Accepted: 09/09/2022] [Indexed: 11/06/2022] Open
Abstract
Reinforcement Learning (RL) models have revolutionized the cognitive and brain sciences, promising to explain behavior from simple conditioning to complex problem solving, to shed light on developmental and individual differences, and to anchor cognitive processes in specific brain mechanisms. However, the RL literature increasingly reveals contradictory results, which might cast doubt on these claims. We hypothesized that many contradictions arise from two commonly-held assumptions about computational model parameters that are actually often invalid: That parameters generalize between contexts (e.g. tasks, models) and that they capture interpretable (i.e. unique, distinctive) neurocognitive processes. To test this, we asked 291 participants aged 8–30 years to complete three learning tasks in one experimental session, and fitted RL models to each. We found that some parameters (exploration / decision noise) showed significant generalization: they followed similar developmental trajectories, and were reciprocally predictive between tasks. Still, generalization was significantly below the methodological ceiling. Furthermore, other parameters (learning rates, forgetting) did not show evidence of generalization, and sometimes even opposite developmental trajectories. Interpretability was low for all parameters. We conclude that the systematic study of context factors (e.g. reward stochasticity; task volatility) will be necessary to enhance the generalizability and interpretability of computational cognitive models.
Collapse
Affiliation(s)
| | - Sarah L Master
- Department of Psychology, University of California, BerkeleyBerkeleyUnited States,Department of Psychology, New York UniversityNew YorkUnited States
| | - Liyu Xia
- Department of Psychology, University of California, BerkeleyBerkeleyUnited States,Department of Mathematics, University of California, BerkeleyBerkeleyUnited States
| | - Ronald E Dahl
- Institute of Human Development, University of California, BerkeleyBerkeleyUnited States
| | - Linda Wilbrecht
- Department of Psychology, University of California, BerkeleyBerkeleyUnited States,Helen Wills Neuroscience Institute, University of California, BerkeleyBerkeleyUnited States
| | - Anne GE Collins
- Department of Psychology, University of California, BerkeleyBerkeleyUnited States,Helen Wills Neuroscience Institute, University of California, BerkeleyBerkeleyUnited States
| |
Collapse
|
13
|
A Computational View on the Nature of Reward and Value in Anhedonia. Curr Top Behav Neurosci 2021; 58:421-441. [PMID: 34935117 DOI: 10.1007/7854_2021_290] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Abstract
Anhedonia - a common feature of depression and other neuropsychiatric disorders - encompasses a reduction in the subjective experience and anticipation of rewarding events, and a reduction in the motivation to seek out such events. The presence of anhedonia often predicts or accompanies treatment resistance, and as such better interventions and treatments are important. Yet the mechanisms giving rise to anhedonia are not well understood. In this chapter, we briefly review existing computational conceptualisations of anhedonia. We argue that they are mostly descriptive and fail to provide an explanatory account of why anhedonia may occur. Working within the framework of reinforcement learning, we examine two potential computational mechanisms that could give rise to anhedonic phenomena. First, we show how anhedonia can arise in multi-dimensional drive-reduction settings through a trade-off between different rewards or needs. We then generalise this in terms of model-based value inference and identify a key role for associational belief structure. We close with a brief discussion of treatment implications of both of these conceptualisations. In summary, computational accounts of anhedonia have provided a useful descriptive framework. Recent advances in reinforcement learning suggest promising avenues by which the mechanisms underlying anhedonia may be teased apart, potentially motivating novel approaches to treatment.
Collapse
|
14
|
Wurm F, Walentowska W, Ernst B, Severo MC, Pourtois G, Steinhauser M. Task Learnability Modulates Surprise but Not Valence Processing for Reinforcement Learning in Probabilistic Choice Tasks. J Cogn Neurosci 2021; 34:34-53. [PMID: 34879392 DOI: 10.1162/jocn_a_01777] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
The goal of temporal difference (TD) reinforcement learning is to maximize outcomes and improve future decision-making. It does so by utilizing a prediction error (PE), which quantifies the difference between the expected and the obtained outcome. In gambling tasks, however, decision-making cannot be improved because of the lack of learnability. On the basis of the idea that TD utilizes two independent bits of information from the PE (valence and surprise), we asked which of these aspects is affected when a task is not learnable. We contrasted behavioral data and ERPs in a learning variant and a gambling variant of a simple two-armed bandit task, in which outcome sequences were matched across tasks. Participants were explicitly informed that feedback could be used to improve performance in the learning task but not in the gambling task, and we predicted a corresponding modulation of the aspects of the PE. We used a model-based analysis of ERP data to extract the neural footprints of the valence and surprise information in the two tasks. Our results revealed that task learnability modulates reinforcement learning via the suppression of surprise processing but leaves the processing of valence unaffected. On the basis of our model and the data, we propose that task learnability can selectively suppress TD learning as well as alter behavioral adaptation based on a flexible cost-benefit arbitration.
Collapse
Affiliation(s)
- Franz Wurm
- Catholic University of Eichstätt-Ingolstadt, Germany.,Leiden University.,Leiden Institute for Brain and Cognition
| | | | | | | | | | | |
Collapse
|
15
|
Liakoni V, Lehmann MP, Modirshanechi A, Brea J, Lutti A, Gerstner W, Preuschoff K. Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making. Neuroimage 2021; 246:118780. [PMID: 34875383 DOI: 10.1016/j.neuroimage.2021.118780] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2021] [Revised: 08/03/2021] [Accepted: 12/04/2021] [Indexed: 11/25/2022] Open
Abstract
Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.
Collapse
Affiliation(s)
- Vasiliki Liakoni
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland.
| | - Marco P Lehmann
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Alireza Modirshanechi
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Johanni Brea
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Antoine Lutti
- Laboratoire de recherche en neuroimagerie (LREN), Department of Clinical Neurosciences, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
| | - Wulfram Gerstner
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Kerstin Preuschoff
- Geneva Finance Research Institute & Interfaculty Center for Affective Sciences, University of Geneva, Geneva, Switzerland
| |
Collapse
|
16
|
FeldmanHall O, Nassar MR. The computational challenge of social learning. Trends Cogn Sci 2021; 25:1045-1057. [PMID: 34583876 PMCID: PMC8585698 DOI: 10.1016/j.tics.2021.09.002] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2021] [Revised: 08/31/2021] [Accepted: 09/01/2021] [Indexed: 10/20/2022]
Abstract
The complex reward structure of the social world and the uncertainty endemic to social contexts poses a challenge for modeling. For example, during social interactions, the actions of one person influence the internal states of another. These social dependencies make it difficult to formalize social learning problems in a mathematically tractable way. While it is tempting to dispense with these complexities, they are a defining feature of social life. Because the structure of social interactions challenges the simplifying assumptions often made in models, they make an ideal testbed for computational models of cognition. By adopting a framework that embeds existing social knowledge into the model, we can go beyond explaining behaviors in laboratory tasks to explaining those observed in the wild.
Collapse
Affiliation(s)
- Oriel FeldmanHall
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI 02912, USA; Carney Institute for Brain Sciences, Brown University, Providence, RI 02912, USA.
| | - Matthew R Nassar
- Carney Institute for Brain Sciences, Brown University, Providence, RI 02912, USA; Department of Neuroscience, Brown University, Providence, RI 02912, USA
| |
Collapse
|
17
|
Xu HA, Modirshanechi A, Lehmann MP, Gerstner W, Herzog MH. Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making. PLoS Comput Biol 2021; 17:e1009070. [PMID: 34081705 PMCID: PMC8205159 DOI: 10.1371/journal.pcbi.1009070] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Revised: 06/15/2021] [Accepted: 05/12/2021] [Indexed: 11/19/2022] Open
Abstract
Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.
Collapse
Affiliation(s)
- He A. Xu
- Laboratory of Psychophysics, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Alireza Modirshanechi
- Brain-Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- School of Computer and Communication Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Marco P. Lehmann
- Brain-Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- School of Computer and Communication Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Wulfram Gerstner
- Brain-Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- School of Computer and Communication Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Michael H. Herzog
- Laboratory of Psychophysics, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- Brain-Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| |
Collapse
|
18
|
Wilson RC, Bonawitz E, Costa VD, Ebitz RB. Balancing exploration and exploitation with information and randomization. Curr Opin Behav Sci 2021; 38:49-56. [PMID: 33184605 PMCID: PMC7654823 DOI: 10.1016/j.cobeha.2020.10.001] [Citation(s) in RCA: 77] [Impact Index Per Article: 25.7] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022]
Abstract
Explore-exploit decisions require us to trade off the benefits of exploring unknown options to learn more about them, with exploiting known options, for immediate reward. Such decisions are ubiquitous in nature, but from a computational perspective, they are notoriously hard. There is therefore much interest in how humans and animals make these decisions and recently there has been an explosion of research in this area. Here we provide a biased and incomplete snapshot of this field focusing on the major finding that many organisms use two distinct strategies to solve the explore-exploit dilemma: a bias for information ('directed exploration') and the randomization of choice ('random exploration'). We review evidence for the existence of these strategies, their computational properties, their neural implementations, as well as how directed and random exploration vary over the lifespan. We conclude by highlighting open questions in this field that are ripe to both explore and exploit.
Collapse
Affiliation(s)
- Robert C. Wilson
- Department of Psychology, University of Arizona, Tucson AZ USA
- Cognitive Science Program, University of Arizona, Tucson AZ USA
- Evelyn F. McKnight Brain Institute, University of Arizona, Tucson AZ USA
| | | | - Vincent D. Costa
- Department of Behavioral Neuroscience, Oregon Health and Science University, Portland OR USA
| | - R. Becket Ebitz
- Department of Neuroscience, University of Montréal, Montréal, Québec, Canada
| |
Collapse
|
19
|
Roy NA, Bak JH, Akrami A, Brody CD, Pillow JW. Extracting the dynamics of behavior in sensory decision-making experiments. Neuron 2021; 109:597-610.e6. [PMID: 33412101 PMCID: PMC7897255 DOI: 10.1016/j.neuron.2020.12.004] [Citation(s) in RCA: 38] [Impact Index Per Article: 12.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2020] [Revised: 10/23/2020] [Accepted: 12/03/2020] [Indexed: 11/21/2022]
Abstract
Decision-making strategies evolve during training and can continue to vary even in well-trained animals. However, studies of sensory decision-making tend to characterize behavior in terms of a fixed psychometric function that is fit only after training is complete. Here, we present PsyTrack, a flexible method for inferring the trajectory of sensory decision-making strategies from choice data. We apply PsyTrack to training data from mice, rats, and human subjects learning to perform auditory and visual decision-making tasks. We show that it successfully captures trial-to-trial fluctuations in the weighting of sensory stimuli, bias, and task-irrelevant covariates such as choice and stimulus history. This analysis reveals dramatic differences in learning across mice and rapid adaptation to changes in task statistics. PsyTrack scales easily to large datasets and offers a powerful tool for quantifying time-varying behavior in a wide variety of animals and tasks.
Collapse
Affiliation(s)
- Nicholas A Roy
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA.
| | - Ji Hyun Bak
- Korea Institute for Advanced Study, Seoul 02455, South Korea; Redwood Center for Theoretical Neuroscience, University of California, Berkeley, Berkeley, CA 94720, USA
| | - Athena Akrami
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA; Sainsbury Wellcome Centre, University College London, London W1T 4JG, UK
| | - Carlos D Brody
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA; Howard Hughes Medical Institute, Princeton University, Princeton, NJ 08544, USA
| | - Jonathan W Pillow
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA; Department of Psychology, Princeton University, Princeton, NJ 08544, USA.
| |
Collapse
|
20
|
Revisiting the importance of model fitting for model-based fMRI: It does matter in computational psychiatry. PLoS Comput Biol 2021; 17:e1008738. [PMID: 33561125 PMCID: PMC7899379 DOI: 10.1371/journal.pcbi.1008738] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2020] [Revised: 02/22/2021] [Accepted: 01/25/2021] [Indexed: 11/19/2022] Open
Abstract
Computational modeling has been applied for data analysis in psychology, neuroscience, and psychiatry. One of its important uses is to infer the latent variables underlying behavior by which researchers can evaluate corresponding neural, physiological, or behavioral measures. This feature is especially crucial for computational psychiatry, in which altered computational processes underlying mental disorders are of interest. For instance, several studies employing model-based fMRI-a method for identifying brain regions correlated with latent variables-have shown that patients with mental disorders (e.g., depression) exhibit diminished neural responses to reward prediction errors (RPEs), which are the differences between experienced and predicted rewards. Such model-based analysis has the drawback that the parameter estimates and inference of latent variables are not necessarily correct-rather, they usually contain some errors. A previous study theoretically and empirically showed that the error in model-fitting does not necessarily cause a serious error in model-based fMRI. However, the study did not deal with certain situations relevant to psychiatry, such as group comparisons between patients and healthy controls. We developed a theoretical framework to explore such situations. We demonstrate that the parameter-misspecification can critically affect the results of group comparison. We demonstrate that even if the RPE response in patients is completely intact, a spurious difference to healthy controls is observable. Such a situation occurs when the ground-truth learning rate differs between groups but a common learning rate is used, as per previous studies. Furthermore, even if the parameters are appropriately fitted to individual participants, spurious group differences in RPE responses are observable when the model lacks a component that differs between groups. These results highlight the importance of appropriate model-fitting and the need for caution when interpreting the results of model-based fMRI.
Collapse
|
21
|
Strasser A, Luksys G, Xin L, Pessiglione M, Gruetter R, Sandi C. Glutamine-to-glutamate ratio in the nucleus accumbens predicts effort-based motivated performance in humans. Neuropsychopharmacology 2020; 45:2048-2057. [PMID: 32688366 PMCID: PMC7547698 DOI: 10.1038/s41386-020-0760-6] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/19/2020] [Revised: 06/18/2020] [Accepted: 07/02/2020] [Indexed: 12/15/2022]
Abstract
Substantial evidence implicates the nucleus accumbens in motivated performance, but very little is known about the neurochemical underpinnings of individual differences in motivation. Here, we applied 1H magnetic resonance spectroscopy (1H-MRS) at ultra-high-field in the nucleus accumbens and inquired whether levels of glutamate (Glu), glutamine (Gln), GABA or their ratios predict interindividual differences in effort-based motivated task performance. Given the incentive value of social competition, we also examined differences in performance under self-motivated or competition settings. Our results indicate that higher accumbal Gln-to-Glu ratio predicts better overall performance and reduced effort perception. As performance is the outcome of multiple cognitive, motor and physiological processes, we applied computational modeling to estimate best-fitting individual parameters related to specific processes modeled with utility, effort and performance functions. This model-based analysis revealed that accumbal Gln-to-Glu ratio specifically relates to stamina; i.e., the capacity to maintain performance over long periods. It also indicated that competition boosts performance from task onset, particularly for low Gln-to-Glu individuals. In conclusion, our findings provide novel insights implicating accumbal Gln and Glu balance on the prediction of specific computational components of motivated performance. This approach and findings can help developing therapeutic strategies based on targeting metabolism to ameliorate deficits in effort engagement.
Collapse
Affiliation(s)
- Alina Strasser
- grid.5333.60000000121839049Laboratory of Behavioral Genetics (LGC), Brain Mind Institute (BMI), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Gediminas Luksys
- Centre for Discovery Brain Sciences (CDBS), University of Edinburgh, Edinburgh, UK. .,ZJU-UoE Institute, Zhejiang University International Campus, Haining, China.
| | - Lijing Xin
- grid.5333.60000000121839049Animal Imaging and Technology Core (AIT), Center for Biomedical Imaging (CIBM), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Mathias Pessiglione
- grid.411439.a0000 0001 2150 9058Motivation, Brain and Behavior Team, Brain and Spine Institute (ICM), Paris, France
| | - Rolf Gruetter
- grid.5333.60000000121839049Laboratory of Functional and Metabolic Imaging, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland ,grid.9851.50000 0001 2165 4204Department of Radiology, University of Lausanne (UNIL), Lausanne, Switzerland ,grid.8591.50000 0001 2322 4988Department of Radiology, University of Geneva (UNIGE), Geneva, Switzerland
| | - Carmen Sandi
- Laboratory of Behavioral Genetics (LGC), Brain Mind Institute (BMI), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland.
| |
Collapse
|
22
|
Master SL, Eckstein MK, Gotlieb N, Dahl R, Wilbrecht L, Collins AGE. Distentangling the systems contributing to changes in learning during adolescence. Dev Cogn Neurosci 2020; 41:100732. [PMID: 31826837 PMCID: PMC6994540 DOI: 10.1016/j.dcn.2019.100732] [Citation(s) in RCA: 35] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2019] [Revised: 09/23/2019] [Accepted: 11/04/2019] [Indexed: 12/18/2022] Open
Abstract
Multiple neurocognitive systems contribute simultaneously to learning. For example, dopamine and basal ganglia (BG) systems are thought to support reinforcement learning (RL) by incrementally updating the value of choices, while the prefrontal cortex (PFC) contributes different computations, such as actively maintaining precise information in working memory (WM). It is commonly thought that WM and PFC show more protracted development than RL and BG systems, yet their contributions are rarely assessed in tandem. Here, we used a simple learning task to test how RL and WM contribute to changes in learning across adolescence. We tested 187 subjects ages 8 to 17 and 53 adults (25-30). Participants learned stimulus-action associations from feedback; the learning load was varied to be within or exceed WM capacity. Participants age 8-12 learned slower than participants age 13-17, and were more sensitive to load. We used computational modeling to estimate subjects' use of WM and RL processes. Surprisingly, we found more protracted changes in RL than WM during development. RL learning rate increased with age until age 18 and WM parameters showed more subtle, gender- and puberty-dependent changes early in adolescence. These results can inform education and intervention strategies based on the developmental science of learning.
Collapse
Affiliation(s)
- Sarah L Master
- Department of Psychology, University of California, Berkeley, United States
| | - Maria K Eckstein
- Department of Psychology, University of California, Berkeley, United States
| | - Neta Gotlieb
- Department of Psychology, University of California, Berkeley, United States
| | - Ronald Dahl
- Institute of Human Development and School of Public Health, University of California, Berkeley, United States
| | - Linda Wilbrecht
- Department of Psychology, University of California, Berkeley, United States; Helen Wills Neuroscience Institute, University of California, Berkeley, United States
| | - Anne G E Collins
- Department of Psychology, University of California, Berkeley, United States; Helen Wills Neuroscience Institute, University of California, Berkeley, United States
| |
Collapse
|
23
|
Wilson RC, Collins AG. Ten simple rules for the computational modeling of behavioral data. eLife 2019; 8:49547. [PMID: 31769410 PMCID: PMC6879303 DOI: 10.7554/elife.49547] [Citation(s) in RCA: 221] [Impact Index Per Article: 44.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2019] [Accepted: 10/09/2019] [Indexed: 02/06/2023] Open
Abstract
Computational modeling of behavior has revolutionized psychology and neuroscience. By fitting models to experimental data we can probe the algorithms underlying behavior, find neural correlates of computational variables and better understand the effects of drugs, illness and interventions. But with great power comes great responsibility. Here, we offer ten simple rules to ensure that computational modeling is used with care and yields meaningful insights. In particular, we present a beginner-friendly, pragmatic and details-oriented introduction on how to relate models to data. What, exactly, can a model tell us about the mind? To answer this, we apply our rules to the simplest modeling techniques most accessible to beginning modelers and illustrate them with examples and code available online. However, most rules apply to more advanced techniques. Our hope is that by following our guidelines, researchers will avoid many pitfalls and unleash the power of computational modeling on their own data.
Collapse
Affiliation(s)
- Robert C Wilson
- Department of Psychology, University of Arizona, Tucson, United States.,Cognitive Science Program, University of Arizona, Tucson, United States
| | - Anne Ge Collins
- Department of Psychology, University of California, Berkeley, Berkeley, United States.,Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, United States
| |
Collapse
|
24
|
Hauser TU, Will GJ, Dubois M, Dolan RJ. Annual Research Review: Developmental computational psychiatry. J Child Psychol Psychiatry 2019; 60:412-426. [PMID: 30252127 DOI: 10.1111/jcpp.12964] [Citation(s) in RCA: 27] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 07/03/2018] [Indexed: 11/29/2022]
Abstract
Most psychiatric disorders emerge during childhood and adolescence. This is also a period that coincides with the brain undergoing substantial growth and reorganisation. However, it remains unclear how a heightened vulnerability to psychiatric disorder relates to this brain maturation. Here, we propose 'developmental computational psychiatry' as a framework for linking brain maturation to cognitive development. We argue that through modelling some of the brain's fundamental cognitive computations, and relating them to brain development, we can bridge the gap between brain and cognitive development. This in turn can lead to a richer understanding of the ontogeny of psychiatric disorders. We illustrate this perspective with examples from reinforcement learning and dopamine function. Specifically, we show how computational modelling deepens an understanding of how cognitive processes, such as reward learning, effort learning, and social learning might go awry in psychiatric disorders. Finally, we sketch the promises and limitations of a developmental computational psychiatry.
Collapse
Affiliation(s)
- Tobias U Hauser
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, London, UK.,Wellcome Centre for Human Neuroimaging, University College London, London, UK
| | - Geert-Jan Will
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, London, UK.,Institute of Psychology, Leiden University, Leiden, The Netherlands
| | - Magda Dubois
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, London, UK
| | - Raymond J Dolan
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, London, UK.,Wellcome Centre for Human Neuroimaging, University College London, London, UK
| |
Collapse
|
25
|
Gluth S, Spektor MS, Rieskamp J. Value-based attentional capture affects multi-alternative decision making. eLife 2018; 7:e39659. [PMID: 30394874 PMCID: PMC6218187 DOI: 10.7554/elife.39659] [Citation(s) in RCA: 36] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2018] [Accepted: 10/01/2018] [Indexed: 12/23/2022] Open
Abstract
Humans and other animals often violate economic principles when choosing between multiple alternatives, but the underlying neurocognitive mechanisms remain elusive. A robust finding is that adding a third option can alter the relative preference for the original alternatives, but studies disagree on whether the third option's value decreases or increases accuracy. To shed light on this controversy, we used and extended the paradigm of one study reporting a positive effect. However, our four experiments with 147 human participants and a reanalysis of the original data revealed that the positive effect is neither replicable nor reproducible. In contrast, our behavioral and eye-tracking results are best explained by assuming that the third option's value captures attention and thereby impedes accuracy. We propose a computational model that accounts for the complex interplay of value, attention, and choice. Our theory explains how choice sets and environments influence the neurocognitive processes of multi-alternative decision making.
Collapse
Affiliation(s)
| | - Mikhail S Spektor
- Department of PsychologyUniversity of BaselBaselSwitzerland
- Department of PsychologyUniversity of FreiburgFreiburgGermany
| | - Jörg Rieskamp
- Department of PsychologyUniversity of BaselBaselSwitzerland
| |
Collapse
|
26
|
van den Bos W, Bruckner R, Nassar MR, Mata R, Eppinger B. Computational neuroscience across the lifespan: Promises and pitfalls. Dev Cogn Neurosci 2018; 33:42-53. [PMID: 29066078 PMCID: PMC5916502 DOI: 10.1016/j.dcn.2017.09.008] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2017] [Revised: 08/19/2017] [Accepted: 09/29/2017] [Indexed: 11/26/2022] Open
Abstract
In recent years, the application of computational modeling in studies on age-related changes in decision making and learning has gained in popularity. One advantage of computational models is that they provide access to latent variables that cannot be directly observed from behavior. In combination with experimental manipulations, these latent variables can help to test hypotheses about age-related changes in behavioral and neurobiological measures at a level of specificity that is not achievable with descriptive analysis approaches alone. This level of specificity can in turn be beneficial to establish the identity of the corresponding behavioral and neurobiological mechanisms. In this paper, we will illustrate applications of computational methods using examples of lifespan research on risk taking, strategy selection and reinforcement learning. We will elaborate on problems that can occur when computational neuroscience methods are applied to data of different age groups. Finally, we will discuss potential targets for future applications and outline general shortcomings of computational neuroscience methods for research on human lifespan development.
Collapse
Affiliation(s)
- Wouter van den Bos
- Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany; Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands; International Max Planck Research School LIFE, Berlin, Germany.
| | - Rasmus Bruckner
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany; International Max Planck Research School LIFE, Berlin, Germany
| | - Matthew R Nassar
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, USA
| | - Rui Mata
- Center for Cognitive and Decision Sciences, Department of Psychology, University of Basel, Basel, Switzerland
| | - Ben Eppinger
- Department of Psychology, Concordia University, Montreal, Canada; Department of Psychology, TU Dresden, Dresden, Germany.
| |
Collapse
|
27
|
Ritz H, Nassar MR, Frank MJ, Shenhav A. A Control Theoretic Model of Adaptive Learning in Dynamic Environments. J Cogn Neurosci 2018; 30:1405-1421. [PMID: 29877769 PMCID: PMC6432773 DOI: 10.1162/jocn_a_01289] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/27/2023]
Abstract
To behave adaptively in environments that are noisy and nonstationary, humans and other animals must monitor feedback from their environment and adjust their predictions and actions accordingly. An understudied approach for modeling these adaptive processes comes from the engineering field of control theory, which provides general principles for regulating dynamical systems, often without requiring a generative model. The proportional-integral-derivative (PID) controller is one of the most popular models of industrial process control. The proportional term is analogous to the "delta rule" in psychology, adjusting estimates in proportion to each error in prediction. The integral and derivative terms augment this update to simultaneously improve accuracy and stability. Here, we tested whether the PID algorithm can describe how people sequentially adjust their predictions in response to new information. Across three experiments, we found that the PID controller was an effective model of participants' decisions in noisy, changing environments. In Experiment 1, we reanalyzed a change-point detection experiment and showed that participants' behavior incorporated elements of PID updating. In Experiments 2-3, we developed a task with gradual transitions that we optimized to detect PID-like adjustments. In both experiments, the PID model offered better descriptions of behavioral adjustments than both the classical delta-rule model and its more sophisticated variant, the Kalman filter. We further examined how participants weighted different PID terms in response to salient environmental events, finding that these control terms were modulated by reward, surprise, and outcome entropy. These experiments provide preliminary evidence that adaptive learning in dynamic environments resembles PID control.
Collapse
Affiliation(s)
- Harrison Ritz
- Cognitive, Linguistic & Psychological Science, Brown University, Providence, RI
| | - Matthew R. Nassar
- Cognitive, Linguistic & Psychological Science, Brown University, Providence, RI
| | - Michael J. Frank
- Cognitive, Linguistic & Psychological Science, Brown University, Providence, RI
- Brown Institute for Brain Science, Brown University, Providence, RI
| | - Amitai Shenhav
- Cognitive, Linguistic & Psychological Science, Brown University, Providence, RI
- Brown Institute for Brain Science, Brown University, Providence, RI
| |
Collapse
|
28
|
Collins AGE. The Tortoise and the Hare: Interactions between Reinforcement Learning and Working Memory. J Cogn Neurosci 2018; 30:1422-1432. [DOI: 10.1162/jocn_a_01238] [Citation(s) in RCA: 34] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Learning to make rewarding choices in response to stimuli depends on a slow but steady process, reinforcement learning, and a fast and flexible, but capacity-limited process, working memory. Using both systems in parallel, with their contributions weighted based on performance, should allow us to leverage the best of each system: rapid early learning, supplemented by long-term robust acquisition. However, this assumes that using one process does not interfere with the other. We use computational modeling to investigate the interactions between the two processes in a behavioral experiment and show that working memory interferes with reinforcement learning. Previous research showed that neural representations of reward prediction errors, a key marker of reinforcement learning, were blunted when working memory was used for learning. We thus predicted that arbitrating in favor of working memory to learn faster in simple problems would weaken the reinforcement learning process. We tested this by measuring performance in a delayed testing phase where the use of working memory was impossible, and thus participant choices depended on reinforcement learning. Counterintuitively, but confirming our predictions, we observed that associations learned most easily were retained worse than associations learned slower: Using working memory to learn quickly came at the cost of long-term retention. Computational modeling confirmed that this could only be accounted for by working memory interference in reinforcement learning computations. These results further our understanding of how multiple systems contribute in parallel to human learning and may have important applications for education and computational psychiatry.
Collapse
|
29
|
Abstract
Current reinforcement-learning models often assume simplified decision processes that do not fully reflect the dynamic complexities of choice processes. Conversely, sequential-sampling models of decision making account for both choice accuracy and response time, but assume that decisions are based on static decision values. To combine these two computational models of decision making and learning, we implemented reinforcement-learning models in which the drift diffusion model describes the choice process, thereby capturing both within- and across-trial dynamics. To exemplify the utility of this approach, we quantitatively fit data from a common reinforcement-learning paradigm using hierarchical Bayesian parameter estimation, and compared model variants to determine whether they could capture the effects of stimulant medication in adult patients with attention-deficit hyperactivity disorder (ADHD). The model with the best relative fit provided a good description of the learning process, choices, and response times. A parameter recovery experiment showed that the hierarchical Bayesian modeling approach enabled accurate estimation of the model parameters. The model approach described here, using simultaneous estimation of reinforcement-learning and drift diffusion model parameters, shows promise for revealing new insights into the cognitive and neural mechanisms of learning and decision making, as well as the alteration of such processes in clinical groups.
Collapse
|
30
|
Bridwell DA, Cavanagh JF, Collins AGE, Nunez MD, Srinivasan R, Stober S, Calhoun VD. Moving Beyond ERP Components: A Selective Review of Approaches to Integrate EEG and Behavior. Front Hum Neurosci 2018; 12:106. [PMID: 29632480 PMCID: PMC5879117 DOI: 10.3389/fnhum.2018.00106] [Citation(s) in RCA: 46] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2018] [Accepted: 03/06/2018] [Indexed: 11/17/2022] Open
Abstract
Relationships between neuroimaging measures and behavior provide important clues about brain function and cognition in healthy and clinical populations. While electroencephalography (EEG) provides a portable, low cost measure of brain dynamics, it has been somewhat underrepresented in the emerging field of model-based inference. We seek to address this gap in this article by highlighting the utility of linking EEG and behavior, with an emphasis on approaches for EEG analysis that move beyond focusing on peaks or “components” derived from averaging EEG responses across trials and subjects (generating the event-related potential, ERP). First, we review methods for deriving features from EEG in order to enhance the signal within single-trials. These methods include filtering based on user-defined features (i.e., frequency decomposition, time-frequency decomposition), filtering based on data-driven properties (i.e., blind source separation, BSS), and generating more abstract representations of data (e.g., using deep learning). We then review cognitive models which extract latent variables from experimental tasks, including the drift diffusion model (DDM) and reinforcement learning (RL) approaches. Next, we discuss ways to access associations among these measures, including statistical models, data-driven joint models and cognitive joint modeling using hierarchical Bayesian models (HBMs). We think that these methodological tools are likely to contribute to theoretical advancements, and will help inform our understandings of brain dynamics that contribute to moment-to-moment cognitive function.
Collapse
Affiliation(s)
| | - James F Cavanagh
- Department of Psychology, University of New Mexico, Albuquerque, NM, United States
| | - Anne G E Collins
- Department of Psychology, University of California, Berkeley, Berkeley, CA, United States.,Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA, United States
| | - Michael D Nunez
- Department of Cognitive Sciences, University of California, Irvine, Irvine, CA, United States.,Department of Biomedical Engineering, University of California, Irvine, Irvine, CA, United States
| | - Ramesh Srinivasan
- Department of Cognitive Sciences, University of California, Irvine, Irvine, CA, United States.,Department of Biomedical Engineering, University of California, Irvine, Irvine, CA, United States
| | - Sebastian Stober
- Research Focus Cognitive Sciences, University of Potsdam, Potsdam, Germany
| | - Vince D Calhoun
- The Mind Research Network, Albuquerque, NM, United States.,Department of ECE, University of New Mexico, Albuquerque, NM, United States
| |
Collapse
|
31
|
Swart JC, Froböse MI, Cook JL, Geurts DEM, Frank MJ, Cools R, den Ouden HEM. Catecholaminergic challenge uncovers distinct Pavlovian and instrumental mechanisms of motivated (in)action. eLife 2017; 6:e22169. [PMID: 28504638 PMCID: PMC5432212 DOI: 10.7554/elife.22169] [Citation(s) in RCA: 52] [Impact Index Per Article: 7.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2016] [Accepted: 04/24/2017] [Indexed: 01/06/2023] Open
Abstract
Catecholamines modulate the impact of motivational cues on action. Such motivational biases have been proposed to reflect cue-based, 'Pavlovian' effects. Here, we assess whether motivational biases may also arise from asymmetrical instrumental learning of active and passive responses following reward and punishment outcomes. We present a novel paradigm, allowing us to disentangle the impact of reward and punishment on instrumental learning from Pavlovian response biasing. Computational analyses showed that motivational biases reflect both Pavlovian and instrumental effects: reward and punishment cues promoted generalized (in)action in a Pavlovian manner, whereas outcomes enhanced instrumental (un)learning of chosen actions. These cue- and outcome-based biases were altered independently by the catecholamine enhancer melthylphenidate. Methylphenidate's effect varied across individuals with a putative proxy of baseline dopamine synthesis capacity, working memory span. Our study uncovers two distinct mechanisms by which motivation impacts behaviour, and helps refine current models of catecholaminergic modulation of motivated action.
Collapse
Affiliation(s)
- Jennifer C Swart
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Monja I Froböse
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Jennifer L Cook
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- School of Psychology, University of Birmingham, Birmingham, United Kingdom
| | - Dirk EM Geurts
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Michael J Frank
- Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, United States
- Brown Institute for Brain Sciences, Brown University, Providence, United States
| | - Roshan Cools
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Hanneke EM den Ouden
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| |
Collapse
|
32
|
Working Memory Load Strengthens Reward Prediction Errors. J Neurosci 2017; 37:4332-4342. [PMID: 28320846 DOI: 10.1523/jneurosci.2700-16.2017] [Citation(s) in RCA: 62] [Impact Index Per Article: 8.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2016] [Revised: 03/08/2017] [Accepted: 03/12/2017] [Indexed: 11/21/2022] Open
Abstract
Reinforcement learning (RL) in simple instrumental tasks is usually modeled as a monolithic process in which reward prediction errors (RPEs) are used to update expected values of choice options. This modeling ignores the different contributions of different memory and decision-making systems thought to contribute even to simple learning. In an fMRI experiment, we investigated how working memory (WM) and incremental RL processes interact to guide human learning. WM load was manipulated by varying the number of stimuli to be learned across blocks. Behavioral results and computational modeling confirmed that learning was best explained as a mixture of two mechanisms: a fast, capacity-limited, and delay-sensitive WM process together with slower RL. Model-based analysis of fMRI data showed that striatum and lateral prefrontal cortex were sensitive to RPE, as shown previously, but, critically, these signals were reduced when the learning problem was within capacity of WM. The degree of this neural interaction related to individual differences in the use of WM to guide behavioral learning. These results indicate that the two systems do not process information independently, but rather interact during learning.SIGNIFICANCE STATEMENT Reinforcement learning (RL) theory has been remarkably productive at improving our understanding of instrumental learning as well as dopaminergic and striatal network function across many mammalian species. However, this neural network is only one contributor to human learning and other mechanisms such as prefrontal cortex working memory also play a key role. Our results also show that these other players interact with the dopaminergic RL system, interfering with its key computation of reward prediction errors.
Collapse
|
33
|
van Duijvenvoorde ACK, Peters S, Braams BR, Crone EA. What motivates adolescents? Neural responses to rewards and their influence on adolescents' risk taking, learning, and cognitive control. Neurosci Biobehav Rev 2016; 70:135-147. [PMID: 27353570 DOI: 10.1016/j.neubiorev.2016.06.037] [Citation(s) in RCA: 131] [Impact Index Per Article: 16.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2016] [Revised: 06/22/2016] [Accepted: 06/24/2016] [Indexed: 12/29/2022]
Abstract
Adolescence is characterized by pronounced changes in motivated behavior, during which emphasis on potential rewards may result in an increased tendency to approach things that are novel and bring potential for positive reinforcement. While this may result in risky and health-endangering behavior, it may also lead to positive consequences, such as behavioral flexibility and greater learning. In this review we will discuss both the maladaptive and adaptive properties of heightened reward-sensitivity in adolescents by reviewing recent cognitive neuroscience findings in relation to behavioral outcomes. First, we identify brain regions involved in processing rewards in adults and adolescents. Second, we discuss how functional changes in reward-related brain activity during adolescence are related to two behavioral domains: risk taking and cognitive control. Finally, we conclude that progress lies in new levels of explanation by further integration of neural results with behavioral theories and computational models. In addition, we highlight that longitudinal measures, and a better conceptualization of adolescence and environmental determinants, are of crucial importance for understanding positive and negative developmental trajectories.
Collapse
Affiliation(s)
- Anna C K van Duijvenvoorde
- Department of Psychology, Leiden University, The Netherlands; Leiden Institute for Brain and Cognition, Leiden University, The Netherlands
| | - Sabine Peters
- Department of Psychology, Leiden University, The Netherlands; Leiden Institute for Brain and Cognition, Leiden University, The Netherlands
| | - Barbara R Braams
- Department of Psychology, Leiden University, The Netherlands; Leiden Institute for Brain and Cognition, Leiden University, The Netherlands; Department of Psychology, Harvard University, United States
| | - Eveline A Crone
- Department of Psychology, Leiden University, The Netherlands; Leiden Institute for Brain and Cognition, Leiden University, The Netherlands.
| |
Collapse
|