1
|
Gregorová K, Eldar E, Deserno L, Reiter AMF. A cognitive-computational account of mood swings in adolescence. Trends Cogn Sci 2024; 28:290-303. [PMID: 38503636 DOI: 10.1016/j.tics.2024.02.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Revised: 02/06/2024] [Accepted: 02/12/2024] [Indexed: 03/21/2024]
Abstract
Teenagers have a reputation for being fickle, in both their choices and their moods. This variability may help adolescents as they begin to independently navigate novel environments. Recently, however, adolescent moodiness has also been linked to psychopathology. Here, we consider adolescents' mood swings from a novel computational perspective, grounded in reinforcement learning (RL). This model proposes that mood is determined by surprises about outcomes in the environment, and how much we learn from these surprises. It additionally suggests that mood biases learning and choice in a bidirectional manner. Integrating independent lines of research, we sketch a cognitive-computational account of how adolescents' mood, learning, and choice dynamics influence each other, with implications for normative and psychopathological development.
Collapse
Affiliation(s)
- Klára Gregorová
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital, Würzburg 97080, Germany; Department of Psychology, Julius-Maximilians-Universität, Würzburg 97070, Germany; German Center of Prevention Research on Mental Health, Würzburg 97080, Germany
| | - Eran Eldar
- Department of Psychology, Hebrew University of Jerusalem, Jerusalem 9190501, Israel; Department of Cognitive & Brain Sciences, Hebrew University of Jerusalem, Jerusalem 9190501, Israel
| | - Lorenz Deserno
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital, Würzburg 97080, Germany; Department of Psychology, Julius-Maximilians-Universität, Würzburg 97070, Germany; Department of Cognitive & Brain Sciences, Hebrew University of Jerusalem, Jerusalem 9190501, Israel; Department of Psychiatry and Psychotherapy, Technical University of Dresden, Dresden 01069, Germany
| | - Andrea M F Reiter
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital, Würzburg 97080, Germany; Department of Psychology, Julius-Maximilians-Universität, Würzburg 97070, Germany; German Center of Prevention Research on Mental Health, Würzburg 97080, Germany; Collaborative Research Centre 940 Volition and Cognitive Control, Technical University of Dresden, Dresden 01069, Germany.
| |
Collapse
|
2
|
Pauli R, Brazil IA, Kohls G, Klein-Flügge MC, Rogers JC, Dikeos D, Dochnal R, Fairchild G, Fernández-Rivas A, Herpertz-Dahlmann B, Hervas A, Konrad K, Popma A, Stadler C, Freitag CM, De Brito SA, Lockwood PL. Action initiation and punishment learning differ from childhood to adolescence while reward learning remains stable. Nat Commun 2023; 14:5689. [PMID: 37709750 PMCID: PMC10502052 DOI: 10.1038/s41467-023-41124-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2022] [Accepted: 08/24/2023] [Indexed: 09/16/2023] Open
Abstract
Theoretical and empirical accounts suggest that adolescence is associated with heightened reward learning and impulsivity. Experimental tasks and computational models that can dissociate reward learning from the tendency to initiate actions impulsively (action initiation bias) are thus critical to characterise the mechanisms that drive developmental differences. However, existing work has rarely quantified both learning ability and action initiation, or it has relied on small samples. Here, using computational modelling of a learning task collected from a large sample (N = 742, 9-18 years, 11 countries), we test differences in reward and punishment learning and action initiation from childhood to adolescence. Computational modelling reveals that whilst punishment learning rates increase with age, reward learning remains stable. In parallel, action initiation biases decrease with age. Results are similar when considering pubertal stage instead of chronological age. We conclude that heightened reward responsivity in adolescence can reflect differences in action initiation rather than enhanced reward learning.
Collapse
Affiliation(s)
- Ruth Pauli
- Centre for Human Brain Health, School of Psychology, University of Birmingham, Birmingham, UK.
| | - Inti A Brazil
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - Gregor Kohls
- Child Neuropsychology Section, Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, RWTH Aachen University, Aachen, Germany
- Department of Child and Adolescent Psychiatry, Faculty of Medicine, TU, Dresden, Germany
| | - Miriam C Klein-Flügge
- Department of Experimental Psychology, University of Oxford, Oxford, UK
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK
| | - Jack C Rogers
- Centre for Human Brain Health, School of Psychology, University of Birmingham, Birmingham, UK
- Institute for Mental Health, School of Psychology, University of Birmingham, Birmingham, UK
| | - Dimitris Dikeos
- Department of Psychiatry, Medical School, National and Kapodistrian University of Athens, Athens, Greece
| | - Roberta Dochnal
- Faculty of Medicine, Child and Adolescent Psychiatry, Department of the Child Health Center, Szeged University, Szeged, Hungary
| | | | | | - Beate Herpertz-Dahlmann
- Child Neuropsychology Section, Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, RWTH Aachen University, Aachen, Germany
| | - Amaia Hervas
- University Hospital Mutua Terrassa, Barcelona, Spain
| | - Kerstin Konrad
- Child Neuropsychology Section, Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, RWTH Aachen University, Aachen, Germany
- JARA-Brain Institute II, Molecular Neuroscience and Neuroimaging, RWTH Aachen and Research Centre Jülich, Jülich, Germany
| | - Arne Popma
- Department of Child and Adolescent Psychiatry, VU University Medical Center, Amsterdam, Netherlands
| | - Christina Stadler
- Department of Child and Adolescent Psychiatry, Psychiatric University Hospital, University of Basel, Basel, Switzerland
| | - Christine M Freitag
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital Frankfurt, Goethe University, Frankfurt am Main, Germany
| | - Stephane A De Brito
- Centre for Human Brain Health, School of Psychology, University of Birmingham, Birmingham, UK
- Institute for Mental Health, School of Psychology, University of Birmingham, Birmingham, UK
| | - Patricia L Lockwood
- Centre for Human Brain Health, School of Psychology, University of Birmingham, Birmingham, UK.
- Department of Experimental Psychology, University of Oxford, Oxford, UK.
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK.
- Institute for Mental Health, School of Psychology, University of Birmingham, Birmingham, UK.
| |
Collapse
|
3
|
van de Groep IH, Bos MGN, Popma A, Crone EA, Jansen LMC. A neurocognitive model of early onset persistent and desistant antisocial behavior in early adulthood. Front Hum Neurosci 2023; 17:1100277. [PMID: 37533586 PMCID: PMC10392129 DOI: 10.3389/fnhum.2023.1100277] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2022] [Accepted: 06/22/2023] [Indexed: 08/04/2023] Open
Abstract
It remains unclear which functional and neurobiological mechanisms are associated with persistent and desistant antisocial behavior in early adulthood. We reviewed the empirical literature and propose a neurocognitive social information processing model for early onset persistent and desistant antisocial behavior in early adulthood, focusing on how young adults evaluate, act upon, monitor, and learn about their goals and self traits. Based on the reviewed literature, we propose that persistent antisocial behavior is characterized by domain-general impairments in self-relevant and goal-related information processing, regulation, and learning, which is accompanied by altered activity in fronto-limbic brain areas. We propose that desistant antisocial development is associated with more effortful information processing, regulation and learning, that possibly balances self-relevant goals and specific situational characteristics. The proposed framework advances insights by considering individual differences such as psychopathic personality traits, and specific emotional characteristics (e.g., valence of social cues), to further illuminate functional and neural mechanisms underlying heterogenous developmental pathways. Finally, we address important open questions and offer suggestions for future research to improve scientific knowledge on general and context-specific expression and development of antisocial behavior in early adulthood.
Collapse
Affiliation(s)
- Ilse H. van de Groep
- Erasmus School of Social and Behavioral Sciences, Erasmus University Rotterdam, Rotterdam, Netherlands
- Leiden Institute for Brain and Cognition, Leiden University, Leiden, Netherlands
- Department of Child and Adolescent Psychiatry and Psychosocial Care, Amsterdam University Medical Center, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
| | - Marieke G. N. Bos
- Leiden Institute for Brain and Cognition, Leiden University, Leiden, Netherlands
- Department of Developmental and Educational Psychology, Institute of Psychology, Leiden University, Leiden, Netherlands
| | - Arne Popma
- Department of Child and Adolescent Psychiatry and Psychosocial Care, Amsterdam University Medical Center, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Amsterdam Public Health, Mental Health, Amsterdam, Netherlands
| | - Eveline A. Crone
- Erasmus School of Social and Behavioral Sciences, Erasmus University Rotterdam, Rotterdam, Netherlands
- Leiden Institute for Brain and Cognition, Leiden University, Leiden, Netherlands
- Department of Developmental and Educational Psychology, Institute of Psychology, Leiden University, Leiden, Netherlands
| | - Lucres M. C. Jansen
- Department of Child and Adolescent Psychiatry and Psychosocial Care, Amsterdam University Medical Center, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Amsterdam Public Health, Mental Health, Amsterdam, Netherlands
| |
Collapse
|
4
|
Topel S, Ma I, Sleutels J, van Steenbergen H, de Bruijn ERA, van Duijvenvoorde ACK. Expecting the unexpected: a review of learning under uncertainty across development. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2023:10.3758/s13415-023-01098-0. [PMID: 37237092 PMCID: PMC10390612 DOI: 10.3758/s13415-023-01098-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 03/28/2023] [Indexed: 05/28/2023]
Abstract
Many of our decisions take place under uncertainty. To successfully navigate the environment, individuals need to estimate the degree of uncertainty and adapt their behaviors accordingly by learning from experiences. However, uncertainty is a broad construct and distinct types of uncertainty may differentially influence our learning. We provide a semi-systematic review to illustrate cognitive and neurobiological processes involved in learning under two types of uncertainty: learning in environments with stochastic outcomes, and with volatile outcomes. We specifically reviewed studies (N = 26 studies) that included an adolescent population, because adolescence is a period in life characterized by heightened exploration and learning, as well as heightened uncertainty due to experiencing many new, often social, environments. Until now, reviews have not comprehensively compared learning under distinct types of uncertainties in this age range. Our main findings show that although the overall developmental patterns were mixed, most studies indicate that learning from stochastic outcomes, as indicated by increased accuracy in performance, improved with age. We also found that adolescents tended to have an advantage compared with adults and children when learning from volatile outcomes. We discuss potential mechanisms explaining these age-related differences and conclude by outlining future research directions.
Collapse
Affiliation(s)
- Selin Topel
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands.
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands.
| | - Ili Ma
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Jan Sleutels
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden University, Institute for Philosophy, Leiden, The Netherlands
| | - Henk van Steenbergen
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Ellen R A de Bruijn
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Anna C K van Duijvenvoorde
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| |
Collapse
|
5
|
Waltmann M, Herzog N, Reiter AMF, Villringer A, Horstmann A, Deserno L. Diminished reinforcement sensitivity in adolescence is associated with enhanced response switching and reduced coding of choice probability in the medial frontal pole. Dev Cogn Neurosci 2023; 60:101226. [PMID: 36905874 PMCID: PMC10005907 DOI: 10.1016/j.dcn.2023.101226] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Revised: 03/06/2023] [Accepted: 03/06/2023] [Indexed: 03/09/2023] Open
Abstract
Precisely charting the maturation of core neurocognitive functions such as reinforcement learning (RL) and flexible adaptation to changing action-outcome contingencies is key for developmental neuroscience and adjacent fields like developmental psychiatry. However, research in this area is both sparse and conflicted, especially regarding potentially asymmetric development of learning for different motives (obtain wins vs avoid losses) and learning from valenced feedback (positive vs negative). In the current study, we investigated the development of RL from adolescence to adulthood, using a probabilistic reversal learning task modified to experimentally separate motivational context and feedback valence, in a sample of 95 healthy participants between 12 and 45. We show that adolescence is characterized by enhanced novelty seeking and response shifting especially after negative feedback, which leads to poorer returns when reward contingencies are stable. Computationally, this is accounted for by reduced impact of positive feedback on behavior. We also show, using fMRI, that activity of the medial frontopolar cortex reflecting choice probability is attenuated in adolescence. We argue that this can be interpreted as reflecting diminished confidence in upcoming choices. Interestingly, we find no age-related differences between learning in win and loss contexts.
Collapse
Affiliation(s)
- Maria Waltmann
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Centre of Mental Health, University of Würzburg, Würzburg, Germany; Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
| | - Nadine Herzog
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Andrea M F Reiter
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Centre of Mental Health, University of Würzburg, Würzburg, Germany; CRC-940 Volition and Cognitive Control, Faculty of Psychology, Technical University of Dresden, Dresden, Germany
| | - Arno Villringer
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; MindBrainBody Institute, Berlin School of Mind and Brain, Charité-Universitätsmedizin Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Annette Horstmann
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; Department of Psychology and Logopedics, Faculty of Medicine, University of Helsinki, Helsinki, Finland
| | - Lorenz Deserno
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Centre of Mental Health, University of Würzburg, Würzburg, Germany; Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; Neuroimaging Center, Technical University of Dresden, Dresden, Germany
| |
Collapse
|
6
|
Hofmans L, van den Bos W. Social learning across adolescence: A Bayesian neurocognitive perspective. Dev Cogn Neurosci 2022; 58:101151. [PMID: 36183664 PMCID: PMC9526184 DOI: 10.1016/j.dcn.2022.101151] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Revised: 09/14/2022] [Accepted: 09/15/2022] [Indexed: 01/13/2023] Open
Abstract
Adolescence is a period of social re-orientation in which we are generally more prone to peer influence and the updating of our beliefs based on social information, also called social learning, than in any other stage of our life. However, how do we know when to use social information and whose information to use and how does this ability develop across adolescence? Here, we review the social learning literature from a behavioral, neural and computational viewpoint, focusing on the development of brain systems related to executive functioning, value-based decision-making and social cognition. We put forward a Bayesian reinforcement learning framework that incorporates social learning about value associated with particular behavior and uncertainty in our environment and experiences. We discuss how this framework can inform us about developmental changes in social learning, including how the assessment of uncertainty and the ability to adaptively discriminate between information from different social sources change across adolescence. By combining reward-based decision-making in the domains of both informational and normative influence, this framework explains both negative and positive social peer influence in adolescence.
Collapse
Affiliation(s)
- Lieke Hofmans
- Department of Developmental Psychology, University of Amsterdam, Amsterdam, the Netherlands,Correspondence to: Nieuwe Achtergracht 129, room G1.05, 1018WS Amsterdam, the Netherlands.
| | - Wouter van den Bos
- Department of Developmental Psychology, University of Amsterdam, Amsterdam, the Netherlands,Amsterdam Brain and Cognition Center, University of Amsterdam, Amsterdam, the Netherlands,Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany
| |
Collapse
|
7
|
Jepma M, Schaaf JV, Visser I, Huizenga HM. Impaired learning to dissociate advantageous and disadvantageous risky choices in adolescents. Sci Rep 2022; 12:6490. [PMID: 35443773 PMCID: PMC9021244 DOI: 10.1038/s41598-022-10100-7] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2021] [Accepted: 03/25/2022] [Indexed: 11/09/2022] Open
Abstract
Adolescence is characterized by a surge in maladaptive risk-taking behaviors, but whether and how this relates to developmental changes in experience-based learning is largely unknown. In this preregistered study, we addressed this issue using a novel task that allowed us to separate the learning-driven optimization of risky choice behavior over time from overall risk-taking tendencies. Adolescents (12-17 years old) learned to dissociate advantageous from disadvantageous risky choices less well than adults (20-35 years old), and this impairment was stronger in early than mid-late adolescents. Computational modeling revealed that adolescents' suboptimal performance was largely due to an inefficiency in core learning and choice processes. Specifically, adolescents used a simpler, suboptimal, expectation-updating process and a more stochastic choice policy. In addition, the modeling results suggested that adolescents, but not adults, overvalued the highest rewards. Finally, an exploratory latent-mixture model analysis indicated that a substantial proportion of the participants in each age group did not engage in experience-based learning but used a gambler's fallacy strategy, stressing the importance of analyzing individual differences. Our results help understand why adolescents tend to make more, and more persistent, maladaptive risky decisions than adults when the values of these decisions have to be learned from experience.
Collapse
Affiliation(s)
- Marieke Jepma
- Department of Psychology, University of Amsterdam, Amsterdam, the Netherlands.
| | - Jessica V Schaaf
- Department of Psychology, University of Amsterdam, Amsterdam, the Netherlands
| | - Ingmar Visser
- Department of Psychology, University of Amsterdam, Amsterdam, the Netherlands
| | - Hilde M Huizenga
- Department of Psychology, University of Amsterdam, Amsterdam, the Netherlands
| |
Collapse
|
8
|
Eckstein MK, Master SL, Dahl RE, Wilbrecht L, Collins AG. Reinforcement learning and bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal. Dev Cogn Neurosci 2022; 55:101106. [PMID: 35537273 PMCID: PMC9108470 DOI: 10.1016/j.dcn.2022.101106] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2021] [Revised: 03/01/2022] [Accepted: 03/25/2022] [Indexed: 12/02/2022] Open
Abstract
During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated how performance changes across adolescent development in a stochastic, volatile reversal-learning task that uniquely taxes the balance of persistence and flexibility. In a sample of 291 participants aged 8–30, we found that in the mid-teen years, adolescents outperformed both younger and older participants. We developed two independent cognitive models, based on Reinforcement learning (RL) and Bayesian inference (BI). The RL parameter for learning from negative outcomes and the BI parameters specifying participants’ mental models were closest to optimal in mid-teen adolescents, suggesting a central role in adolescent cognitive processing. By contrast, persistence and noise parameters improved monotonically with age. We distilled the insights of RL and BI using principal component analysis and found that three shared components interacted to form the adolescent performance peak: adult-like behavioral quality, child-like time scales, and developmentally-unique processing of positive feedback. This research highlights adolescence as a neurodevelopmental window that can create performance advantages in volatile and uncertain environments. It also shows how detailed insights can be gleaned by using cognitive models in new ways.
Collapse
|
9
|
Eckstein MK, Master SL, Xia L, Dahl RE, Wilbrecht L, Collins AGE. The interpretation of computational model parameters depends on the context. eLife 2022; 11:75474. [PMID: 36331872 PMCID: PMC9635876 DOI: 10.7554/elife.75474] [Citation(s) in RCA: 18] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Accepted: 09/09/2022] [Indexed: 11/06/2022] Open
Abstract
Reinforcement Learning (RL) models have revolutionized the cognitive and brain sciences, promising to explain behavior from simple conditioning to complex problem solving, to shed light on developmental and individual differences, and to anchor cognitive processes in specific brain mechanisms. However, the RL literature increasingly reveals contradictory results, which might cast doubt on these claims. We hypothesized that many contradictions arise from two commonly-held assumptions about computational model parameters that are actually often invalid: That parameters generalize between contexts (e.g. tasks, models) and that they capture interpretable (i.e. unique, distinctive) neurocognitive processes. To test this, we asked 291 participants aged 8–30 years to complete three learning tasks in one experimental session, and fitted RL models to each. We found that some parameters (exploration / decision noise) showed significant generalization: they followed similar developmental trajectories, and were reciprocally predictive between tasks. Still, generalization was significantly below the methodological ceiling. Furthermore, other parameters (learning rates, forgetting) did not show evidence of generalization, and sometimes even opposite developmental trajectories. Interpretability was low for all parameters. We conclude that the systematic study of context factors (e.g. reward stochasticity; task volatility) will be necessary to enhance the generalizability and interpretability of computational cognitive models.
Collapse
Affiliation(s)
| | - Sarah L Master
- Department of Psychology, University of California, BerkeleyBerkeleyUnited States,Department of Psychology, New York UniversityNew YorkUnited States
| | - Liyu Xia
- Department of Psychology, University of California, BerkeleyBerkeleyUnited States,Department of Mathematics, University of California, BerkeleyBerkeleyUnited States
| | - Ronald E Dahl
- Institute of Human Development, University of California, BerkeleyBerkeleyUnited States
| | - Linda Wilbrecht
- Department of Psychology, University of California, BerkeleyBerkeleyUnited States,Helen Wills Neuroscience Institute, University of California, BerkeleyBerkeleyUnited States
| | - Anne GE Collins
- Department of Psychology, University of California, BerkeleyBerkeleyUnited States,Helen Wills Neuroscience Institute, University of California, BerkeleyBerkeleyUnited States
| |
Collapse
|
10
|
Yoo AH, Collins AGE. How Working Memory and Reinforcement Learning Are Intertwined: A Cognitive, Neural, and Computational Perspective. J Cogn Neurosci 2021; 34:551-568. [PMID: 34942642 DOI: 10.1162/jocn_a_01808] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Reinforcement learning and working memory are two core processes of human cognition and are often considered cognitively, neuroscientifically, and algorithmically distinct. Here, we show that the brain networks that support them actually overlap significantly and that they are less distinct cognitive processes than often assumed. We review literature demonstrating the benefits of considering each process to explain properties of the other and highlight recent work investigating their more complex interactions. We discuss how future research in both computational and cognitive sciences can benefit from one another, suggesting that a key missing piece for artificial agents to learn to behave with more human-like efficiency is taking working memory's role in learning seriously. This review highlights the risks of neglecting the interplay between different processes when studying human behavior (in particular when considering individual differences). We emphasize the importance of investigating these dynamics to build a comprehensive understanding of human cognition.
Collapse
|
11
|
Galván A. Adolescent Brain Development and Contextual Influences: A Decade in Review. JOURNAL OF RESEARCH ON ADOLESCENCE : THE OFFICIAL JOURNAL OF THE SOCIETY FOR RESEARCH ON ADOLESCENCE 2021; 31:843-869. [PMID: 34820955 DOI: 10.1111/jora.12687] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Adolescence is a developmental period characterized by substantial psychological, biological, and neurobiological changes. This review discusses the past decade of research on the adolescent brain, as based on the overarching framework that development is a dynamic process both within the individual and between the individual and external inputs. As such, this review focuses on research showing that the development of the brain is influenced by multiple ongoing and dynamic elements. It highlights the implications this body of work on behavioral development and offers areas of opportunity for future research in the coming decade.
Collapse
|
12
|
Eckstein MK, Wilbrecht L, Collins AGE. What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience. Curr Opin Behav Sci 2021; 41:128-137. [PMID: 34984213 PMCID: PMC8722372 DOI: 10.1016/j.cobeha.2021.06.004] [Citation(s) in RCA: 37] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
Reinforcement learning (RL) is a concept that has been invaluable to fields including machine learning, neuroscience, and cognitive science. However, what RL entails differs between fields, leading to difficulties when interpreting and translating findings. After laying out these differences, this paper focuses on cognitive (neuro)science to discuss how we as a field might over-interpret RL modeling results. We too often assume-implicitly-that modeling results generalize between tasks, models, and participant populations, despite negative empirical evidence for this assumption. We also often assume that parameters measure specific, unique (neuro)cognitive processes, a concept we call interpretability, when evidence suggests that they capture different functions across studies and tasks. We conclude that future computational research needs to pay increased attention to implicit assumptions when using RL models, and suggest that a more systematic understanding of contextual factors will help address issues and improve the ability of RL to explain brain and behavior.
Collapse
Affiliation(s)
- Maria K Eckstein
- Department of Psychology, UC Berkeley, 2121 Berkeley Way West, Berkeley, 94720, CA, USA
| | - Linda Wilbrecht
- Department of Psychology, UC Berkeley, 2121 Berkeley Way West, Berkeley, 94720, CA, USA
- Helen Wills Neuroscience Institute, UC Berkeley, 175 Li Ka Shing Center, Berkeley, 94720, CA, USA
| | - Anne G E Collins
- Department of Psychology, UC Berkeley, 2121 Berkeley Way West, Berkeley, 94720, CA, USA
- Helen Wills Neuroscience Institute, UC Berkeley, 175 Li Ka Shing Center, Berkeley, 94720, CA, USA
| |
Collapse
|
13
|
Jepma M, Schaaf JV, Visser I, Huizenga HM. Effects of advice on experienced-based learning in adolescents and adults. J Exp Child Psychol 2021; 211:105230. [PMID: 34256185 DOI: 10.1016/j.jecp.2021.105230] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2020] [Revised: 03/24/2021] [Accepted: 06/16/2021] [Indexed: 01/04/2023]
Abstract
Recent studies that compared effects of pre-learning advice on experience-based learning in adolescents and adults have yielded mixed results. Previous studies on this topic used choice tasks in which age-related differences in advice-related learning bias and exploratory choice behavior are difficult to dissociate. Moreover, these studies did not examine whether effects of advice depend on working memory load. In this preregistered study (in adolescents [13-15 years old] and adults [18-31 years old]), we addressed these issues by factorially combining advice and working memory load manipulations in an estimation task that does not require choices and hence eliminates the influence of known age-related differences in exploration. We found that advice guided participants' initial estimates in both age groups. When advice was correct, this improved estimation performance, especially in adolescents when working memory load was high. When advice was incorrect, it had a longer-lasting effect on adolescents' performance than on adults' performance. In contrast to previous findings in choice tasks, we found no evidence that advice biased learning in either age group. Taken together, our results suggest that learning in an estimation task improves between adolescence and adulthood but that the effects of advice on learning do not differ substantially between adolescents and adults.
Collapse
Affiliation(s)
- Marieke Jepma
- Department of Psychology, University of Amsterdam, 1012 WX Amsterdam, the Netherlands.
| | - Jessica V Schaaf
- Department of Psychology, University of Amsterdam, 1012 WX Amsterdam, the Netherlands
| | - Ingmar Visser
- Department of Psychology, University of Amsterdam, 1012 WX Amsterdam, the Netherlands
| | - Hilde M Huizenga
- Department of Psychology, University of Amsterdam, 1012 WX Amsterdam, the Netherlands
| |
Collapse
|
14
|
Xia L, Master SL, Eckstein MK, Baribault B, Dahl RE, Wilbrecht L, Collins AGE. Modeling changes in probabilistic reinforcement learning during adolescence. PLoS Comput Biol 2021; 17:e1008524. [PMID: 34197447 PMCID: PMC8279421 DOI: 10.1371/journal.pcbi.1008524] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2020] [Revised: 07/14/2021] [Accepted: 05/26/2021] [Indexed: 01/17/2023] Open
Abstract
In the real world, many relationships between events are uncertain and probabilistic. Uncertainty is also likely to be a more common feature of daily experience for youth because they have less experience to draw from than adults. Some studies suggest probabilistic learning may be inefficient in youths compared to adults, while others suggest it may be more efficient in youths in mid adolescence. Here we used a probabilistic reinforcement learning task to test how youth age 8-17 (N = 187) and adults age 18-30 (N = 110) learn about stable probabilistic contingencies. Performance increased with age through early-twenties, then stabilized. Using hierarchical Bayesian methods to fit computational reinforcement learning models, we show that all participants' performance was better explained by models in which negative outcomes had minimal to no impact on learning. The performance increase over age was driven by 1) an increase in learning rate (i.e. decrease in integration time scale); 2) a decrease in noisy/exploratory choices. In mid-adolescence age 13-15, salivary testosterone and learning rate were positively related. We discuss our findings in the context of other studies and hypotheses about adolescent brain development.
Collapse
Affiliation(s)
- Liyu Xia
- Department of Mathematics, University of California Berkeley, Berkeley, California, United States of America
| | - Sarah L. Master
- Department of Psychology, New York University, New York, New York, United States of America
| | - Maria K. Eckstein
- Department of Psychology, University of California Berkeley, Berkeley, California, United States of America
| | - Beth Baribault
- Department of Psychology, University of California Berkeley, Berkeley, California, United States of America
| | - Ronald E. Dahl
- School of Public Health, University of California Berkeley, Berkeley, California, United States of America
| | - Linda Wilbrecht
- Department of Psychology, University of California Berkeley, Berkeley, California, United States of America
- Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, California, United States of America
| | - Anne Gabrielle Eva Collins
- Department of Psychology, University of California Berkeley, Berkeley, California, United States of America
- Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, California, United States of America
| |
Collapse
|
15
|
Uncertainty-driven regulation of learning and exploration in adolescents: A computational account. PLoS Comput Biol 2020; 16:e1008276. [PMID: 32997659 PMCID: PMC7549782 DOI: 10.1371/journal.pcbi.1008276] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2019] [Revised: 10/12/2020] [Accepted: 08/20/2020] [Indexed: 01/31/2023] Open
Abstract
Healthy adults flexibly adapt their learning strategies to ongoing changes in uncertainty, a key feature of adaptive behaviour. However, the developmental trajectory of this ability is yet unknown, as developmental studies have not incorporated trial-to-trial variation in uncertainty in their analyses or models. To address this issue, we compared adolescents’ and adults’ trial-to-trial dynamics of uncertainty, learning rate, and exploration in two tasks that assess learning in noisy but otherwise stable environments. In an estimation task—which provides direct indices of trial-specific learning rate—both age groups reduced their learning rate over time, as self-reported uncertainty decreased. Accordingly, the estimation data in both groups was better explained by a Bayesian model with dynamic learning rate (Kalman filter) than by conventional reinforcement-learning models. Furthermore, adolescents’ learning rates asymptoted at a higher level, reflecting an over-weighting of the most recent outcome, and the estimated Kalman-filter parameters suggested that this was due to an overestimation of environmental volatility. In a choice task, both age groups became more likely to choose the higher-valued option over time, but this increase in choice accuracy was smaller in the adolescents. In contrast to the estimation task, we found no evidence for a Bayesian expectation-updating process in the choice task, suggesting that estimation and choice tasks engage different learning processes. However, our modeling results of the choice task suggested that both age groups reduced their degree of exploration over time, and that the adolescents explored overall more than the adults. Finally, age-related differences in exploration parameters from fits to the choice data were mediated by participants’ volatility parameter from fits to the estimation data. Together, these results suggest that adolescents overestimate the rate of environmental change, resulting in elevated learning rates and increased exploration, which may help understand developmental changes in learning and decision-making. To successfully learn the value of stimuli and actions, people should take into account their current (un)certainty about these values: Learning rates and exploration should be high when one’s value estimates are highly uncertain (in the beginning of learning), and decrease over time as evidence accumulates and uncertainty decreases. Recent studies have shown that healthy adults flexibly adapt their learning strategies based on ongoing changes in uncertainty, consistent with normative learning. However, the development of this ability prior to adulthood is yet unknown, as developmental learning studies have not considered trial-to-trial changes in uncertainty. Here, we show that adolescents, as compared to adults, showed a smaller decrease in both learning rate and exploration over time. Computational modeling revealed that both of these effects were due to adolescents overestimating the amount of environmental volatility, which made them more sensitive to recent relative to older evidence. The overestimation of volatility during adolescence may represent the rapidly changing environmental demands during this developmental period, and can help understand the surge in real-life risk taking and exploratory behaviours characteristic of adolescents.
Collapse
|
16
|
Moin Afshar N, Keip AJ, Taylor JR, Lee D, Groman SM. Reinforcement Learning during Adolescence in Rats. J Neurosci 2020; 40:5857-5870. [PMID: 32601244 PMCID: PMC7380962 DOI: 10.1523/jneurosci.0910-20.2020] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2020] [Revised: 06/07/2020] [Accepted: 06/17/2020] [Indexed: 12/14/2022] Open
Abstract
The most dynamic period of postnatal brain development occurs during adolescence, the period between childhood and adulthood. Neuroimaging studies have observed morphologic and functional changes during adolescence, and it is believed that these changes serve to improve the functions of circuits that underlie decision-making. Direct evidence in support of this hypothesis, however, has been limited because most preclinical decision-making paradigms are not readily translated to humans. Here, we developed a reversal-learning protocol for the rapid assessment of adaptive choice behavior in dynamic environments in rats as young as postnatal day 30. A computational framework was used to elucidate the reinforcement-learning mechanisms that change in adolescence and into adulthood. Using a cross-sectional and longitudinal design, we provide the first evidence that value-based choice behavior in a reversal-learning task improves during adolescence in male and female Long-Evans rats and demonstrate that the increase in reversal performance is due to alterations in value updating for positive outcomes. Furthermore, we report that reversal-learning trajectories in adolescence reliably predicted reversal performance in adulthood. This novel behavioral protocol provides a unique platform for conducting biological and systems-level analyses of the neurodevelopmental mechanisms of decision-making.SIGNIFICANCE STATEMENT The neurodevelopmental adaptations that occur during adolescence are hypothesized to underlie age-related improvements in decision-making, but evidence to support this hypothesis has been limited. Here, we describe a novel behavioral protocol for rapidly assessing adaptive choice behavior in adolescent rats with a reversal-learning paradigm. Using a computational approach, we demonstrate that age-related changes in reversal-learning performance in male and female Long-Evans rats are linked to specific reinforcement-learning mechanisms and are predictive of reversal-learning performance in adulthood. Our behavioral protocol provides a unique platform for elucidating key components of adolescent brain function.
Collapse
Affiliation(s)
- Neema Moin Afshar
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
| | - Alex J Keip
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
| | - Jane R Taylor
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
- Department of Neuroscience, Yale School of Medicine, New Haven, Connecticut 06520-8001
| | - Daeyeol Lee
- The Zanvyl Krieger Mind/Brain Institute, Johns Hopkins University, Baltimore, Maryland 21218
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins University School of Medicine, Baltimore, Maryland 21205
- Department of Psychological & Brain Sciences, Johns Hopkins University, Baltimore, Maryland 21218
- Kavli Neuroscience Discovery Institute, Johns Hopkins University, Baltimore, Maryland 21205
| | - Stephanie M Groman
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
| |
Collapse
|
17
|
Affiliation(s)
- Quenten Highgate
- School of Psychology, Victoria University of Wellington, Wellington, New Zealand
| | - Susan Schenk
- School of Psychology, Victoria University of Wellington, Wellington, New Zealand
| |
Collapse
|
18
|
Nussenbaum K, Hartley CA. Reinforcement learning across development: What insights can we draw from a decade of research? Dev Cogn Neurosci 2019; 40:100733. [PMID: 31770715 PMCID: PMC6974916 DOI: 10.1016/j.dcn.2019.100733] [Citation(s) in RCA: 72] [Impact Index Per Article: 14.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2019] [Revised: 10/24/2019] [Accepted: 11/04/2019] [Indexed: 01/02/2023] Open
Abstract
The past decade has seen the emergence of the use of reinforcement learning models to study developmental change in value-based learning. It is unclear, however, whether these computational modeling studies, which have employed a wide variety of tasks and model variants, have reached convergent conclusions. In this review, we examine whether the tuning of model parameters that govern different aspects of learning and decision-making processes vary consistently as a function of age, and what neurocognitive developmental changes may account for differences in these parameter estimates across development. We explore whether patterns of developmental change in these estimates are better described by differences in the extent to which individuals adapt their learning processes to the statistics of different environments, or by more static learning biases that emerge across varied contexts. We focus specifically on learning rates and inverse temperature parameter estimates, and find evidence that from childhood to adulthood, individuals become better at optimally weighting recent outcomes during learning across diverse contexts and less exploratory in their value-based decision-making. We provide recommendations for how these two possibilities - and potential alternative accounts - can be tested more directly to build a cohesive body of research that yields greater insight into the development of core learning processes.
Collapse
|
19
|
van den Bos W, Bruckner R, Nassar MR, Mata R, Eppinger B. Computational neuroscience across the lifespan: Promises and pitfalls. Dev Cogn Neurosci 2018; 33:42-53. [PMID: 29066078 PMCID: PMC5916502 DOI: 10.1016/j.dcn.2017.09.008] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2017] [Revised: 08/19/2017] [Accepted: 09/29/2017] [Indexed: 11/26/2022] Open
Abstract
In recent years, the application of computational modeling in studies on age-related changes in decision making and learning has gained in popularity. One advantage of computational models is that they provide access to latent variables that cannot be directly observed from behavior. In combination with experimental manipulations, these latent variables can help to test hypotheses about age-related changes in behavioral and neurobiological measures at a level of specificity that is not achievable with descriptive analysis approaches alone. This level of specificity can in turn be beneficial to establish the identity of the corresponding behavioral and neurobiological mechanisms. In this paper, we will illustrate applications of computational methods using examples of lifespan research on risk taking, strategy selection and reinforcement learning. We will elaborate on problems that can occur when computational neuroscience methods are applied to data of different age groups. Finally, we will discuss potential targets for future applications and outline general shortcomings of computational neuroscience methods for research on human lifespan development.
Collapse
Affiliation(s)
- Wouter van den Bos
- Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany; Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands; International Max Planck Research School LIFE, Berlin, Germany.
| | - Rasmus Bruckner
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany; International Max Planck Research School LIFE, Berlin, Germany
| | - Matthew R Nassar
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, USA
| | - Rui Mata
- Center for Cognitive and Decision Sciences, Department of Psychology, University of Basel, Basel, Switzerland
| | - Ben Eppinger
- Department of Psychology, Concordia University, Montreal, Canada; Department of Psychology, TU Dresden, Dresden, Germany.
| |
Collapse
|
20
|
Bernardoni F, Geisler D, King JA, Javadi AH, Ritschel F, Murr J, Reiter AMF, Rössner V, Smolka MN, Kiebel S, Ehrlich S. Altered Medial Frontal Feedback Learning Signals in Anorexia Nervosa. Biol Psychiatry 2018; 83:235-243. [PMID: 29025688 DOI: 10.1016/j.biopsych.2017.07.024] [Citation(s) in RCA: 39] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/10/2017] [Revised: 06/12/2017] [Accepted: 07/05/2017] [Indexed: 12/11/2022]
Abstract
BACKGROUND In their relentless pursuit of thinness, individuals with anorexia nervosa (AN) engage in maladaptive behaviors (restrictive food choices and overexercising) that may originate in altered decision making and learning. METHODS In this functional magnetic resonance imaging study, we employed computational modeling to elucidate the neural correlates of feedback learning and value-based decision making in 36 female patients with AN and 36 age-matched healthy volunteers (12-24 years). Participants performed a decision task that required adaptation to changing reward contingencies. Data were analyzed within a hierarchical Gaussian filter model that captures interindividual variability in learning under uncertainty. RESULTS Behaviorally, patients displayed an increased learning rate specifically after punishments. At the neural level, hemodynamic correlates for the learning rate, expected value, and prediction error did not differ between the groups. However, activity in the posterior medial frontal cortex was elevated in AN following punishment. CONCLUSIONS Our findings suggest that the neural underpinning of feedback learning is selectively altered for punishment in AN.
Collapse
Affiliation(s)
- Fabio Bernardoni
- Division of Psychological and Social Medicine and Developmental Neuroscience, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| | - Daniel Geisler
- Division of Psychological and Social Medicine and Developmental Neuroscience, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| | - Joseph A King
- Division of Psychological and Social Medicine and Developmental Neuroscience, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| | | | - Franziska Ritschel
- Division of Psychological and Social Medicine and Developmental Neuroscience, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany; Eating Disorder Treatment and Research Center, Department of Child and Adolescent Psychiatry, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| | - Julia Murr
- Department of Psychotherapy and Psychosomatic Medicine, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| | - Andrea M F Reiter
- Institute of General Psychology, Biopsychology and Methods of Psychology, Department of Psychology, Technische Universität Dresden, Dresden, Germany; Lifespan Developmental Neuroscience, Technische Universität Dresden, Dresden, Germany
| | - Veit Rössner
- Eating Disorder Treatment and Research Center, Department of Child and Adolescent Psychiatry, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| | - Michael N Smolka
- Department of Psychiatry and Neuroimaging, Technische Universität Dresden, Dresden, Germany
| | - Stefan Kiebel
- Institute of General Psychology, Biopsychology and Methods of Psychology, Department of Psychology, Technische Universität Dresden, Dresden, Germany; Department of Psychiatry and Neuroimaging, Technische Universität Dresden, Dresden, Germany
| | - Stefan Ehrlich
- Division of Psychological and Social Medicine and Developmental Neuroscience, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany; Eating Disorder Treatment and Research Center, Department of Child and Adolescent Psychiatry, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany.
| |
Collapse
|
21
|
Bolenz F, Reiter AMF, Eppinger B. Developmental Changes in Learning: Computational Mechanisms and Social Influences. Front Psychol 2017; 8:2048. [PMID: 29250006 PMCID: PMC5715389 DOI: 10.3389/fpsyg.2017.02048] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2017] [Accepted: 11/09/2017] [Indexed: 11/13/2022] Open
Abstract
Our ability to learn from the outcomes of our actions and to adapt our decisions accordingly changes over the course of the human lifespan. In recent years, there has been an increasing interest in using computational models to understand developmental changes in learning and decision-making. Moreover, extensions of these models are currently applied to study socio-emotional influences on learning in different age groups, a topic that is of great relevance for applications in education and health psychology. In this article, we aim to provide an introduction to basic ideas underlying computational models of reinforcement learning and focus on parameters and model variants that might be of interest to developmental scientists. We then highlight recent attempts to use reinforcement learning models to study the influence of social information on learning across development. The aim of this review is to illustrate how computational models can be applied in developmental science, what they can add to our understanding of developmental mechanisms and how they can be used to bridge the gap between psychological and neurobiological theories of development.
Collapse
Affiliation(s)
- Florian Bolenz
- Chair of Lifespan Developmental Neuroscience, Department of Psychology, Technische Universität Dresden, Dresden, Germany
| | - Andrea M F Reiter
- Chair of Lifespan Developmental Neuroscience, Department of Psychology, Technische Universität Dresden, Dresden, Germany.,Department of Neurology, Max-Planck-Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Ben Eppinger
- Chair of Lifespan Developmental Neuroscience, Department of Psychology, Technische Universität Dresden, Dresden, Germany.,Department of Psychology, Concordia University, Montreal, QC, Canada.,PERFORM Centre, Concordia University, Montreal, QC, Canada
| |
Collapse
|
22
|
Boehme R, Lorenz RC, Gleich T, Romund L, Pelz P, Golde S, Flemming E, Wold A, Deserno L, Behr J, Raufelder D, Heinz A, Beck A. Reversal learning strategy in adolescence is associated with prefrontal cortex activation. Eur J Neurosci 2016; 45:129-137. [PMID: 27628616 DOI: 10.1111/ejn.13401] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2016] [Revised: 08/24/2016] [Accepted: 09/12/2016] [Indexed: 01/14/2023]
Abstract
Adolescence is a critical maturation period for human cognitive control and executive function. In this study, a large sample of adolescents (n = 85) performed a reversal learning task during functional magnetic resonance imaging. We analyzed behavioral data using a reinforcement learning model to provide individually fitted parameters and imaging data with regard to reward prediction errors (PE). Following a model-based approach, we formed two groups depending on whether individuals tended to update expectations predominantly for the chosen stimulus or also for the unchosen one. These groups significantly differed in their problem behavior score obtained using the child behavior checklist (CBCL) and in a measure of their developmental stage. Imaging results showed that dorsolateral striatal areas covaried with PE. Participants who relied less on learning based on task structure showed less prefrontal activation compared with participants who relied more on task structure. An exploratory analysis revealed that PE-related activity was associated with pubertal development in prefrontal areas, insula and anterior cingulate. These findings support the hypothesis that the prefrontal cortex is implicated in mediating flexible goal-directed behavioral control.
Collapse
Affiliation(s)
- Rebecca Boehme
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany.,Center for Social and Affective Neuroscience, Linköping University, Linköping, 58245, Sweden
| | - Robert C Lorenz
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany.,Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany
| | - Tobias Gleich
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany.,NeuroCure Excellence Cluster/Medical Neuroscience, Berlin, Germany
| | - Lydia Romund
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany
| | - Patricia Pelz
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany
| | - Sabrina Golde
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany
| | - Eva Flemming
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany
| | - Andrew Wold
- Center for Social and Affective Neuroscience, Linköping University, Linköping, 58245, Sweden
| | - Lorenz Deserno
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany.,Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,Department of Neurology, Otto-von-Guericke University, Magdeburg, Germany
| | - Joachim Behr
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany.,Institute of Neurophysiology, Charité - Universitätsmedizin Berlin, Berlin, Germany.,Department of Psychiatry, Psychotherapy and Psychosomatics, Medical School Brandenburg - Campus Neuruppin, Neuruppin, Germany
| | - Diana Raufelder
- Department of Educational Science and Psychology, Freie Universität, Berlin, Germany
| | - Andreas Heinz
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany
| | - Anne Beck
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Campus Mitte, Berlin, 10117, Germany
| |
Collapse
|
23
|
Palminteri S, Kilford EJ, Coricelli G, Blakemore SJ. The Computational Development of Reinforcement Learning during Adolescence. PLoS Comput Biol 2016; 12:e1004953. [PMID: 27322574 PMCID: PMC4920542 DOI: 10.1371/journal.pcbi.1004953] [Citation(s) in RCA: 70] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2015] [Accepted: 04/29/2016] [Indexed: 11/19/2022] Open
Abstract
Adolescence is a period of life characterised by changes in learning and decision-making. Learning and decision-making do not rely on a unitary system, but instead require the coordination of different cognitive processes that can be mathematically formalised as dissociable computational modules. Here, we aimed to trace the developmental time-course of the computational modules responsible for learning from reward or punishment, and learning from counterfactual feedback. Adolescents and adults carried out a novel reinforcement learning paradigm in which participants learned the association between cues and probabilistic outcomes, where the outcomes differed in valence (reward versus punishment) and feedback was either partial or complete (either the outcome of the chosen option only, or the outcomes of both the chosen and unchosen option, were displayed). Computational strategies changed during development: whereas adolescents’ behaviour was better explained by a basic reinforcement learning algorithm, adults’ behaviour integrated increasingly complex computational features, namely a counterfactual learning module (enabling enhanced performance in the presence of complete feedback) and a value contextualisation module (enabling symmetrical reward and punishment learning). Unlike adults, adolescent performance did not benefit from counterfactual (complete) feedback. In addition, while adults learned symmetrically from both reward and punishment, adolescents learned from reward but were less likely to learn from punishment. This tendency to rely on rewards and not to consider alternative consequences of actions might contribute to our understanding of decision-making in adolescence. We employed a novel learning task to investigate how adolescents and adults learn from reward versus punishment, and to counterfactual feedback about decisions. Computational analyses revealed that adults and adolescents did not implement the same algorithm to solve the learning task. In contrast to adults, adolescents’ performance did not take into account counterfactual information; adolescents also learned preferentially to seek rewards rather than to avoid punishments, whereas adults learned to seek and avoid both equally. Increasing our understanding of computational changes in reinforcement learning during adolescence may provide insights into adolescent value-based decision-making. Our results might also have implications for education, since they suggest that adolescents benefit more from positive feedback than from negative feedback in learning tasks.
Collapse
Affiliation(s)
- Stefano Palminteri
- Institute of Cognitive Neuroscience, University College London, London, United Kingdom
- Laboratoire de Neurosciences Cognitive, École Normale Supérieure, Paris, France
- * E-mail:
| | - Emma J. Kilford
- Institute of Cognitive Neuroscience, University College London, London, United Kingdom
| | - Giorgio Coricelli
- Interdepartmental Centre for Mind/Brain Sciences, Università degli Studi di Trento, Trento, Italy
- Department of Economics, University of Southern California, Los Angeles, California, United States of America
| | - Sarah-Jayne Blakemore
- Institute of Cognitive Neuroscience, University College London, London, United Kingdom
| |
Collapse
|
24
|
Kamarajan C, Pandey AK, Chorlian DB, Manz N, Stimus AT, Anokhin AP, Bauer LO, Kuperman S, Kramer J, Bucholz KK, Schuckit MA, Hesselbrock VM, Porjesz B. Deficient Event-Related Theta Oscillations in Individuals at Risk for Alcoholism: A Study of Reward Processing and Impulsivity Features. PLoS One 2015; 10:e0142659. [PMID: 26580209 PMCID: PMC4651365 DOI: 10.1371/journal.pone.0142659] [Citation(s) in RCA: 21] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2015] [Accepted: 10/26/2015] [Indexed: 01/02/2023] Open
Abstract
BACKGROUND Individuals at high risk to develop alcoholism often manifest neurocognitive deficits as well as increased impulsivity. Event-related oscillations (EROs) have been used to effectively measure brain (dys)function during cognitive tasks in individuals with alcoholism and related disorders and in those at risk to develop these disorders. The current study examines ERO theta power during reward processing as well as impulsivity in adolescent and young adult subjects at high risk for alcoholism. METHODS EROs were recorded during a monetary gambling task (MGT) in 12-25 years old participants (N = 1821; males = 48%) from high risk alcoholic families (HR, N = 1534) and comparison low risk community families (LR, N = 287) from the Collaborative Study on the Genetics of Alcoholism (COGA). Impulsivity scores and prevalence of externalizing diagnoses were also compared between LR and HR groups. RESULTS HR offspring showed lower theta power and decreased current source density (CSD) activity than LR offspring during loss and gain conditions. Younger males had higher theta power than younger females in both groups, while the older HR females showed more theta power than older HR males. Younger subjects showed higher theta power than older subjects in each comparison. Differences in topography (i.e., frontalization) between groups were also observed. Further, HR subjects across gender had higher impulsivity scores and increased prevalence of externalizing disorders compared to LR subjects. CONCLUSIONS As theta power during reward processing is found to be lower not only in alcoholics, but also in HR subjects, it is proposed that reduced reward-related theta power, in addition to impulsivity and externalizing features, may be related in a predisposition to develop alcoholism and related disorders.
Collapse
Affiliation(s)
- Chella Kamarajan
- SUNY Downstate Medical Center, Brooklyn, NY, United States of America
| | - Ashwini K. Pandey
- SUNY Downstate Medical Center, Brooklyn, NY, United States of America
| | - David B. Chorlian
- SUNY Downstate Medical Center, Brooklyn, NY, United States of America
| | - Niklas Manz
- SUNY Downstate Medical Center, Brooklyn, NY, United States of America
| | - Arthur T. Stimus
- SUNY Downstate Medical Center, Brooklyn, NY, United States of America
| | - Andrey P. Anokhin
- Washington University School of Medicine, St. Louis, MO, United States of America
| | - Lance O. Bauer
- University of Connecticut Health Center, Farmington, CT, United States of America
| | | | - John Kramer
- University of Iowa, Iowa City, IA, United States of America
| | - Kathleen K. Bucholz
- Washington University School of Medicine, St. Louis, MO, United States of America
| | - Marc A. Schuckit
- University of California San Diego, San Diego, CA, United States of America
| | | | - Bernice Porjesz
- SUNY Downstate Medical Center, Brooklyn, NY, United States of America
| |
Collapse
|
25
|
Cognitive flexibility in adolescence: neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development. Neuroimage 2014; 104:347-54. [PMID: 25234119 PMCID: PMC4330550 DOI: 10.1016/j.neuroimage.2014.09.018] [Citation(s) in RCA: 101] [Impact Index Per Article: 10.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2014] [Revised: 09/05/2014] [Accepted: 09/06/2014] [Indexed: 01/24/2023] Open
Abstract
Adolescence is associated with quickly changing environmental demands which require excellent adaptive skills and high cognitive flexibility. Feedback-guided adaptive learning and cognitive flexibility are driven by reward prediction error (RPE) signals, which indicate the accuracy of expectations and can be estimated using computational models. Despite the importance of cognitive flexibility during adolescence, only little is known about how RPE processing in cognitive flexibility deviates between adolescence and adulthood. In this study, we investigated the developmental aspects of cognitive flexibility by means of computational models and functional magnetic resonance imaging (fMRI). We compared the neural and behavioral correlates of cognitive flexibility in healthy adolescents (12–16 years) to adults performing a probabilistic reversal learning task. Using a modified risk-sensitive reinforcement learning model, we found that adolescents learned faster from negative RPEs than adults. The fMRI analysis revealed that within the RPE network, the adolescents had a significantly altered RPE-response in the anterior insula. This effect seemed to be mainly driven by increased responses to negative prediction errors. In summary, our findings indicate that decision making in adolescence goes beyond merely increased reward-seeking behavior and provides a developmental perspective to the behavioral and neural mechanisms underlying cognitive flexibility in the context of reinforcement learning. Adolescents and adults show differences in processing RPEs. Adolescents learn faster from negative prediction errors. The anterior insula activation may cause altered sensitivity to RPEs.
Collapse
|