1
|
Brown CS, Devine S, Otto AR, Bischoff-Grethe A, Wierenga CE. Greater reliance on model-free learning in adolescent anorexia nervosa: An examination of dual-system reinforcement learning. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.01.31.24302097. [PMID: 38352608 PMCID: PMC10863009 DOI: 10.1101/2024.01.31.24302097] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/19/2024]
Abstract
Alterations in learning and decision-making systems are thought to contribute to core features of anorexia nervosa (AN), a psychiatric disorder characterized by persistent dietary restriction and weight loss. Instrumental learning theory identifies a dual-system of habit and goal-directed decision-making, linked to model-free and model-based reinforcement learning algorithms. Difficulty arbitrating between these systems, resulting in an over-reliance on one strategy over the other, has been implicated in compulsivity and extreme goal pursuit, both of which are observed in AN. Characterizing alterations in model-free and model-based systems, and their neural correlates, in AN may clarify mechanisms contributing to symptom heterogeneity (e.g., binge/purge symptoms). This study tested whether adolescents with restricting AN (AN-R; n = 36) and binge/purge AN (AN-BP; n = 20) differentially utilized model-based and model-free learning systems compared to a healthy control group (HC; n = 28) during a Markov two-step decision-making task under conditions of reward and punishment. Associations between model-free and model-based learning and resting-state functional connectivity between neural regions of interest, including orbitofrontal cortex (OFC), nucleus accumbens (NAcc), putamen, and sensory motor cortex (SMC) were examined. AN-R showed higher utilization of model-free learning compared to HC for reward, but attenuated model-free and model-based learning for punishment. In AN-R only, higher model-based learning was associated with stronger OFC-to-left NAcc functional connectivity, regions linked to goal-directed behavior. Greater utilization of model-free learning for reward in AN-R may differentiate this group, particularly during adolescence, and facilitate dietary restriction by prioritizing habitual control in rewarding contexts.
Collapse
Affiliation(s)
- Carina S. Brown
- San Diego State University/University of California San Diego Joint Doctoral Program in Clinical Psychology
- Department of Psychiatry, University of California, San Diego
| | | | | | | | - Christina E. Wierenga
- San Diego State University/University of California San Diego Joint Doctoral Program in Clinical Psychology
- Department of Psychiatry, University of California, San Diego
| |
Collapse
|
2
|
Pezzoli P, Parsons S, Kievit RA, Astle DE, Huys QJM, Steinbeis N, Viding E. Challenges and Solutions to the Measurement of Neurocognitive Mechanisms in Developmental Settings. BIOLOGICAL PSYCHIATRY. COGNITIVE NEUROSCIENCE AND NEUROIMAGING 2023; 8:815-821. [PMID: 37003410 DOI: 10.1016/j.bpsc.2023.03.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 03/15/2023] [Accepted: 03/20/2023] [Indexed: 03/31/2023]
Abstract
Identifying early neurocognitive mechanisms that confer risk for mental health problems is one important avenue as we seek to develop successful early interventions. Currently, however, we have limited understanding of the neurocognitive mechanisms involved in shaping mental health trajectories from childhood through young adulthood, and this constrains our ability to develop effective clinical interventions. In particular, there is an urgent need to develop more sensitive, reliable, and scalable measures of individual differences for use in developmental settings. In this review, we outline methodological shortcomings that explain why widely used task-based measures of neurocognition currently tell us little about mental health risk. We discuss specific challenges that arise when studying neurocognitive mechanisms in developmental settings, and we share suggestions for overcoming them. We also propose a novel experimental approach-which we refer to as "cognitive microscopy"-that involves adaptive design optimization, temporally sensitive task administration, and multilevel modeling. This approach addresses some of the methodological shortcomings outlined above and provides measures of stability, variability, and developmental change in neurocognitive mechanisms within a multivariate framework.
Collapse
Affiliation(s)
- Patrizia Pezzoli
- Division of Psychology and Language Sciences, University College London, London, United Kingdom.
| | - Sam Parsons
- Donders Institute for Brain, Cognition and Behavior, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Rogier A Kievit
- Donders Institute for Brain, Cognition and Behavior, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Duncan E Astle
- Medical Research Council Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, United Kingdom; Department of Psychiatry, University of Cambridge, Cambridge, United Kingdom
| | - Quentin J M Huys
- Applied Computational Psychiatry Laboratory, Mental Health Neuroscience Department, Division of Psychiatry and Max Planck Centre for Computational Psychiatry and Ageing Research, Queen Square Institute of Neurology, University College London, London, United Kingdom
| | - Nikolaus Steinbeis
- Division of Psychology and Language Sciences, University College London, London, United Kingdom
| | - Essi Viding
- Division of Psychology and Language Sciences, University College London, London, United Kingdom.
| |
Collapse
|
3
|
Goldway N, Eldar E, Shoval G, Hartley CA. Computational Mechanisms of Addiction and Anxiety: A Developmental Perspective. Biol Psychiatry 2023; 93:739-750. [PMID: 36775050 PMCID: PMC10038924 DOI: 10.1016/j.biopsych.2023.02.004] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/20/2022] [Revised: 02/05/2023] [Accepted: 02/07/2023] [Indexed: 02/12/2023]
Abstract
A central goal of computational psychiatry is to identify systematic relationships between transdiagnostic dimensions of psychiatric symptomatology and the latent learning and decision-making computations that inform individuals' thoughts, feelings, and choices. Most psychiatric disorders emerge prior to adulthood, yet little work has extended these computational approaches to study the development of psychopathology. Here, we lay out a roadmap for future studies implementing this approach by developing empirically and theoretically informed hypotheses about how developmental changes in model-based control of action and Pavlovian learning processes may modulate vulnerability to anxiety and addiction. We highlight how insights from studies leveraging computational approaches to characterize the normative developmental trajectories of clinically relevant learning and decision-making processes may suggest promising avenues for future developmental computational psychiatry research.
Collapse
Affiliation(s)
- Noam Goldway
- Department of Psychology, New York University, New York, New York
| | - Eran Eldar
- Department of Psychology, The Hebrew University of Jerusalem, Jerusalem, Israel; Department of Cognitive and Brain Sciences, The Hebrew University of Jerusalem, Jerusalem, Israel
| | - Gal Shoval
- Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey; Child and Adolescent Division, Geha Mental Health Center, Petah Tikva, Israel; Department of Psychiatry, Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
| | - Catherine A Hartley
- Department of Psychology, New York University, New York, New York; Center for Neural Science, New York University, New York, New York.
| |
Collapse
|
4
|
Waltmann M, Herzog N, Reiter AMF, Villringer A, Horstmann A, Deserno L. Diminished reinforcement sensitivity in adolescence is associated with enhanced response switching and reduced coding of choice probability in the medial frontal pole. Dev Cogn Neurosci 2023; 60:101226. [PMID: 36905874 PMCID: PMC10005907 DOI: 10.1016/j.dcn.2023.101226] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Revised: 03/06/2023] [Accepted: 03/06/2023] [Indexed: 03/09/2023] Open
Abstract
Precisely charting the maturation of core neurocognitive functions such as reinforcement learning (RL) and flexible adaptation to changing action-outcome contingencies is key for developmental neuroscience and adjacent fields like developmental psychiatry. However, research in this area is both sparse and conflicted, especially regarding potentially asymmetric development of learning for different motives (obtain wins vs avoid losses) and learning from valenced feedback (positive vs negative). In the current study, we investigated the development of RL from adolescence to adulthood, using a probabilistic reversal learning task modified to experimentally separate motivational context and feedback valence, in a sample of 95 healthy participants between 12 and 45. We show that adolescence is characterized by enhanced novelty seeking and response shifting especially after negative feedback, which leads to poorer returns when reward contingencies are stable. Computationally, this is accounted for by reduced impact of positive feedback on behavior. We also show, using fMRI, that activity of the medial frontopolar cortex reflecting choice probability is attenuated in adolescence. We argue that this can be interpreted as reflecting diminished confidence in upcoming choices. Interestingly, we find no age-related differences between learning in win and loss contexts.
Collapse
Affiliation(s)
- Maria Waltmann
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Centre of Mental Health, University of Würzburg, Würzburg, Germany; Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
| | - Nadine Herzog
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Andrea M F Reiter
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Centre of Mental Health, University of Würzburg, Würzburg, Germany; CRC-940 Volition and Cognitive Control, Faculty of Psychology, Technical University of Dresden, Dresden, Germany
| | - Arno Villringer
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; MindBrainBody Institute, Berlin School of Mind and Brain, Charité-Universitätsmedizin Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Annette Horstmann
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; Department of Psychology and Logopedics, Faculty of Medicine, University of Helsinki, Helsinki, Finland
| | - Lorenz Deserno
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Centre of Mental Health, University of Würzburg, Würzburg, Germany; Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; Neuroimaging Center, Technical University of Dresden, Dresden, Germany
| |
Collapse
|
5
|
Smid CR, Kool W, Hauser TU, Steinbeis N. Computational and behavioral markers of model-based decision making in childhood. Dev Sci 2023; 26:e13295. [PMID: 35689563 DOI: 10.1111/desc.13295] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2021] [Revised: 05/12/2022] [Accepted: 05/22/2022] [Indexed: 11/28/2022]
Abstract
Human decision-making is underpinned by distinct systems that differ in flexibility and associated cognitive cost. A widely accepted dichotomy distinguishes between a cheap but rigid model-free system and a flexible but costly model-based system. Typically, humans use a hybrid of both types of decision-making depending on environmental demands. However, children's use of a model-based system during decision-making has not yet been shown. While prior developmental work has identified simple building blocks of model-based reasoning in young children (1-4 years old), there has been little evidence of this complex cognitive system influencing behavior before adolescence. Here, by using a modified task to make engagement in cognitively costly strategies more rewarding, we show that children aged 5-11-years (N = 85), including the youngest children, displayed multiple indicators of model-based decision making, and that the degree of its use increased throughout childhood. Unlike adults (N = 24), however, children did not display adaptive arbitration between model-free and model-based decision-making. Our results demonstrate that throughout childhood, children can engage in highly sophisticated and costly decision-making strategies. However, the flexible arbitration between decision-making strategies might be a critically late-developing component in human development.
Collapse
Affiliation(s)
- Claire R Smid
- Department of Psychology and Language Sciences, University College London, London, the United Kingdom
| | - Wouter Kool
- Department of Psychological & Brain Sciences, Washington University in St. Louis, St. Louis, Missouri, USA
| | - Tobias U Hauser
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, London, the United Kingdom.,Wellcome Centre for Human Neuroimaging, University College London, London, the United Kingdom
| | - Nikolaus Steinbeis
- Department of Psychology and Language Sciences, University College London, London, the United Kingdom
| |
Collapse
|
6
|
Nussenbaum K, Velez JA, Washington BT, Hamling HE, Hartley CA. Flexibility in valenced reinforcement learning computations across development. Child Dev 2022; 93:1601-1615. [PMID: 35596654 PMCID: PMC9831067 DOI: 10.1111/cdev.13791] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/12/2023]
Abstract
Optimal integration of positive and negative outcomes during learning varies depending on an environment's reward statistics. The present study investigated the extent to which children, adolescents, and adults (N = 142 8-25 year-olds, 55% female, 42% White, 31% Asian, 17% mixed race, and 8% Black; data collected in 2021) adapt their weighting of better-than-expected and worse-than-expected outcomes when learning from reinforcement. Participants made choices across two contexts: one in which weighting positive outcomes more heavily than negative outcomes led to better performance, and one in which the reverse was true. Reinforcement learning modeling revealed that across age, participants shifted their valence biases in accordance with environmental structure. Exploratory analyses revealed strengthening of context-dependent flexibility with increasing age.
Collapse
Affiliation(s)
| | | | | | | | - Catherine A. Hartley
- Corresponding Author: Catherine A. Hartley, Department of Psychology, New York University, 6 Washington Place, Room 871A, New York, NY, 10003.
| |
Collapse
|
7
|
Bolenz F, Profitt MF, Stechbarth F, Eppinger B, Strobel A. Need for cognition does not account for individual differences in metacontrol of decision making. Sci Rep 2022; 12:8240. [PMID: 35581395 PMCID: PMC9114337 DOI: 10.1038/s41598-022-12341-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2022] [Accepted: 05/09/2022] [Indexed: 12/05/2022] Open
Abstract
Humans show metacontrol of decision making, that is they adapt their reliance on decision-making strategies toward situational differences such as differences in reward magnitude. Specifically, when higher rewards are at stake, individuals increase reliance on a more accurate but cognitively effortful strategy. We investigated whether the personality trait Need for Cognition (NFC) explains individual differences in metacontrol. Based on findings of cognitive effort expenditure in executive functions, we expected more metacontrol in individuals low in NFC. In two independent studies, metacontrol was assessed by means of a decision-making task that dissociates different reinforcement-learning strategies and in which reward magnitude was manipulated across trials. In contrast to our expectations, NFC did not account for individual differences in metacontrol of decision making. In fact, a Bayesian analysis provided moderate to strong evidence against a relationship between NFC and metacontrol. Beyond this, there was no consistent evidence for relationship between NFC and overall model-based decision making. These findings show that the effect of rewards on the engagement of effortful decision-making strategies is largely independent of the intrinsic motivation for engaging in cognitively effortful tasks and suggest a differential role of NFC for the regulation of cognitive effort in decision making and executive functions.
Collapse
Affiliation(s)
- Florian Bolenz
- Faculty of Psychology, Technische Universität Dresden, Dresden, Germany. .,Max Planck Institute for Human Development, Lentzeallee 94, 14195, Berlin, Germany. .,Cluster of Excellence "Science of Intelligence", Technische Universität Berlin, Berlin, Germany.
| | - Maxine F Profitt
- Department of Psychology, Concordia University, Montreal, Canada
| | - Fabian Stechbarth
- Faculty of Psychology, Technische Universität Dresden, Dresden, Germany
| | - Ben Eppinger
- Faculty of Psychology, Technische Universität Dresden, Dresden, Germany.,Department of Psychology, Concordia University, Montreal, Canada.,PERFORM Centre, Concordia University, Montreal, Canada
| | - Alexander Strobel
- Faculty of Psychology, Technische Universität Dresden, Dresden, Germany
| |
Collapse
|