1
|
Kang P, Tobler PN, Dayan P. Bayesian reinforcement learning: A basic overview. Neurobiol Learn Mem 2024; 211:107924. [PMID: 38579896 DOI: 10.1016/j.nlm.2024.107924] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Revised: 03/21/2024] [Accepted: 04/02/2024] [Indexed: 04/07/2024]
Abstract
We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.
Collapse
Affiliation(s)
- Pyungwon Kang
- University of Zurich, Department of Economics, Laboratory for Social and Neural Systems Research, Zurich, Switzerland.
| | - Philippe N Tobler
- University of Zurich, Department of Economics, Laboratory for Social and Neural Systems Research, Zurich, Switzerland.
| | - Peter Dayan
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany; University of Tübingen, Tübingen Germany.
| |
Collapse
|
2
|
Paunov A, L'Hôtellier M, Guo D, He Z, Yu A, Meyniel F. Multiple and subject-specific roles of uncertainty in reward-guided decision-making. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.27.587016. [PMID: 38585958 PMCID: PMC10996615 DOI: 10.1101/2024.03.27.587016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/09/2024]
Abstract
Decision-making in noisy, changing, and partially observable environments entails a basic tradeoff between immediate reward and longer-term information gain, known as the exploration-exploitation dilemma. Computationally, an effective way to balance this tradeoff is by leveraging uncertainty to guide exploration. Yet, in humans, empirical findings are mixed, from suggesting uncertainty-seeking to indifference and avoidance. In a novel bandit task that better captures uncertainty-driven behavior, we find multiple roles for uncertainty in human choices. First, stable and psychologically meaningful individual differences in uncertainty preferences actually range from seeking to avoidance, which can manifest as null group-level effects. Second, uncertainty modulates the use of basic decision heuristics that imperfectly exploit immediate rewards: a repetition bias and win-stay-lose-shift heuristic. These heuristics interact with uncertainty, favoring heuristic choices under higher uncertainty. These results, highlighting the rich and varied structure of reward-based choice, are a step to understanding its functional basis and dysfunction in psychopathology.
Collapse
Affiliation(s)
- Alexander Paunov
- INSERM-CEA Cognitive Neuroimaging Unit (UNICOG), NeuroSpin Center, CEA Paris-Saclay, Gif-sur-Yvette, France Université de Paris, Paris, France
- Institut de Neuromodulation, GHU Paris, Psychiatrie et Neurosciences, Centre Hospitalier Sainte-Anne, Pôle Hospitalo-universitaire 15, Université Paris Cité, Paris, France
| | - Maëva L'Hôtellier
- INSERM-CEA Cognitive Neuroimaging Unit (UNICOG), NeuroSpin Center, CEA Paris-Saclay, Gif-sur-Yvette, France Université de Paris, Paris, France
| | - Dalin Guo
- Department of Cognitive Science, University of California San Diego, San Diego, CA, USA
| | - Zoe He
- Department of Cognitive Science, University of California San Diego, San Diego, CA, USA
| | - Angela Yu
- Department of Cognitive Science, University of California San Diego, San Diego, CA, USA
- Centre for Cognitive Science & Hessian AI Center, Technical University of Darmstadt, Germany
| | - Florent Meyniel
- INSERM-CEA Cognitive Neuroimaging Unit (UNICOG), NeuroSpin Center, CEA Paris-Saclay, Gif-sur-Yvette, France Université de Paris, Paris, France
- Institut de Neuromodulation, GHU Paris, Psychiatrie et Neurosciences, Centre Hospitalier Sainte-Anne, Pôle Hospitalo-universitaire 15, Université Paris Cité, Paris, France
| |
Collapse
|
3
|
Wyatt LE, Hewan PA, Hogeveen J, Spreng RN, Turner GR. Exploration versus exploitation decisions in the human brain: A systematic review of functional neuroimaging and neuropsychological studies. Neuropsychologia 2024; 192:108740. [PMID: 38036246 DOI: 10.1016/j.neuropsychologia.2023.108740] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2023] [Revised: 10/15/2023] [Accepted: 11/21/2023] [Indexed: 12/02/2023]
Abstract
Thoughts and actions are often driven by a decision to either explore new avenues with unknown outcomes, or to exploit known options with predictable outcomes. Yet, the neural mechanisms underlying this exploration-exploitation trade-off in humans remain poorly understood. This is attributable to variability in the operationalization of exploration and exploitation as psychological constructs, as well as the heterogeneity of experimental protocols and paradigms used to study these choice behaviours. To address this gap, here we present a comprehensive review of the literature to investigate the neural basis of explore-exploit decision-making in humans. We first conducted a systematic review of functional magnetic resonance imaging (fMRI) studies of exploration-versus exploitation-based decision-making in healthy adult humans during foraging, reinforcement learning, and information search. Eleven fMRI studies met inclusion criterion for this review. Adopting a network neuroscience framework, synthesis of the findings across these studies revealed that exploration-based choice was associated with the engagement of attentional, control, and salience networks. In contrast, exploitation-based choice was associated with engagement of default network brain regions. We interpret these results in the context of a network architecture that supports the flexible switching between externally and internally directed cognitive processes, necessary for adaptive, goal-directed behaviour. To further investigate potential neural mechanisms underlying the exploration-exploitation trade-off we next surveyed studies involving neurodevelopmental, neuropsychological, and neuropsychiatric disorders, as well as lifespan development, and neurodegenerative diseases. We observed striking differences in patterns of explore-exploit decision-making across these populations, again suggesting that these two decision-making modes are supported by independent neural circuits. Taken together, our review highlights the need for precision-mapping of the neural circuitry and behavioural correlates associated with exploration and exploitation in humans. Characterizing exploration versus exploitation decision-making biases may offer a novel, trans-diagnostic approach to assessment, surveillance, and intervention for cognitive decline and dysfunction in normal development and clinical populations.
Collapse
Affiliation(s)
- Lindsay E Wyatt
- Department of Psychology, York University, Toronto, ON, Canada
| | - Patrick A Hewan
- Department of Psychology, York University, Toronto, ON, Canada
| | - Jeremy Hogeveen
- Department of Psychology, The University of New Mexico, Albuquerque, NM, USA
| | - R Nathan Spreng
- Montréal Neurological Institute, Department of Neurology and Neurosurgery, McGill University, Montréal, QC, H3A 2B4, Canada; Department of Psychology, McGill University, Montréal, QC, Canada; Department of Psychiatry, McGill University, Montréal, QC, Canada; McConnell Brain Imaging Centre, Montréal Neurological Institute, McGill University, Montréal, QC, Canada.
| | - Gary R Turner
- Department of Psychology, York University, Toronto, ON, Canada.
| |
Collapse
|
4
|
Katabi G, Shahar N. Exploring the steps of learning: computational modeling of initiatory-actions among individuals with attention-deficit/hyperactivity disorder. Transl Psychiatry 2024; 14:10. [PMID: 38191535 PMCID: PMC10774270 DOI: 10.1038/s41398-023-02717-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/02/2023] [Revised: 12/13/2023] [Accepted: 12/14/2023] [Indexed: 01/10/2024] Open
Abstract
Attention-deficit/hyperactivity disorder (ADHD) is characterized by difficulty in acting in a goal-directed manner. While most environments require a sequence of actions for goal attainment, ADHD was never studied in the context of value-based sequence learning. Here, we made use of current advancements in hierarchical reinforcement-learning algorithms to track the internal value and choice policy of individuals with ADHD performing a three-stage sequence learning task. Specifically, 54 participants (28 ADHD, 26 controls) completed a value-based reinforcement-learning task that allowed us to estimate internal action values for each trial and stage using computational modeling. We found attenuated sensitivity to action values in ADHD compared to controls, both in choice and reaction-time variability estimates. Remarkably, this was found only for first-stage actions (i.e., initiatory actions), while for actions performed just before outcome delivery the two groups were strikingly indistinguishable. These results suggest a difficulty in following value estimation for initiatory actions in ADHD.
Collapse
Affiliation(s)
- Gili Katabi
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel.
| | - Nitzan Shahar
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
| |
Collapse
|
5
|
Pollak Y. Poor learning or hyper-exploration?: A commentary on Hulsbosch et al. (2023). J Child Psychol Psychiatry 2023; 64:1641-1642. [PMID: 37527827 DOI: 10.1111/jcpp.13875] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 07/15/2023] [Indexed: 08/03/2023]
Abstract
In an outstanding paper, Hulsbosch et al. (Journal of Child Psychology and Psychiatry, 2023) tested major reinforcement learning theories of attention-deficit/hyperactivity disorder (ADHD). On a reinforcement learning task, children with ADHD needed more trials to reach the criterion in the learning phase and chose the target less often in the extinction phase. These results only partially support either of the theories and may be interpreted as reflecting a general deficit in learning. Still, distinguishing between learning and decision-making stages suggests that poor performance on reinforcement learning tasks by children with ADHD is accounted for by increased exploration or decreased choice consistency rather than deficient learning.
Collapse
Affiliation(s)
- Yehuda Pollak
- The Seymour Fox School of Education, The Hebrew University of Jerusalem, Jerusalem, Israel
| |
Collapse
|
6
|
Daumas L, Zory R, Junquera-Badilla I, Ferrandez M, Ettore E, Robert P, Sacco G, Manera V, Ramanoël S. How does apathy impact exploration-exploitation decision-making in older patients with neurocognitive disorders? NPJ AGING 2023; 9:25. [PMID: 37903801 PMCID: PMC10616174 DOI: 10.1038/s41514-023-00121-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/18/2023] [Accepted: 09/14/2023] [Indexed: 11/01/2023]
Abstract
Apathy is a pervasive clinical syndrome in neurocognitive disorders, characterized by a quantitative reduction in goal-directed behaviors. The brain structures involved in the physiopathology of apathy have also been connected to the brain structures involved in probabilistic reward learning in the exploration-exploitation dilemma. This dilemma in question involves the challenge of selecting between a familiar option with a more predictable outcome, and another option whose outcome is uncertain and may yield potentially greater rewards compared to the known option. The aim of this study was to combine experimental procedures and computational modeling to examine whether, in older adults with mild neurocognitive disorders, apathy affects performance in the exploration-exploitation dilemma. Through using a four-armed bandit reinforcement-learning task, we showed that apathetic older adults explored more and performed worse than non-apathetic subjects. Moreover, the mental flexibility assessed by the Trail-making test-B was negatively associated with the percentage of exploration. These results suggest that apathy is characterized by an increased explorative behavior and inefficient decision-making, possibly due to weak mental flexibility to switch toward the exploitation of the more rewarding options. Apathetic participants also took longer to make a choice and failed more often to respond in the allotted time, which could reflect the difficulties in action initiation and selection. In conclusion, the present results suggest that apathy in participants with neurocognitive disorders is associated with specific disturbances in the exploration-exploitation trade-off and sheds light on the disturbances in reward processing in patients with apathy.
Collapse
Affiliation(s)
- Lyne Daumas
- Université Côte d'Azur, LAMHESS, Nice, France.
- Université Côte d'Azur, CoBTeK, Nice, France.
| | - Raphaël Zory
- Université Côte d'Azur, LAMHESS, Nice, France
- Institut Universitaire de France, Paris, France
| | | | - Marion Ferrandez
- Université Côte d'Azur, CoBTeK, Nice, France
- Université Côte d'Azur, Centre Hospitalier Universitaire de Nice, service Clinique Gériatrique de Soins Ambulatoires, Centre Mémoire de Ressources et de Recherche, Nice, France
| | - Eric Ettore
- Université Côte d'Azur, CoBTeK, Nice, France
- Université Côte d'Azur, Centre Hospitalier Universitaire de Nice, service Clinique Gériatrique de Soins Ambulatoires, Centre Mémoire de Ressources et de Recherche, Nice, France
- Association Innovation Alzheimer, Nice, France
| | - Philippe Robert
- Université Côte d'Azur, CoBTeK, Nice, France
- Association Innovation Alzheimer, Nice, France
| | - Guillaume Sacco
- Université Côte d'Azur, CoBTeK, Nice, France
- Université Côte d'Azur, Centre Hospitalier Universitaire de Nice, service Clinique Gériatrique de Soins Ambulatoires, Centre Mémoire de Ressources et de Recherche, Nice, France
- Association Innovation Alzheimer, Nice, France
- Univ Angers, Université de Nantes, LPPL, SFR CONFLUENCES, 49000, Angers, France
| | - Valeria Manera
- Université Côte d'Azur, CoBTeK, Nice, France
- Association Innovation Alzheimer, Nice, France
| | - Stephen Ramanoël
- Université Côte d'Azur, LAMHESS, Nice, France
- Sorbonne Université, INSERM, CNRS, Institut de la Vision, 17 rue Moreau, 75012, Paris, France
| |
Collapse
|
7
|
Chen CS, Mueller D, Knep E, Ebitz RB, Grissom NM. Dopamine and norepinephrine differentially mediate the exploration-exploitation tradeoff. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.01.09.523322. [PMID: 36711959 PMCID: PMC9881999 DOI: 10.1101/2023.01.09.523322] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/11/2023]
Abstract
The catecholamines dopamine (DA) and norepinephrine (NE) have been repeatedly implicated in neuropsychiatric vulnerability, in part via their roles in mediating the decision making processes. Although the two neuromodulators share a synthesis pathway and are co-activated under states of arousal, they engage in distinct circuits and roles in modulating neural activity across the brain. However, in the computational neuroscience literature, they have been assigned similar roles in modulating the latent cognitive processes of decision making, in particular the exploration-exploitation tradeoff. Revealing how each neuromodulator contributes to this explore-exploit process will be important in guiding mechanistic hypotheses emerging from computational psychiatric approaches. To understand the differences and overlaps of the roles of these two catecholamine systems in regulating exploration and exploitation, a direct comparison using the same dynamic decision making task is needed. Here, we ran mice in a restless two-armed bandit task, which encourages both exploration and exploitation. We systemically administered a nonselective DA receptor antagonist (flupenthixol), a nonselective DA receptor agonist (apomorphine), a NE beta-receptor antagonist (propranolol), and a NE beta-receptor agonist (isoproterenol), and examined changes in exploration within subjects across sessions. We found a bidirectional modulatory effect of dopamine receptor activity on the level of exploration. Increasing dopamine activity decreased exploration and decreasing dopamine activity increased exploration. Beta-noradrenergic receptor activity also modulated exploration, but the modulatory effect was mediated by sex. Reinforcement learning model parameters suggested that dopamine modulation affected exploration via decision noise and norepinephrine modulation affected exploration via outcome sensitivity. Together, these findings suggested that the mechanisms that govern the transition between exploration and exploitation are sensitive to changes in both catecholamine functions and revealed differential roles for NE and DA in mediating exploration.
Collapse
|
8
|
Baribault B, Collins AGE. Troubleshooting Bayesian cognitive models. Psychol Methods 2023:2023-57852-001. [PMID: 36972080 PMCID: PMC10522800 DOI: 10.1037/met0000554] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/29/2023]
Abstract
Using Bayesian methods to apply computational models of cognitive processes, or Bayesian cognitive modeling, is an important new trend in psychological research. The rise of Bayesian cognitive modeling has been accelerated by the introduction of software that efficiently automates the Markov chain Monte Carlo sampling used for Bayesian model fitting-including the popular Stan and PyMC packages, which automate the dynamic Hamiltonian Monte Carlo and No-U-Turn Sampler (HMC/NUTS) algorithms that we spotlight here. Unfortunately, Bayesian cognitive models can struggle to pass the growing number of diagnostic checks required of Bayesian models. If any failures are left undetected, inferences about cognition based on the model's output may be biased or incorrect. As such, Bayesian cognitive models almost always require troubleshooting before being used for inference. Here, we present a deep treatment of the diagnostic checks and procedures that are critical for effective troubleshooting, but are often left underspecified by tutorial papers. After a conceptual introduction to Bayesian cognitive modeling and HMC/NUTS sampling, we outline the diagnostic metrics, procedures, and plots necessary to detect problems in model output with an emphasis on how these requirements have recently been changed and extended. Throughout, we explain how uncovering the exact nature of the problem is often the key to identifying solutions. We also demonstrate the troubleshooting process for an example hierarchical Bayesian model of reinforcement learning, including supplementary code. With this comprehensive guide to techniques for detecting, identifying, and overcoming problems in fitting Bayesian cognitive models, psychologists across subfields can more confidently build and use Bayesian cognitive models in their research. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Collapse
|
9
|
van den Bosch R, Lambregts B, Määttä J, Hofmans L, Papadopetraki D, Westbrook A, Verkes RJ, Booij J, Cools R. Striatal dopamine dissociates methylphenidate effects on value-based versus surprise-based reversal learning. Nat Commun 2022; 13:4962. [PMID: 36002446 PMCID: PMC9402573 DOI: 10.1038/s41467-022-32679-1] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2021] [Accepted: 08/10/2022] [Indexed: 11/13/2022] Open
Abstract
Psychostimulants such as methylphenidate are widely used for their cognitive enhancing effects, but there is large variability in the direction and extent of these effects. We tested the hypothesis that methylphenidate enhances or impairs reward/punishment-based reversal learning depending on baseline striatal dopamine levels and corticostriatal gating of reward/punishment-related representations in stimulus-specific sensory cortex. Young healthy adults (N = 100) were scanned with functional magnetic resonance imaging during a reward/punishment reversal learning task, after intake of methylphenidate or the selective D2/3-receptor antagonist sulpiride. Striatal dopamine synthesis capacity was indexed with [18F]DOPA positron emission tomography. Methylphenidate improved and sulpiride decreased overall accuracy and response speed. Both drugs boosted reward versus punishment learning signals to a greater degree in participants with higher dopamine synthesis capacity. By contrast, striatal and stimulus-specific sensory surprise signals were boosted in participants with lower dopamine synthesis. These results unravel the mechanisms by which methylphenidate gates both attention and reward learning. The mechanisms underpinning the variability in methylphenidate’s effects on cognition remain unclear. Here, the authors show that such effects reflect changes in striatal dopamine-related output gating of task-relevant cortical signals, and that these changes depend on baseline dopamine synthesis capacity.
Collapse
Affiliation(s)
- Ruben van den Bosch
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.
| | - Britt Lambregts
- Radboud University Medical Center, Department of Psychiatry, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - Jessica Määttä
- Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden
| | - Lieke Hofmans
- Department of Developmental Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - Danae Papadopetraki
- Radboud University Medical Center, Department of Psychiatry, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - Andrew Westbrook
- Cognitive, Linguistic & Psychological Sciences Department, Brown University, Providence, RI, USA
| | - Robbert-Jan Verkes
- Radboud University Medical Center, Department of Psychiatry, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - Jan Booij
- Department of Radiology and Nuclear Medicine, Amsterdam University Medical Centers, location Academic Medical Center, Amsterdam, The Netherlands.,Radboud University Medical Center, Department of Medical Imaging, Nijmegen, The Netherlands
| | - Roshan Cools
- Radboud University Medical Center, Department of Psychiatry, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| |
Collapse
|
10
|
Tei S, Tanicha M, Itahashi T, Aoki YY, Ohta H, Qian C, Hashimoto RI, Nakamura M, Takahashi H, Kato N, Fujino J. Decision Flexibilities in Autism Spectrum Disorder: An fMRI Study of Moral Dilemmas. Soc Cogn Affect Neurosci 2022; 17:904-911. [PMID: 35333369 PMCID: PMC9527470 DOI: 10.1093/scan/nsac023] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2021] [Revised: 02/07/2022] [Accepted: 03/24/2022] [Indexed: 11/21/2022] Open
Abstract
People make flexible decisions across a wide range of contexts to resolve social or moral conflicts. Individuals with autism spectrum disorder (ASD) frequently report difficulties in such behaviors, which hinders the flexibility in changing strategies during daily activities or adjustment of perspective during communication. However, the underlying mechanisms of this issue are insufficiently understood. This study aimed to investigate decision flexibility in ASD using a functional magnetic resonance imaging task that involved recognizing and resolving two types of moral dilemmas: cost–benefit analysis (CBA) and mitigating inevitable misconducts (MIM). The CBA session assessed the participants’ pitting of result-oriented outcomes against distressful harmful actions, whereas the MIM session assessed their pitting of the extenuation of a criminal sentence against a sympathetic situation of defendants suffering from violence or disease. The behavioral outcome in CBA-related flexibility was significantly lower in the ASD group compared to that of the typical development group. In the corresponding CBA contrast, activation in the left inferior frontal gyrus was lower in the ASD group. Meanwhile, in the MIM-related flexibility, there were no significant group differences in behavioral outcome or brain activity. Our findings add to our understanding of flexible decision-making in ASD.
Collapse
Affiliation(s)
- Shisei Tei
- Medical Institute of Developmental Disabilities Research, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan.,Department of Psychiatry, Graduate School of Medicine, Kyoto University, 54 Shogoin-Kawaracho, Sakyo-ku, Kyoto, Japan.,Institute of Applied Brain Sciences, Waseda University, 2-579-15 Mikajima, Tokorozawa, Saitama, Japan.,School of Human and Social Sciences, Tokyo International University, 2509 Matoba, Kawagoe, Saitama, Japan
| | - Mizuki Tanicha
- Department of Psychiatry and Behavioral Sciences, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, 1-5-45 Yushima, Bunkyo-ku, Tokyo, Japan
| | - Takashi Itahashi
- Medical Institute of Developmental Disabilities Research, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan
| | - Yuta Y Aoki
- Medical Institute of Developmental Disabilities Research, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan
| | - Haruhisa Ohta
- Medical Institute of Developmental Disabilities Research, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan.,Department of Psychiatry, School of Medicine, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan
| | - Chenyu Qian
- Department of Psychiatry and Behavioral Sciences, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, 1-5-45 Yushima, Bunkyo-ku, Tokyo, Japan
| | - Ryu-Ichiro Hashimoto
- Medical Institute of Developmental Disabilities Research, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan.,Department of Language Sciences, Graduate School of Humanities, Tokyo Metropolitan University, 1-1 Minami-Osawa, Hachioji-shi, Tokyo, Japan
| | - Motoaki Nakamura
- Medical Institute of Developmental Disabilities Research, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan.,Kanagawa Psychiatric Center, 2-5-1 Serigaya, Yokohama, Kanagawa, Japan
| | - Hidehiko Takahashi
- Medical Institute of Developmental Disabilities Research, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan.,Department of Psychiatry, Graduate School of Medicine, Kyoto University, 54 Shogoin-Kawaracho, Sakyo-ku, Kyoto, Japan.,Department of Psychiatry and Behavioral Sciences, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, 1-5-45 Yushima, Bunkyo-ku, Tokyo, Japan
| | - Nobumasa Kato
- Medical Institute of Developmental Disabilities Research, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan
| | - Junya Fujino
- Medical Institute of Developmental Disabilities Research, Showa University, 6-11-11 Kita-karasuyama, Setagaya-ku, Tokyo, Japan.,Department of Psychiatry, Graduate School of Medicine, Kyoto University, 54 Shogoin-Kawaracho, Sakyo-ku, Kyoto, Japan.,Department of Psychiatry and Behavioral Sciences, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, 1-5-45 Yushima, Bunkyo-ku, Tokyo, Japan
| |
Collapse
|
11
|
Exploration heuristics decrease during youth. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2022; 22:969-983. [PMID: 35589910 PMCID: PMC9458685 DOI: 10.3758/s13415-022-01009-9] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Accepted: 04/22/2022] [Indexed: 01/01/2023]
Abstract
Deciding between exploring new avenues and exploiting known choices is central to learning, and this exploration-exploitation trade-off changes during development. Exploration is not a unitary concept, and humans deploy multiple distinct mechanisms, but little is known about their specific emergence during development. Using a previously validated task in adults, changes in exploration mechanisms were investigated between childhood (8-9 y/o, N = 26; 16 females), early (12-13 y/o, N = 38; 21 females), and late adolescence (16-17 y/o, N = 33; 19 females) in ethnically and socially diverse schools from disadvantaged areas. We find an increased usage of a computationally light exploration heuristic in younger groups, effectively accommodating their limited neurocognitive resources. Moreover, this heuristic was associated with self-reported, attention-deficit/hyperactivity disorder symptoms in this population-based sample. This study enriches our mechanistic understanding about how exploration strategies mature during development.
Collapse
|
12
|
Chen CS, Knep E, Han A, Ebitz RB, Grissom N. Sex differences in learning from exploration. eLife 2021; 10:69748. [PMID: 34796870 PMCID: PMC8794469 DOI: 10.7554/elife.69748] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2021] [Accepted: 11/18/2021] [Indexed: 11/13/2022] Open
Abstract
Sex-based modulation of cognitive processes could set the stage for individual differences in vulnerability to neuropsychiatric disorders. While value-based decision making processes in particular have been proposed to be influenced by sex differences, the overall correct performance in decision making tasks often show variable or minimal differences across sexes. Computational tools allow us to uncover latent variables that define different decision making approaches, even in animals with similar correct performance. Here, we quantify sex differences in mice in the latent variables underlying behavior in a classic value-based decision making task: a restless 2-armed bandit. While male and female mice had similar accuracy, they achieved this performance via different patterns of exploration. Male mice tended to make more exploratory choices overall, largely because they appeared to get 'stuck' in exploration once they had started. Female mice tended to explore less but learned more quickly during exploration. Together, these results suggest that sex exerts stronger influences on decision making during periods of learning and exploration than during stable choices. Exploration during decision making is altered in people diagnosed with addictions, depression, and neurodevelopmental disabilities, pinpointing the neural mechanisms of exploration as a highly translational avenue for conferring sex-modulated vulnerability to neuropsychiatric diagnoses.
Collapse
Affiliation(s)
- Cathy S Chen
- University of Minnesota, Minneapolis, United States
| | - Evan Knep
- University of Minnesota, Minneapolis, United States
| | - Autumn Han
- University of Minnesota, Minneapolis, United States
| | - R Becket Ebitz
- Department of Neurosciences, Princeton University, Princeton, United States
| | | |
Collapse
|