1
|
Ngetich R, Villalba-García C, Soborun Y, Vékony T, Czakó A, Demetrovics Z, Németh D. Learning and memory processes in behavioural addiction: A systematic review. Neurosci Biobehav Rev 2024; 163:105747. [PMID: 38870547 DOI: 10.1016/j.neubiorev.2024.105747] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2024] [Revised: 05/28/2024] [Accepted: 06/01/2024] [Indexed: 06/15/2024]
Abstract
Similar to addictive substances, addictive behaviours such as gambling and gaming are associated with maladaptive modulation of key brain areas and functional networks implicated in learning and memory. Therefore, this review sought to understand how different learning and memory processes relate to behavioural addictions and to unravel their underlying neural mechanisms. Adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we systematically searched four databases - PsycINFO, PubMed, Scopus, and Web of Science using the agreed-upon search string. Findings suggest altered executive function-dependent learning processes and enhanced habit learning in behavioural addiction. Whereas the relationship between working memory and behavioural addiction is influenced by addiction type, working memory aspect, and task nature. Additionally, long-term memory is incoherent in individuals with addictive behaviours. Consistently, neurophysiological evidence indicates alterations in brain areas and networks implicated in learning and memory processes in behavioural addictions. Overall, the present review argues that, like substance use disorders, alteration in learning and memory processes may underlie the development and maintenance of behavioural addictions.
Collapse
Affiliation(s)
- Ronald Ngetich
- Centre of Excellence in Responsible Gaming, University of Gibraltar, Gibraltar, Gibraltar
| | | | - Yanisha Soborun
- Centre of Excellence in Responsible Gaming, University of Gibraltar, Gibraltar, Gibraltar
| | - Teodóra Vékony
- Centre de Recherche en Neurosciences de Lyon CRNL U1028 UMR5292, INSERM, CNRS, Université Claude Bernard Lyon 1, Bron, France; Department of Education and Psychology, Faculty of Social Sciences, University of Atlántico Medio, Las Palmas de Gran Canaria, Spain
| | - Andrea Czakó
- Centre of Excellence in Responsible Gaming, University of Gibraltar, Gibraltar, Gibraltar; Institute of Psychology, ELTE Eötvös Loránd University, Budapest, Hungary
| | - Zsolt Demetrovics
- Centre of Excellence in Responsible Gaming, University of Gibraltar, Gibraltar, Gibraltar; Institute of Psychology, ELTE Eötvös Loránd University, Budapest, Hungary; College of Education, Psychology and Social Work, Flinders University, Adelaide, Australia.
| | - Dezső Németh
- Centre de Recherche en Neurosciences de Lyon CRNL U1028 UMR5292, INSERM, CNRS, Université Claude Bernard Lyon 1, Bron, France; Department of Education and Psychology, Faculty of Social Sciences, University of Atlántico Medio, Las Palmas de Gran Canaria, Spain; BML-NAP Research Group, Institute of Psychology, Eötvös Loránd University & Institute of Cognitive Neuroscience and Psychology, HUN-REN Research Centre for Natural Sciences, Budapest, Hungary
| |
Collapse
|
2
|
Ramírez-Ruiz J, Grytskyy D, Mastrogiuseppe C, Habib Y, Moreno-Bote R. Complex behavior from intrinsic motivation to occupy future action-state path space. Nat Commun 2024; 15:6368. [PMID: 39075046 PMCID: PMC11286966 DOI: 10.1038/s41467-024-49711-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2023] [Accepted: 06/13/2024] [Indexed: 07/31/2024] Open
Abstract
Most theories of behavior posit that agents tend to maximize some form of reward or utility. However, animals very often move with curiosity and seem to be motivated in a reward-free manner. Here we abandon the idea of reward maximization and propose that the goal of behavior is maximizing occupancy of future paths of actions and states. According to this maximum occupancy principle, rewards are the means to occupy path space, not the goal per se; goal-directedness simply emerges as rational ways of searching for resources so that movement, understood amply, never ends. We find that action-state path entropy is the only measure consistent with additivity and other intuitive properties of expected future action-state path occupancy. We provide analytical expressions that relate the optimal policy and state-value function and prove convergence of our value iteration algorithm. Using discrete and continuous state tasks, including a high-dimensional controller, we show that complex behaviors such as "dancing", hide-and-seek, and a basic form of altruistic behavior naturally result from the intrinsic motivation to occupy path space. All in all, we present a theory of behavior that generates both variability and goal-directedness in the absence of reward maximization.
Collapse
Affiliation(s)
- Jorge Ramírez-Ruiz
- Center for Brain and Cognition, Departament d'Enginyeria i Escola d'Enginyeria, Universitat Pompeu Fabra, Barcelona, Spain.
| | - Dmytro Grytskyy
- Center for Brain and Cognition, Departament d'Enginyeria i Escola d'Enginyeria, Universitat Pompeu Fabra, Barcelona, Spain
| | - Chiara Mastrogiuseppe
- Center for Brain and Cognition, Departament d'Enginyeria i Escola d'Enginyeria, Universitat Pompeu Fabra, Barcelona, Spain
| | - Yamen Habib
- Center for Brain and Cognition, Departament d'Enginyeria i Escola d'Enginyeria, Universitat Pompeu Fabra, Barcelona, Spain
| | - Rubén Moreno-Bote
- Center for Brain and Cognition, Departament d'Enginyeria i Escola d'Enginyeria, Universitat Pompeu Fabra, Barcelona, Spain
- Serra Húnter Fellow Programme, Universitat Pompeu Fabra, Barcelona, Spain
| |
Collapse
|
3
|
Zhang R, Pitkow X, Angelaki DE. Inductive biases of neural network modularity in spatial navigation. SCIENCE ADVANCES 2024; 10:eadk1256. [PMID: 39028809 PMCID: PMC11259174 DOI: 10.1126/sciadv.adk1256] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/03/2023] [Accepted: 06/14/2024] [Indexed: 07/21/2024]
Abstract
The brain may have evolved a modular architecture for daily tasks, with circuits featuring functionally specialized modules that match the task structure. We hypothesize that this architecture enables better learning and generalization than architectures with less specialized modules. To test this, we trained reinforcement learning agents with various neural architectures on a naturalistic navigation task. We found that the modular agent, with an architecture that segregates computations of state representation, value, and action into specialized modules, achieved better learning and generalization. Its learned state representation combines prediction and observation, weighted by their relative uncertainty, akin to recursive Bayesian estimation. This agent's behavior also resembles macaques' behavior more closely. Our results shed light on the possible rationale for the brain's modularity and suggest that artificial systems can use this insight from neuroscience to improve learning and generalization in natural tasks.
Collapse
Affiliation(s)
- Ruiyi Zhang
- Tandon School of Engineering, New York University, New York, NY, USA
| | - Xaq Pitkow
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA
- Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA
- Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA
| | - Dora E. Angelaki
- Tandon School of Engineering, New York University, New York, NY, USA
- Center for Neural Science, New York University, New York, NY, USA
| |
Collapse
|
4
|
Oza A, Kumar A, Sharma A, Mutha PK. Limb-related sensory prediction errors and task-related performance errors facilitate human sensorimotor learning through separate mechanisms. PLoS Biol 2024; 22:e3002703. [PMID: 38959259 PMCID: PMC11221701 DOI: 10.1371/journal.pbio.3002703] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2024] [Accepted: 06/06/2024] [Indexed: 07/05/2024] Open
Abstract
The unpredictable nature of our world can introduce a variety of errors in our actions, including sensory prediction errors (SPEs) and task performance errors (TPEs). SPEs arise when our existing internal models of limb-environment properties and interactions become miscalibrated due to changes in the environment, while TPEs occur when environmental perturbations hinder achievement of task goals. The precise mechanisms employed by the sensorimotor system to learn from such limb- and task-related errors and improve future performance are not comprehensively understood. To gain insight into these mechanisms, we performed a series of learning experiments wherein the location and size of a reach target were varied, the visual feedback of the motion was perturbed in different ways, and instructions were carefully manipulated. Our findings indicate that the mechanisms employed to compensate SPEs and TPEs are dissociable. Specifically, our results fail to support theories that suggest that TPEs trigger implicit refinement of reach plans or that their occurrence automatically modulates SPE-mediated learning. Rather, TPEs drive improved action selection, that is, the selection of verbally sensitive, volitional strategies that reduce future errors. Moreover, we find that exposure to SPEs is necessary and sufficient to trigger implicit recalibration. When SPE-mediated implicit learning and TPE-driven improved action selection combine, performance gains are larger. However, when actions are always successful and strategies are not employed, refinement in behavior is smaller. Flexibly weighting strategic action selection and implicit recalibration could thus be a way of controlling how much, and how quickly, we learn from errors.
Collapse
Affiliation(s)
- Anushka Oza
- Center for Cognitive and Brain Sciences, Indian Institute of Technology Gandhinagar, Gujarat, India
| | - Adarsh Kumar
- Center for Cognitive and Brain Sciences, Indian Institute of Technology Gandhinagar, Gujarat, India
- Department of Mechanical Engineering, Indian Institute of Technology Gandhinagar, Gujarat, India
| | - Apoorva Sharma
- Center for Cognitive and Brain Sciences, Indian Institute of Technology Gandhinagar, Gujarat, India
| | - Pratik K. Mutha
- Center for Cognitive and Brain Sciences, Indian Institute of Technology Gandhinagar, Gujarat, India
- Department of Biological Engineering, Indian Institute of Technology Gandhinagar, Gujarat, India
| |
Collapse
|
5
|
Bredenberg C, Savin C. Desiderata for Normative Models of Synaptic Plasticity. Neural Comput 2024; 36:1245-1285. [PMID: 38776950 DOI: 10.1162/neco_a_01671] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Accepted: 02/06/2024] [Indexed: 05/25/2024]
Abstract
Normative models of synaptic plasticity use computational rationales to arrive at predictions of behavioral and network-level adaptive phenomena. In recent years, there has been an explosion of theoretical work in this realm, but experimental confirmation remains limited. In this review, we organize work on normative plasticity models in terms of a set of desiderata that, when satisfied, are designed to ensure that a given model demonstrates a clear link between plasticity and adaptive behavior, is consistent with known biological evidence about neural plasticity and yields specific testable predictions. As a prototype, we include a detailed analysis of the REINFORCE algorithm. We also discuss how new models have begun to improve on the identified criteria and suggest avenues for further development. Overall, we provide a conceptual guide to help develop neural learning theories that are precise, powerful, and experimentally testable.
Collapse
Affiliation(s)
- Colin Bredenberg
- Center for Neural Science, New York University, New York, NY 10003, U.S.A
- Mila-Quebec AI Institute, Montréal, QC H2S 3H1, Canada
| | - Cristina Savin
- Center for Neural Science, New York University, New York, NY 10003, U.S.A
- Center for Data Science, New York University, New York, NY 10011, U.S.A.
| |
Collapse
|
6
|
Lehmann CM, Miller NE, Nair VS, Costa KM, Schoenbaum G, Moussawi K. Generalized cue reactivity in dopamine neurons after opioids. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.02.597025. [PMID: 38853878 PMCID: PMC11160774 DOI: 10.1101/2024.06.02.597025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2024]
Abstract
Cue reactivity is the maladaptive neurobiological and behavioral response upon exposure to drug cues and is a major driver of relapse. The leading hypothesis is that dopamine release by addictive drugs represents a persistently positive reward prediction error that causes runaway enhancement of dopamine responses to drug cues, leading to their pathological overvaluation compared to non-drug reward alternatives. However, this hypothesis has not been directly tested. Here we developed Pavlovian and operant procedures to measure firing responses, within the same dopamine neurons, to drug versus natural reward cues, which we found to be similarly enhanced compared to cues predicting natural rewards in drug-naïve controls. This enhancement was associated with increased behavioral reactivity to the drug cue, suggesting that dopamine release is still critical to cue reactivity, albeit not as previously hypothesized. These results challenge the prevailing hypothesis of cue reactivity, warranting new models of dopaminergic function in drug addiction, and provide critical insights into the neurobiology of cue reactivity with potential implications for relapse prevention.
Collapse
Affiliation(s)
- Collin M. Lehmann
- Department of Psychiatry, University of Pittsburgh; Pittsburgh, 15219, USA
| | - Nora E. Miller
- Department of Psychiatry, University of Pittsburgh; Pittsburgh, 15219, USA
| | - Varun S. Nair
- Department of Psychiatry, University of Pittsburgh; Pittsburgh, 15219, USA
| | - Kauê M. Costa
- Department of Psychology, University of Alabama at Birmingham; Birmingham, 35233, USA
| | - Geoffrey Schoenbaum
- National Institute on Drug Abuse, National Institutes of Health; Baltimore, 21224, USA
| | - Khaled Moussawi
- Department of Psychiatry, University of Pittsburgh; Pittsburgh, 15219, USA
- Department of Neurology, University of California San Francisco; San Francisco, 94158, USA
| |
Collapse
|
7
|
Alejandro RJ, Holroyd CB. Hierarchical control over foraging behavior by anterior cingulate cortex. Neurosci Biobehav Rev 2024; 160:105623. [PMID: 38490499 DOI: 10.1016/j.neubiorev.2024.105623] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Revised: 02/14/2024] [Accepted: 03/13/2024] [Indexed: 03/17/2024]
Abstract
Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).
Collapse
Affiliation(s)
| | - Clay B Holroyd
- Department of Experimental Psychology, Ghent University, Ghent, Belgium
| |
Collapse
|
8
|
Osuna-Orozco R, Zhao Y, Stealey HM, Lu HY, Contreras-Hernandez E, Santacruz SR. Adaptation and learning as strategies to maximize reward in neurofeedback tasks. Front Hum Neurosci 2024; 18:1368115. [PMID: 38590363 PMCID: PMC11000125 DOI: 10.3389/fnhum.2024.1368115] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Accepted: 03/04/2024] [Indexed: 04/10/2024] Open
Abstract
Introduction Adaptation and learning have been observed to contribute to the acquisition of new motor skills and are used as strategies to cope with changing environments. However, it is hard to determine the relative contribution of each when executing goal directed motor tasks. This study explores the dynamics of neural activity during a center-out reaching task with continuous visual feedback under the influence of rotational perturbations. Methods Results for a brain-computer interface (BCI) task performed by two non-human primate (NHP) subjects are compared to simulations from a reinforcement learning agent performing an analogous task. We characterized baseline activity and compared it to the activity after rotational perturbations of different magnitudes were introduced. We employed principal component analysis (PCA) to analyze the spiking activity driving the cursor in the NHP BCI task as well as the activation of the neural network of the reinforcement learning agent. Results and discussion Our analyses reveal that both for the NHPs and the reinforcement learning agent, the task-relevant neural manifold is isomorphic with the task. However, for the NHPs the manifold is largely preserved for all rotational perturbations explored and adaptation of neural activity occurs within this manifold as rotations are compensated by reassignment of regions of the neural space in an angular pattern that cancels said rotations. In contrast, retraining the reinforcement learning agent to reach the targets after rotation results in substantial modifications of the underlying neural manifold. Our findings demonstrate that NHPs adapt their existing neural dynamic repertoire in a quantitatively precise manner to account for perturbations of different magnitudes and they do so in a way that obviates the need for extensive learning.
Collapse
Affiliation(s)
- Rodrigo Osuna-Orozco
- Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, United States
| | - Yi Zhao
- Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, United States
| | - Hannah Marie Stealey
- Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, United States
| | - Hung-Yun Lu
- Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, United States
| | | | - Samantha Rose Santacruz
- Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, United States
- Department of Electrical and Computer Engineering, University of Texas at Austin, Austin, TX, United States
- Institute for Neuroscience, University of Texas at Austin, Austin, TX, United States
| |
Collapse
|
9
|
Johannsen L, Koch I. Learning a covert sequence of effector movements: limits to its acquisition. PSYCHOLOGICAL RESEARCH 2024; 88:197-206. [PMID: 37422801 PMCID: PMC10805866 DOI: 10.1007/s00426-023-01855-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2023] [Accepted: 06/27/2023] [Indexed: 07/11/2023]
Abstract
Sequence learning in serial reaction time (SRT) tasks is an established, lab-based experimental paradigm to study acquisition and transfer of skills based on the detection of predictable regularities in stimulus and motor response sequences. Participants learn a sequence of targets and responses to these targets by associating the responses with subsequently presented targets. In the traditional paradigm, however, actions and response targets are directly related. In contrast, the present study asked whether participants would demonstrate acquisition of a sequence of effector movements of the left vs. right hand (e.g., hand sequence learning), whilst the actual targets and associated finger responses are unpredictable. Twenty-seven young adults performed a SRT task to visually presented characters with the index or middle fingers of both hands. While the specific fingers to respond with were randomly selected for each target presentation, both hands followed a covert sequence. We asked whether participants would learn the underlying hand sequence as demonstrated by shortened response latencies and increased accuracy compared to a fully randomized hand sequence. The results show sequence-specific learning effects. However, categorization of hand responses depending on the previous response suggested that learning occurred predominantly for subsequent finger responses of the same hand, which added to general hand-based priming. Nevertheless, a marginally significant effect was observed even for predictable shifts between hands when homologous fingers were involved. Our results thus suggest that humans are able to benefit from predictable within-hand finger shifts but less so for predicted shifts between hands.
Collapse
Affiliation(s)
- Leif Johannsen
- Cognitive and Experimental Psychology, Institute of Psychology, RWTH Aachen University, Jägerstrasse 17/19, 52066, Aachen, Germany.
- Department of Psychology, Durham University, Durham, UK.
| | - Iring Koch
- Cognitive and Experimental Psychology, Institute of Psychology, RWTH Aachen University, Jägerstrasse 17/19, 52066, Aachen, Germany
| |
Collapse
|
10
|
Howlett JR, Paulus MP. Out of control: computational dynamic control dysfunction in stress- and anxiety-related disorders. DISCOVER MENTAL HEALTH 2024; 4:5. [PMID: 38236488 PMCID: PMC10796870 DOI: 10.1007/s44192-023-00058-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 12/28/2023] [Indexed: 01/19/2024]
Abstract
Control theory, which has played a central role in technological progress over the last 150 years, has also yielded critical insights into biology and neuroscience. Recently, there has been a surging interest in integrating control theory with computational psychiatry. Here, we review the state of the field of using control theory approaches in computational psychiatry and show that recent research has mapped a neural control circuit consisting of frontal cortex, parietal cortex, and the cerebellum. This basic feedback control circuit is modulated by estimates of reward and cost via the basal ganglia as well as by arousal states coordinated by the insula, dorsal anterior cingulate cortex, amygdala, and locus coeruleus. One major approach within the broader field of control theory, known as proportion-integral-derivative (PID) control, has shown promise as a model of human behavior which enables precise and reliable estimates of underlying control parameters at the individual level. These control parameters correlate with self-reported fear and with both structural and functional variation in affect-related brain regions. This suggests that dysfunctional engagement of stress and arousal systems may suboptimally modulate parameters of domain-general goal-directed control algorithms, impairing performance in complex tasks involving movement, cognition, and affect. Future directions include clarifying the causal role of control deficits in stress- and anxiety-related disorders and developing clinically useful tools based on insights from control theory.
Collapse
Affiliation(s)
- Jonathon R Howlett
- VA San Diego Healthcare System, 3350 La Jolla Village Dr, San Diego, CA, 92161, USA.
- Department of Psychiatry, University of California San Diego, La Jolla, CA, USA.
| | | |
Collapse
|
11
|
Nicholas J, Daw ND, Shohamy D. Proactive and reactive construction of memory-based preferences. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.12.10.570977. [PMID: 38106137 PMCID: PMC10723393 DOI: 10.1101/2023.12.10.570977] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/19/2023]
Abstract
We are often faced with decisions we have never encountered before, requiring us to infer possible outcomes before making a choice. Computational theories suggest that one way to make these types of decisions is by accessing and linking related experiences stored in memory. Past work has shown that such memory-based preference construction can occur at a number of different timepoints relative to the moment a decision is made. Some studies have found that memories are integrated at the time a decision is faced (reactively) while others found that memory integration happens earlier, when memories are encoded (proactively). Here we offer a resolution to this inconsistency. We demonstrate behavioral and neural evidence for both strategies and for how they tradeoff rationally depending on the associative structure of memory. Using fMRI to decode patterns of brain responses unique to categories of images in memory, we found that proactive memory access is more common and allows more efficient inference. However, participants also use reactive access when choice options are linked to more numerous memory associations. Together, these results indicate that the brain judiciously conducts proactive inference by accessing memories ahead of time in conditions when this strategy is most favorable.
Collapse
Affiliation(s)
- Jonathan Nicholas
- Department of Psychology, Columbia University, New York, NY, USA
- Mortimer B. Zuckerman Mind, Brain, Behavior Institute, Columbia University, New York, NY, USA
- Department of Psychology, New York University, New York, NY, USA
| | - Nathaniel D Daw
- Department of Psychology, Princeton University, Princeton, NJ, USA
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Daphna Shohamy
- Department of Psychology, Columbia University, New York, NY, USA
- Mortimer B. Zuckerman Mind, Brain, Behavior Institute, Columbia University, New York, NY, USA
- The Kavli Institute for Brain Science, Columbia University, New York, NY, USA
| |
Collapse
|
12
|
Peng XR, Bundil I, Schulreich S, Li SC. Neural correlates of valence-dependent belief and value updating during uncertainty reduction: An fNIRS study. Neuroimage 2023; 279:120327. [PMID: 37582418 DOI: 10.1016/j.neuroimage.2023.120327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Revised: 08/07/2023] [Accepted: 08/11/2023] [Indexed: 08/17/2023] Open
Abstract
Selective use of new information is crucial for adaptive decision-making. Combining a gamble bidding task with assessing cortical responses using functional near-infrared spectroscopy (fNIRS), we investigated potential effects of information valence on behavioral and neural processes of belief and value updating during uncertainty reduction in young adults. By modeling changes in the participants' expressed subjective values using a Bayesian model, we dissociated processes of (i) updating beliefs about statistical properties of the gamble, (ii) updating values of a gamble based on new information about its winning probabilities, as well as (iii) expectancy violation. The results showed that participants used new information to update their beliefs and values about the gambles in a quasi-optimal manner, as reflected in the selective updating only in situations with reducible uncertainty. Furthermore, their updating was valence-dependent: information indicating an increase in winning probability was underweighted, whereas information about a decrease in winning probability was updated in good agreement with predictions of the Bayesian decision theory. Results of model-based and moderation analyses showed that this valence-dependent asymmetry was associated with a distinct contribution of expectancy violation, besides belief updating, to value updating after experiencing new positive information regarding winning probabilities. In line with the behavioral results, we replicated previous findings showing involvements of frontoparietal brain regions in the different components of updating. Furthermore, this study provided novel results suggesting a valence-dependent recruitment of brain regions. Individuals with stronger oxyhemoglobin responses during value updating was more in line with predictions of the Bayesian model while integrating new information that indicates an increase in winning probability. Taken together, this study provides first results showing expectancy violation as a contributing factor to sub-optimal valence-dependent updating during uncertainty reduction and suggests limitations of normative Bayesian decision theory.
Collapse
Affiliation(s)
- Xue-Rui Peng
- Chair of Lifespan Developmental Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop, Technische Universität Dresden, Dresden, Germany.
| | - Indra Bundil
- Cardiff University Brain Research Imaging Centre, School of Psychology, Cardiff University, Cardiff, United Kingdom
| | - Stefan Schulreich
- Department of Nutritional Sciences, Faculty of Life Sciences, University of Vienna, Vienna, Austria; Department of Cognitive Psychology, Faculty of Psychology and Human Movement Science, Universität Hamburg, Hamburg, Germany
| | - Shu-Chen Li
- Chair of Lifespan Developmental Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop, Technische Universität Dresden, Dresden, Germany.
| |
Collapse
|
13
|
Shi W, Meisner OC, Blackmore S, Jadi MP, Nandy AS, Chang SWC. The orbitofrontal cortex: A goal-directed cognitive map framework for social and non-social behaviors. Neurobiol Learn Mem 2023; 203:107793. [PMID: 37353191 PMCID: PMC10527225 DOI: 10.1016/j.nlm.2023.107793] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Revised: 05/28/2023] [Accepted: 06/19/2023] [Indexed: 06/25/2023]
Abstract
The orbitofrontal cortex (OFC) is regarded as one of the core brain areas in a variety of value-based behaviors. Over the past two decades, tremendous knowledge about the OFC function was gained from studying the behaviors of single subjects. As a result, our previous understanding of the OFC's function of encoding decision variables, such as the value and identity of choices, has evolved to the idea that the OFC encodes a more complex representation of the task space as a cognitive map. Accumulating evidence also indicates that the OFC importantly contributes to behaviors in social contexts, especially those involved in cooperative interactions. However, it remains elusive how exactly OFC neurons contribute to social functions and how non-social and social behaviors are related to one another in the computations performed by OFC neurons. In this review, we aim to provide an integrated view of the OFC function across both social and non-social behavioral contexts. We propose that seemingly complex functions of the OFC may be explained by its role in providing a goal-directed cognitive map to guide a wide array of adaptive reward-based behaviors in both social and non-social domains.
Collapse
Affiliation(s)
- Weikang Shi
- Wu Tsai Institute, Yale University, New Haven, CT 06510, USA; Department of Psychology, Yale University, New Haven, CT 06510, USA; Department of Neuroscience, Yale University School of Medicine, New Haven, CT 06510, USA
| | - Olivia C Meisner
- Department of Psychology, Yale University, New Haven, CT 06510, USA; Department of Neuroscience, Yale University School of Medicine, New Haven, CT 06510, USA
| | - Sylvia Blackmore
- Wu Tsai Institute, Yale University, New Haven, CT 06510, USA; Department of Psychology, Yale University, New Haven, CT 06510, USA
| | - Monika P Jadi
- Wu Tsai Institute, Yale University, New Haven, CT 06510, USA; Department of Neuroscience, Yale University School of Medicine, New Haven, CT 06510, USA; Department of Psychiatry, Yale University School of Medicine, New Haven, CT 06510, USA
| | - Anirvan S Nandy
- Wu Tsai Institute, Yale University, New Haven, CT 06510, USA; Department of Neuroscience, Yale University School of Medicine, New Haven, CT 06510, USA; Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT 06510, USA
| | - Steve W C Chang
- Wu Tsai Institute, Yale University, New Haven, CT 06510, USA; Department of Psychology, Yale University, New Haven, CT 06510, USA; Department of Neuroscience, Yale University School of Medicine, New Haven, CT 06510, USA; Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT 06510, USA.
| |
Collapse
|
14
|
Abstract
Humans often generalize rewarding experiences across abstract social roles. Theories of reward learning suggest that people generalize through model-based learning, but such learning is cognitively costly. Why do people seem to generalize across social roles with ease? Humans are social experts who easily recognize social roles that reflect familiar semantic concepts (e.g., "helper" or "teacher"). People may associate these roles with model-free reward (e.g., learning that helpers are rewarding), allowing them to generalize easily (e.g., interacting with novel individuals identified as helpers). In four online experiments with U.S. adults (N = 577), we found evidence that social concepts ease complex learning (people generalize more and at faster speed) and that people attach reward directly to abstract roles (they generalize even when roles are unrelated to task structure). These results demonstrate how familiar concepts allow complex behavior to emerge from simple strategies, highlighting social interaction as a prototype for studying cognitive ease in the face of environmental complexity.
Collapse
Affiliation(s)
- Leor M Hackel
- Department of Psychology, University of Southern California
| | | |
Collapse
|
15
|
Bredenberg C, Savin C. Desiderata for normative models of synaptic plasticity. ARXIV 2023:arXiv:2308.04988v1. [PMID: 37608931 PMCID: PMC10441445] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Subscribe] [Scholar Register] [Indexed: 08/24/2023]
Abstract
Normative models of synaptic plasticity use a combination of mathematics and computational simulations to arrive at predictions of behavioral and network-level adaptive phenomena. In recent years, there has been an explosion of theoretical work on these models, but experimental confirmation is relatively limited. In this review, we organize work on normative plasticity models in terms of a set of desiderata which, when satisfied, are designed to guarantee that a model has a clear link between plasticity and adaptive behavior, consistency with known biological evidence about neural plasticity, and specific testable predictions. We then discuss how new models have begun to improve on these criteria and suggest avenues for further development. As prototypes, we provide detailed analyses of two specific models - REINFORCE and the Wake-Sleep algorithm. We provide a conceptual guide to help develop neural learning theories that are precise, powerful, and experimentally testable.
Collapse
Affiliation(s)
- Colin Bredenberg
- Center for Neural Science, New York University, New York, NY 10003, USA
- Mila-Quebec AI Institute, 6666 Rue Saint-Urbain, Montréal, QC H2S 3H1
| | - Cristina Savin
- Center for Neural Science, New York University, New York, NY 10003, USA
- Center for Data Science, New York University, New York, NY 10011, USA
| |
Collapse
|
16
|
Manzur HE, Vlasov K, Jhong YJ, Chen HY, Lin SC. The behavioral signature of stepwise learning strategy in male rats and its neural correlate in the basal forebrain. Nat Commun 2023; 14:4415. [PMID: 37479696 PMCID: PMC10362048 DOI: 10.1038/s41467-023-40145-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Accepted: 07/13/2023] [Indexed: 07/23/2023] Open
Abstract
Studies of associative learning have commonly focused on how rewarding outcomes are predicted by either sensory stimuli or animals' actions. However, in many learning scenarios, reward delivery requires the occurrence of both sensory stimuli and animals' actions in a specific order, in the form of behavioral sequences. How such behavioral sequences are learned is much less understood. Here we provide behavioral and neurophysiological evidence to show that behavioral sequences are learned using a stepwise strategy. In male rats learning a new association, learning started from the behavioral event closest to the reward and sequentially incorporated earlier events. This led to the sequential refinement of reward-seeking behaviors, which was characterized by the stepwise elimination of ineffective and non-rewarded behavioral sequences. At the neuronal level, this stepwise learning process was mirrored by the sequential emergence of basal forebrain neuronal responses toward each event, which quantitatively conveyed a reward prediction error signal and promoted reward-seeking behaviors. Together, these behavioral and neural signatures revealed how behavioral sequences were learned in discrete steps and when each learning step took place.
Collapse
Affiliation(s)
- Hachi E Manzur
- Neural Circuits and Cognition Unit, Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institutes of Health, Baltimore, MD, USA
| | - Ksenia Vlasov
- Neural Circuits and Cognition Unit, Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institutes of Health, Baltimore, MD, USA
| | - You-Jhe Jhong
- Institute of Neuroscience, National Yang Ming Chiao Tung University, Taipei, Taiwan
| | - Hung-Yen Chen
- Institute of Neuroscience, National Yang Ming Chiao Tung University, Taipei, Taiwan
| | - Shih-Chieh Lin
- Neural Circuits and Cognition Unit, Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institutes of Health, Baltimore, MD, USA.
- Institute of Neuroscience, National Yang Ming Chiao Tung University, Taipei, Taiwan.
- Brain Research Center, National Yang Ming Chiao Tung University, Taipei, Taiwan.
| |
Collapse
|
17
|
Navidi P, Saeedpour S, Ershadmanesh S, Hossein MM, Bahrami B. Prosocial learning: Model-based or model-free? PLoS One 2023; 18:e0287563. [PMID: 37352225 PMCID: PMC10289351 DOI: 10.1371/journal.pone.0287563] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2022] [Accepted: 06/07/2023] [Indexed: 06/25/2023] Open
Abstract
Prosocial learning involves the acquisition of knowledge and skills necessary for making decisions that benefit others. We asked if, in the context of value-based decision-making, there is any difference between learning strategies for oneself vs. for others. We implemented a 2-step reinforcement learning paradigm in which participants learned, in separate blocks, to make decisions for themselves or for a present other confederate who evaluated their performance. We replicated the canonical features of the model-based and model-free reinforcement learning in our results. The behaviour of the majority of participants was best explained by a mixture of the model-based and model-free control, while most participants relied more heavily on MB control, and this strategy enhanced their learning success. Regarding our key self-other hypothesis, we did not find any significant difference between the behavioural performances nor in the model-based parameters of learning when comparing self and other conditions.
Collapse
Affiliation(s)
- Parisa Navidi
- Department of Cognitive Psychology, Institute for Cognitive Science Studies, Tehran, Iran
| | - Sepehr Saeedpour
- Department of Electrical and Computer Engineering, University of Tehran, Tehran, Iran
| | - Sara Ershadmanesh
- School of Cognitive Sciences, Institute for Research in Fundamental Sciences, Tehran, Iran
- Department of Computational Neuroscience, MPI for Biological Cybernetics, Tuebingen, Germany
| | | | - Bahador Bahrami
- Crowd Cognition Group, Department of General Psychology and Education, Ludwig Maximilians University, Munich, Germany
| |
Collapse
|
18
|
Feher da Silva C, Lombardi G, Edelson M, Hare TA. Rethinking model-based and model-free influences on mental effort and striatal prediction errors. Nat Hum Behav 2023:10.1038/s41562-023-01573-1. [PMID: 37012365 DOI: 10.1038/s41562-023-01573-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Accepted: 02/27/2023] [Indexed: 04/05/2023]
Abstract
A standard assumption in neuroscience is that low-effort model-free learning is automatic and continuously used, whereas more complex model-based strategies are only used when the rewards they generate are worth the additional effort. We present evidence refuting this assumption. First, we demonstrate flaws in previous reports of combined model-free and model-based reward prediction errors in the ventral striatum that probably led to spurious results. More appropriate analyses yield no evidence of model-free prediction errors in this region. Second, we find that task instructions generating more correct model-based behaviour reduce rather than increase mental effort. This is inconsistent with cost-benefit arbitration between model-based and model-free strategies. Together, our data indicate that model-free learning may not be automatic. Instead, humans can reduce mental effort by using a model-based strategy alone rather than arbitrating between multiple strategies. Our results call for re-evaluation of the assumptions in influential theories of learning and decision-making.
Collapse
Affiliation(s)
| | - Gaia Lombardi
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
| | - Micah Edelson
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
| | - Todd A Hare
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland.
| |
Collapse
|
19
|
Letkiewicz AM, Kottler HC, Shankman SA, Cochran AL. Quantifying aberrant approach-avoidance conflict in psychopathology: A review of computational approaches. Neurosci Biobehav Rev 2023; 147:105103. [PMID: 36804398 PMCID: PMC10023482 DOI: 10.1016/j.neubiorev.2023.105103] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 02/02/2023] [Accepted: 02/16/2023] [Indexed: 02/19/2023]
Abstract
Making effective decisions during approach-avoidance conflict is critical in daily life. Aberrant decision-making during approach-avoidance conflict is evident in a range of psychological disorders, including anxiety, depression, trauma-related disorders, substance use disorders, and alcohol use disorders. To help clarify etiological pathways and reveal novel intervention targets, clinical research into decision-making is increasingly adopting a computational psychopathology approach. This approach uses mathematical models that can identify specific decision-making related processes that are altered in mental health disorders. In our review, we highlight foundational approach-avoidance conflict research, followed by more in-depth discussion of computational approaches that have been used to model behavior in these tasks. Specifically, we describe the computational models that have been applied to approach-avoidance conflict (e.g., drift-diffusion, active inference, and reinforcement learning models), and provide resources to guide clinical researchers who may be interested in applying computational modeling. Finally, we identify notable gaps in the current literature and potential future directions for computational approaches aimed at identifying mechanisms of approach-avoidance conflict in psychopathology.
Collapse
Affiliation(s)
- Allison M Letkiewicz
- Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago, IL, USA.
| | - Haley C Kottler
- Department of Mathematics, University of Wisconsin, Madison, WI, USA
| | - Stewart A Shankman
- Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago, IL, USA; Department of Psychology, Northwestern University, Evanston, IL, USA
| | - Amy L Cochran
- Department of Mathematics, University of Wisconsin, Madison, WI, USA; Department of Population Health Sciences, University of Wisconsin, Madison, WI, USA
| |
Collapse
|
20
|
Explainable reinforcement learning for broad-XAI: a conceptual framework and survey. Neural Comput Appl 2023. [DOI: 10.1007/s00521-023-08423-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/08/2023]
Abstract
AbstractBroad-XAI moves away from interpreting individual decisions based on a single datum and aims to provide integrated explanations from multiple machine learning algorithms into a coherent explanation of an agent’s behaviour that is aligned to the communication needs of the explainee. Reinforcement Learning (RL) methods, we propose, provide a potential backbone for the cognitive model required for the development of Broad-XAI. RL represents a suite of approaches that have had increasing success in solving a range of sequential decision-making problems. However, these algorithms operate as black-box problem solvers, where they obfuscate their decision-making policy through a complex array of values and functions. EXplainable RL (XRL) aims to develop techniques to extract concepts from the agent’s: perception of the environment; intrinsic/extrinsic motivations/beliefs; Q-values, goals and objectives. This paper aims to introduce the Causal XRL Framework (CXF), that unifies the current XRL research and uses RL as a backbone to the development of Broad-XAI. CXF is designed to incorporate many standard RL extensions and integrated with external ontologies and communication facilities so that the agent can answer questions that explain outcomes its decisions. This paper aims to: establish XRL as a distinct branch of XAI; introduce a conceptual framework for XRL; review existing approaches explaining agent behaviour; and identify opportunities for future research. Finally, this paper discusses how additional information can be extracted and ultimately integrated into models of communication, facilitating the development of Broad-XAI.
Collapse
|
21
|
Ruel A, Bolenz F, Li SC, Fischer A, Eppinger B. Neural evidence for age-related deficits in the representation of state spaces. Cereb Cortex 2023; 33:1768-1781. [PMID: 35510942 DOI: 10.1093/cercor/bhac171] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2022] [Revised: 04/08/2022] [Accepted: 04/09/2022] [Indexed: 11/12/2022] Open
Abstract
Under high cognitive demands, older adults tend to resort to simpler, habitual, or model-free decision strategies. This age-related shift in decision behavior has been attributed to deficits in the representation of the cognitive maps, or state spaces, necessary for more complex model-based decision-making. Yet, the neural mechanisms behind this shift remain unclear. In this study, we used a modified 2-stage Markov task in combination with computational modeling and single-trial EEG analyses to establish neural markers of age-related changes in goal-directed decision-making under different demands on the representation of state spaces. Our results reveal that the shift to simpler decision strategies in older adults is due to (i) impairments in the representation of the transition structure of the task and (ii) a diminished signaling of the reward value associated with decision options. In line with the diminished state space hypothesis of human aging, our findings suggest that deficits in goal-directed, model-based behavior in older adults result from impairments in the representation of state spaces of cognitive tasks.
Collapse
Affiliation(s)
- Alexa Ruel
- Department of Psychology, Concordia University, 7141 Sherbrooke Street W. Montreal, Quebec H4B 1R6, Canada
| | - Florian Bolenz
- Chair of Lifespan Developmental Neuroscience, Faculty of Psychology, Technische Universität Dresden, Bürogebäude, Zi. 244 Strehlener Straße 22/24, Dresden 01069, Germany.,Center for Adaptive Rationality, Max Planck Institute for Human Development, Lentzeallee 94, Berlin 14195, Germany.,Cluster of Excellence "Science of Intelligence", Technische Universität Berlin, Straße des 17. Juni 135, Berlin 10623, Germany
| | - Shu-Chen Li
- Chair of Lifespan Developmental Neuroscience, Faculty of Psychology, Technische Universität Dresden, Bürogebäude, Zi. 244 Strehlener Straße 22/24, Dresden 01069, Germany.,Cluster of Excellence "Centre for Tactile Internet with Human-in-the-Loop", Technische Universität Dresden, Bürogebäude, Zi. 244 Strehlener Straße 22/24, Dresden 01069, Germany
| | - Adrian Fischer
- Department of Education and Psychology, Freie Universität Berlin, Habelschwerdter Allee 45, Berlin 14195, Germany
| | - Ben Eppinger
- Department of Psychology, Concordia University, 7141 Sherbrooke Street W. Montreal, Quebec H4B 1R6, Canada.,Chair of Lifespan Developmental Neuroscience, Faculty of Psychology, Technische Universität Dresden, Bürogebäude, Zi. 244 Strehlener Straße 22/24, Dresden 01069, Germany.,PERFORM Center, Concordia University, 7200 Sherbrooke St. W. Montreal, Quebec H4B 1R6, Canada
| |
Collapse
|
22
|
Bergenholtz C, Vuculescu O, Amidi A. Microfoundations of Adaptive Search in Complex Tasks: The Role of Cognitive Abilities and Styles. ORGANIZATION SCIENCE 2023. [DOI: 10.1287/orsc.2023.1654] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/29/2023]
Abstract
Problem-solving in complex environments requires a cognitively demanding search for task solutions. Managing this search process presents a major challenge in organizations. We contribute to the literature on this topic by providing new evidence on the cognitive antecedents that shape how individuals search when engaged in complex problem-solving tasks. We present results from three laboratory studies, wherein 335 individuals solved a complex task. In doing so, they generated behavioral data coupled with survey-based measurements of the individuals’ cognitive styles and performance-based tests of their cognitive abilities. Our data analysis contributes to the current literature by documenting systematic heterogeneity in the persistence and distance of search that can be explained by the participants’ level of creativity, attention to detail, and executive functions. We extend the research on the microfoundations of adaptive search by linking cognitive antecedents with a complex search task, widening our insight into what search behavior certain cognitive microfoundations lead to, and showing how managers can more effectively shape organizational search. History: This paper has been accepted for the Organization Science Special Issue on Experiments in Organizational Theory. Supplemental Material: The online appendix is available at https://doi.org/10.1287/orsc.2023.1654 .
Collapse
Affiliation(s)
| | - Oana Vuculescu
- Department of Management, Aarhus University, Aarhus, Denmark
| | - Ali Amidi
- Department of Psychology and Behavioural Sciences, Aarhus University, Aarhus, Denmark
| |
Collapse
|
23
|
Emanuel A, Eldar E. Emotions as computations. Neurosci Biobehav Rev 2023; 144:104977. [PMID: 36435390 PMCID: PMC9805532 DOI: 10.1016/j.neubiorev.2022.104977] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2022] [Revised: 10/26/2022] [Accepted: 11/22/2022] [Indexed: 11/26/2022]
Abstract
Emotions ubiquitously impact action, learning, and perception, yet their essence and role remain widely debated. Computational accounts of emotion aspire to answer these questions with greater conceptual precision informed by normative principles and neurobiological data. We examine recent progress in this regard and find that emotions may implement three classes of computations, which serve to evaluate states, actions, and uncertain prospects. For each of these, we use the formalism of reinforcement learning to offer a new formulation that better accounts for existing evidence. We then consider how these distinct computations may map onto distinct emotions and moods. Integrating extensive research on the causes and consequences of different emotions suggests a parsimonious one-to-one mapping, according to which emotions are integral to how we evaluate outcomes (pleasure & pain), learn to predict them (happiness & sadness), use them to inform our (frustration & content) and others' (anger & gratitude) actions, and plan in order to realize (desire & hope) or avoid (fear & anxiety) uncertain outcomes.
Collapse
Affiliation(s)
- Aviv Emanuel
- Department of Psychology, Hebrew University of Jerusalem, Jerusalem 9190501, Israel; Department of Cognitive and Brain Sciences, Hebrew University of Jerusalem, Jerusalem 9190501, Israel.
| | - Eran Eldar
- Department of Psychology, Hebrew University of Jerusalem, Jerusalem 9190501, Israel; Department of Cognitive and Brain Sciences, Hebrew University of Jerusalem, Jerusalem 9190501, Israel.
| |
Collapse
|
24
|
Mathar D, Erfanian Abdoust M, Marrenbach T, Tuzsus D, Peters J. The catecholamine precursor Tyrosine reduces autonomic arousal and decreases decision thresholds in reinforcement learning and temporal discounting. PLoS Comput Biol 2022; 18:e1010785. [PMID: 36548401 PMCID: PMC9822114 DOI: 10.1371/journal.pcbi.1010785] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Revised: 01/06/2023] [Accepted: 12/01/2022] [Indexed: 12/24/2022] Open
Abstract
Supplementation with the catecholamine precursor L-Tyrosine might enhance cognitive performance, but overall findings are mixed. Here, we investigate the effect of a single dose of tyrosine (2g) vs. placebo on two catecholamine-dependent trans-diagnostic traits: model-based control during reinforcement learning (2-step task) and temporal discounting, using a double-blind, placebo-controlled, within-subject design (n = 28 healthy male participants). We leveraged drift diffusion models in a hierarchical Bayesian framework to jointly model participants' choices and response times (RTS) in both tasks. Furthermore, comprehensive autonomic monitoring (heart rate, heart rate variability, pupillometry, spontaneous eye blink rate) was performed both pre- and post-supplementation, to explore potential physiological effects of supplementation. Across tasks, tyrosine consistently reduced participants' RTs without deteriorating task-performance. Diffusion modeling linked this effect to attenuated decision-thresholds in both tasks and further revealed increased model-based control (2-step task) and (if anything) attenuated temporal discounting. On the physiological level, participants' pupil dilation was predictive of the individual degree of temporal discounting. Tyrosine supplementation reduced physiological arousal as revealed by increases in pupil dilation variability and reductions in heart rate. Supplementation-related changes in physiological arousal predicted individual changes in temporal discounting. Our findings provide first evidence that tyrosine supplementation might impact psychophysiological parameters, and suggest that modeling approaches based on sequential sampling models can yield novel insights into latent cognitive processes modulated by amino-acid supplementation.
Collapse
Affiliation(s)
- David Mathar
- Department of Psychology, Biological Psychology, University of Cologne, Cologne, Germany
- * E-mail:
| | - Mani Erfanian Abdoust
- Biological Psychology of Decision Making, Institute of Experimental Psychology, Heinrich Heine University Duesseldorf, Duesseldorf, Germany
| | - Tobias Marrenbach
- Biological Psychology of Decision Making, Institute of Experimental Psychology, Heinrich Heine University Duesseldorf, Duesseldorf, Germany
| | - Deniz Tuzsus
- Department of Psychology, Biological Psychology, University of Cologne, Cologne, Germany
| | - Jan Peters
- Department of Psychology, Biological Psychology, University of Cologne, Cologne, Germany
| |
Collapse
|
25
|
Prospective and retrospective values integrated in frontal cortex drive predictive choice. Proc Natl Acad Sci U S A 2022; 119:e2206067119. [PMID: 36417435 PMCID: PMC9889848 DOI: 10.1073/pnas.2206067119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
To make a deliberate action in a volatile environment, the brain must frequently reassess the value of each action (action-value). Choice can be initially made from the experience of trial-and-errors, but once the dynamics of the environment is learned, the choice can be made from the knowledge of the environment. The action-values constructed from the experience (retrospective value) and the ones from the knowledge (prospective value) were identified in various regions of the brain. However, how and which neural circuit integrates these values and executes the chosen action remains unknown. Combining reinforcement learning and two-photon calcium imaging, we found that the preparatory activity of neurons in a part of the frontal cortex, the anterior-lateral motor (ALM) area, initially encodes retrospective value, but after extensive training, they jointly encode the retrospective and prospective value. Optogenetic inhibition of ALM preparatory activity specifically abolished the expert mice's predictive choice behavior and returned them to the novice-like state. Thus, the integrated action-value encoded in the preparatory activity of ALM plays an important role to bias the action toward the knowledge-dependent, predictive choice behavior.
Collapse
|
26
|
Cazares C, Schreiner DC, Valencia ML, Gremel CM. Orbitofrontal cortex populations are differentially recruited to support actions. Curr Biol 2022; 32:4675-4687.e5. [PMID: 36195096 PMCID: PMC9643660 DOI: 10.1016/j.cub.2022.09.022] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2022] [Revised: 08/03/2022] [Accepted: 09/09/2022] [Indexed: 11/09/2022]
Abstract
The ability to use information from one's prior actions is necessary for decision-making. While orbitofrontal cortex (OFC) has been hypothesized as key for inferences made using cue and value-related information, whether OFC populations contribute to the use of information from volitional actions to guide behavior is not clear. Here, we used a self-paced lever-press hold-down task in which mice infer prior lever-press durations to guide subsequent action performance. We show that the activity of genetically identified lateral OFC (lOFC) subpopulations differentially instantiate current and prior action information during ongoing action execution. Transient state-dependent lOFC circuit disruptions of specified subpopulations reduced the encoding of ongoing press durations but did not disrupt the use of prior action information to guide future action performance. In contrast, a chronic functional loss of lOFC circuit activity resulted in increased reliance on recently executed lever-press durations and impaired contingency reversal, suggesting the recruitment of compensatory mechanisms that resulted in repetitive action control. Our results identify a novel role for lOFC in the integration of action information to guide adaptive behavior.
Collapse
Affiliation(s)
- Christian Cazares
- The Neurosciences Graduate Program, University of California, San Diego, 9500 Gilman Dr., La Jolla, CA 92093, USA
| | - Drew C Schreiner
- Department of Psychology, University of California, San Diego, 9500 Gilman Dr., La Jolla, CA 92093, USA
| | - Mariela Lopez Valencia
- Department of Psychology, University of California, San Diego, 9500 Gilman Dr., La Jolla, CA 92093, USA
| | - Christina M Gremel
- The Neurosciences Graduate Program, University of California, San Diego, 9500 Gilman Dr., La Jolla, CA 92093, USA; Department of Psychology, University of California, San Diego, 9500 Gilman Dr., La Jolla, CA 92093, USA.
| |
Collapse
|
27
|
Atlas LY, Dildine TC, Palacios-Barrios EE, Yu Q, Reynolds RC, Banker LA, Grant SS, Pine DS. Instructions and experiential learning have similar impacts on pain and pain-related brain responses but produce dissociations in value-based reversal learning. eLife 2022; 11:73353. [PMID: 36317867 PMCID: PMC9681218 DOI: 10.7554/elife.73353] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2021] [Accepted: 10/25/2022] [Indexed: 11/22/2022] Open
Abstract
Recent data suggest that interactions between systems involved in higher order knowledge and associative learning drive responses during value-based learning. However, it is unknown how these systems impact subjective responses, such as pain. We tested how instructions and reversal learning influence pain and pain-evoked brain activation. Healthy volunteers (n=40) were either instructed about contingencies between cues and aversive outcomes or learned through experience in a paradigm where contingencies reversed three times. We measured predictive cue effects on pain and heat-evoked brain responses using functional magnetic resonance imaging. Predictive cues dynamically modulated pain perception as contingencies changed, regardless of whether participants received contingency instructions. Heat-evoked responses in the insula, anterior cingulate, and other regions updated as contingencies changed, and responses in the prefrontal cortex mediated dynamic cue effects on pain, whereas responses in the brainstem’s rostroventral medulla (RVM) were shaped by initial contingencies throughout the task. Quantitative modeling revealed that expected value was shaped purely by instructions in the Instructed Group, whereas expected value updated dynamically in the Uninstructed Group as a function of error-based learning. These differences were accompanied by dissociations in the neural correlates of value-based learning in the rostral anterior cingulate, thalamus, and posterior insula, among other regions. These results show how predictions dynamically impact subjective pain. Moreover, imaging data delineate three types of networks involved in pain generation and value-based learning: those that respond to initial contingencies, those that update dynamically during feedback-driven learning as contingencies change, and those that are sensitive to instruction. Together, these findings provide multiple points of entry for therapies designs to impact pain.
Collapse
Affiliation(s)
- Lauren Y Atlas
- National Center for Complementary and Integrative Health, National Institutes of Health
- National Institute on Drug Abuse, National Institutes of Health
- National Institute of Mental Health, National Institutes of Health
| | - Troy C Dildine
- National Center for Complementary and Integrative Health, National Institutes of Health
- Department of Clinical Neuroscience, Karolinska Institutet
| | | | - Qingbao Yu
- National Center for Complementary and Integrative Health, National Institutes of Health
| | | | - Lauren A Banker
- National Center for Complementary and Integrative Health, National Institutes of Health
| | - Shara S Grant
- National Center for Complementary and Integrative Health, National Institutes of Health
| | - Daniel S Pine
- National Institute of Mental Health, National Institutes of Health
| |
Collapse
|
28
|
Colas JT, Dundon NM, Gerraty RT, Saragosa‐Harris NM, Szymula KP, Tanwisuth K, Tyszka JM, van Geen C, Ju H, Toga AW, Gold JI, Bassett DS, Hartley CA, Shohamy D, Grafton ST, O'Doherty JP. Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T. Hum Brain Mapp 2022; 43:4750-4790. [PMID: 35860954 PMCID: PMC9491297 DOI: 10.1002/hbm.25988] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2022] [Revised: 05/20/2022] [Accepted: 06/10/2022] [Indexed: 11/12/2022] Open
Abstract
The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.
Collapse
Affiliation(s)
- Jaron T. Colas
- Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraCaliforniaUSA
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
- Computation and Neural Systems Program, California Institute of TechnologyPasadenaCaliforniaUSA
| | - Neil M. Dundon
- Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraCaliforniaUSA
- Department of Child and Adolescent Psychiatry, Psychotherapy, and PsychosomaticsUniversity of FreiburgFreiburg im BreisgauGermany
| | - Raphael T. Gerraty
- Department of PsychologyColumbia UniversityNew YorkNew YorkUSA
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkNew YorkUSA
- Center for Science and SocietyColumbia UniversityNew YorkNew YorkUSA
| | - Natalie M. Saragosa‐Harris
- Department of PsychologyNew York UniversityNew YorkNew YorkUSA
- Department of PsychologyUniversity of CaliforniaLos AngelesCaliforniaUSA
| | - Karol P. Szymula
- Department of BioengineeringUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Koranis Tanwisuth
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
- Department of PsychologyUniversity of CaliforniaBerkeleyCaliforniaUSA
| | - J. Michael Tyszka
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
| | - Camilla van Geen
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkNew YorkUSA
- Department of PsychologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Harang Ju
- Neuroscience Graduate GroupUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Arthur W. Toga
- Laboratory of Neuro ImagingUSC Stevens Neuroimaging and Informatics Institute, Keck School of Medicine of USC, University of Southern CaliforniaLos AngelesCaliforniaUSA
| | - Joshua I. Gold
- Department of NeuroscienceUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Dani S. Bassett
- Department of BioengineeringUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of Electrical and Systems EngineeringUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of NeurologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of PsychiatryUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of Physics and AstronomyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Santa Fe InstituteSanta FeNew MexicoUSA
| | - Catherine A. Hartley
- Department of PsychologyNew York UniversityNew YorkNew YorkUSA
- Center for Neural ScienceNew York UniversityNew YorkNew YorkUSA
| | - Daphna Shohamy
- Department of PsychologyColumbia UniversityNew YorkNew YorkUSA
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkNew YorkUSA
- Kavli Institute for Brain ScienceColumbia UniversityNew YorkNew YorkUSA
| | - Scott T. Grafton
- Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraCaliforniaUSA
| | - John P. O'Doherty
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
- Computation and Neural Systems Program, California Institute of TechnologyPasadenaCaliforniaUSA
| |
Collapse
|
29
|
Wang M, Wang Y, Sai AMVV, Liu Z, Gao Y, Tong X, Cai Z. Task assignment for hybrid scenarios in spatial crowdsourcing : A Q-Learning-based approach. Appl Soft Comput 2022. [DOI: 10.1016/j.asoc.2022.109749] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
30
|
De Corte BJ, Farley SJ, Heslin KA, Parker KL, Freeman JH. The dorsal hippocampus' role in context-based timing in rodents. Neurobiol Learn Mem 2022; 194:107673. [PMID: 35985617 DOI: 10.1016/j.nlm.2022.107673] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2022] [Revised: 08/05/2022] [Accepted: 08/11/2022] [Indexed: 01/13/2023]
Abstract
To act proactively, we must predict when future events will occur. Individuals generate temporal predictions using cues that indicate an event will happen after a certain duration elapses. Neural models of timing focus on how the brain represents these cue-duration associations. However, these models often overlook the fact that situational factors frequently modulate temporal expectations. For example, in realistic environments, the intervals associated with different cues will often covary due to a common underlying cause. According to the 'common cause hypothesis,' observers anticipate this covariance such that, when one cue's interval changes, temporal expectations for other cues shift in the same direction. Furthermore, as conditions will often differ across environments, the same cue can mean different things in different contexts. Therefore, updates to temporal expectations should be context-specific. Behavioral work supports these predictions, yet their underlying neural mechanisms are unclear. Here, we asked whether the dorsal hippocampus mediates context-based timing, given its broad role in context-conditioning. Specifically, we trained rats with either hippocampal or sham lesions that two cues predicted reward after either a short or long duration elapsed (e.g., tone-8 s/light-16 s). Then, we moved rats to a new context and extended the long cue's interval (e.g., light-32 s). This caused rats to respond later to the short cue, despite never being trained to do so. Importantly, when returned to the initial training context, sham rats shifted back toward both cues' original intervals. In contrast, lesion rats continued to respond at the long cue's newer interval. Surprisingly, they still showed contextual modulation for the short cue, responding earlier like shams. These data suggest the hippocampus only mediates context-based timing if a cue is explicitly paired and/or rewarded across distinct contexts. Furthermore, as lesions did not impact timing measures at baseline or acquisition for the long cue's new interval, our data suggests that the hippocampus only modulates timing when context is relevant.
Collapse
Affiliation(s)
- Benjamin J De Corte
- Department of Psychiatry, Columbia University, New York, NY, USA; New York State Psychiatric Institute, New York, NY, USA
| | - Sean J Farley
- Department of Psychological and Brain Sciences, The University of Iowa, Iowa City, IA, USA
| | - Kelsey A Heslin
- Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Krystal L Parker
- Department of Psychiatry, The University of Iowa, Iowa City, IA, USA
| | - John H Freeman
- Department of Psychological and Brain Sciences, The University of Iowa, Iowa City, IA, USA.
| |
Collapse
|
31
|
Wassum KM. Amygdala-cortical collaboration in reward learning and decision making. eLife 2022; 11:80926. [PMID: 36062909 PMCID: PMC9444241 DOI: 10.7554/elife.80926] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Accepted: 08/22/2022] [Indexed: 12/16/2022] Open
Abstract
Adaptive reward-related decision making requires accurate prospective consideration of the specific outcome of each option and its current desirability. These mental simulations are informed by stored memories of the associative relationships that exist within an environment. In this review, I discuss recent investigations of the function of circuitry between the basolateral amygdala (BLA) and lateral (lOFC) and medial (mOFC) orbitofrontal cortex in the learning and use of associative reward memories. I draw conclusions from data collected using sophisticated behavioral approaches to diagnose the content of appetitive memory in combination with modern circuit dissection tools. I propose that, via their direct bidirectional connections, the BLA and OFC collaborate to help us encode detailed, outcome-specific, state-dependent reward memories and to use those memories to enable the predictions and inferences that support adaptive decision making. Whereas lOFC→BLA projections mediate the encoding of outcome-specific reward memories, mOFC→BLA projections regulate the ability to use these memories to inform reward pursuit decisions. BLA projections to lOFC and mOFC both contribute to using reward memories to guide decision making. The BLA→lOFC pathway mediates the ability to represent the identity of a specific predicted reward and the BLA→mOFC pathway facilitates understanding of the value of predicted events. Thus, I outline a neuronal circuit architecture for reward learning and decision making and provide new testable hypotheses as well as implications for both adaptive and maladaptive decision making.
Collapse
Affiliation(s)
- Kate M Wassum
- Department of Psychology, University of California, Los Angeles, Los Angeles, United States.,Brain Research Institute, University of California, Los Angeles, Los Angeles, United States.,Integrative Center for Learning and Memory, University of California, Los Angeles, Los Angeles, United States.,Integrative Center for Addictive Disorders, University of California, Los Angeles, Los Angeles, United States
| |
Collapse
|
32
|
Bo O'Connor B, Fowler Z. How Imagination and Memory Shape the Moral Mind. PERSONALITY AND SOCIAL PSYCHOLOGY REVIEW 2022; 27:226-249. [PMID: 36062349 DOI: 10.1177/10888683221114215] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Interdisciplinary research has proposed a multifaceted view of human cognition and morality, establishing that inputs from multiple cognitive and affective processes guide moral decisions. However, extant work on moral cognition has largely overlooked the contributions of episodic representation. The ability to remember or imagine a specific moment in time plays a broadly influential role in cognition and behavior. Yet, existing research has only begun exploring the influence of episodic representation on moral cognition. Here, we evaluate the theoretical connections between episodic representation and moral cognition, review emerging empirical work revealing how episodic representation affects moral decision-making, and conclude by highlighting gaps in the literature and open questions. We argue that a comprehensive model of moral cognition will require including the episodic memory system, further delineating its direct influence on moral thought, and better understanding its interactions with other mental processes to fundamentally shape our sense of right and wrong.
Collapse
Affiliation(s)
| | - Zoë Fowler
- University at Albany, State University of New York, USA
| |
Collapse
|
33
|
Maria Manuel Vianny D, John A, Kumar Mohan S, Sarlan A, Adimoolam, Ahmadian A. Water optimization technique for precision irrigation system using IoT and machine learning. SUSTAINABLE ENERGY TECHNOLOGIES AND ASSESSMENTS 2022; 52:102307. [DOI: 10.1016/j.seta.2022.102307] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/15/2023]
|
34
|
Wu Y, Morita M, Izawa J. Reward prediction errors, not sensory prediction errors, play a major role in model selection in human reinforcement learning. Neural Netw 2022; 154:109-121. [PMID: 35872516 DOI: 10.1016/j.neunet.2022.07.002] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2021] [Revised: 03/25/2022] [Accepted: 07/06/2022] [Indexed: 10/17/2022]
Abstract
Model-based reinforcement learning enables an agent to learn in variable environments and tasks by optimizing its actions based on the predicted states and outcomes. This mechanism has also been considered in the brain. However, exactly how the brain selects an appropriate model for confronting environments has remained unclear. Here, we investigated the model selection algorithm in the human brain during a reinforcement learning task. One primary theory of model selection in the brain is based on sensory prediction errors. Here, we compared this theory with an alternative possibility of internal model selection with reward prediction errors. To compare these two theories, we devised a switching experiment from a first-order Markov decision process to a second-order Markov decision process that provides either reward- or sensory prediction error regarding environmental change. We tested two representative computational models driven by different prediction errors. One is the sensory prediction-error-driven Bayesian algorithm, which has been discussed as a representative internal model selection algorithm in the animal reinforcement learning task. The other is the reward-prediction-error-driven policy gradient algorithm. We compared the simulation results of these two computational models with human reinforcement learning behaviors. The model fitting result supports that the policy gradient algorithm is preferable to the Bayesian algorithm. This suggests that the human brain employs the reward prediction error to select an appropriate internal model in the reinforcement learning task.
Collapse
Affiliation(s)
- Yihao Wu
- School of Integrative and Global Majors, University of Tsukuba, Tennodai 1-1-1, Tsukuba, Ibaraki, 305-8573, Japan.
| | - Masahiko Morita
- Faculty of Engineering, Information and Systems, University of Tsukuba, Tennodai 1-1-1, Tsukuba, Ibaraki, 305-8573, Japan.
| | - Jun Izawa
- Faculty of Engineering, Information and Systems, University of Tsukuba, Tennodai 1-1-1, Tsukuba, Ibaraki, 305-8573, Japan.
| |
Collapse
|
35
|
Wagner B, Mathar D, Peters J. Gambling Environment Exposure Increases Temporal Discounting but Improves Model-Based Control in Regular Slot-Machine Gamblers. COMPUTATIONAL PSYCHIATRY (CAMBRIDGE, MASS.) 2022; 6:142-165. [PMID: 38774777 PMCID: PMC11104401 DOI: 10.5334/cpsy.84] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/14/2021] [Accepted: 05/26/2022] [Indexed: 11/20/2022]
Abstract
Gambling disorder is a behavioral addiction that negatively impacts personal finances, work, relationships and mental health. In this pre-registered study (https://osf.io/5ptz9/) we investigated the impact of real-life gambling environments on two computational markers of addiction, temporal discounting and model-based reinforcement learning. Gambling disorder is associated with increased temporal discounting and reduced model-based learning. Regular gamblers (n = 30, DSM-5 score range 3-9) performed both tasks in a neutral (café) and a gambling-related environment (slot-machine venue) in counterbalanced order. Data were modeled using drift diffusion models for temporal discounting and reinforcement learning via hierarchical Bayesian estimation. Replicating previous findings, gamblers discounted rewards more steeply in the gambling-related context. This effect was positively correlated with gambling related cognitive distortions (pre-registered analysis). In contrast to our pre-registered hypothesis, model-based reinforcement learning was improved in the gambling context. Here we show that temporal discounting and model-based reinforcement learning are modulated in opposite ways by real-life gambling cue exposure. Results challenge aspects of habit theories of addiction, and reveal that laboratory-based computational markers of psychopathology are under substantial contextual control.
Collapse
Affiliation(s)
- Ben Wagner
- Department of Psychology, Biological Psychology, University of Cologne, Cologne, Germany
- Faculty of Psychology, Chair of Neuroimaging, Technical University of Dresden, Dresden, Germany
| | - David Mathar
- Department of Psychology, Biological Psychology, University of Cologne, Cologne, Germany
| | - Jan Peters
- Department of Psychology, Biological Psychology, University of Cologne, Cologne, Germany
| |
Collapse
|
36
|
Abstract
People with damage to the orbitofrontal cortex (OFC) have specific problems making decisions, whereas their other cognitive functions are spared. Neurophysiological studies have shown that OFC neurons fire in proportion to the value of anticipated outcomes. Thus, a central role of the OFC is to guide optimal decision-making by signalling values associated with different choices. Until recently, this view of OFC function dominated the field. New data, however, suggest that the OFC may have a much broader role in cognition by representing cognitive maps that can be used to guide behaviour and that value is just one of many variables that are important for behavioural control. In this Review, we critically evaluate these two alternative accounts of OFC function and examine how they might be reconciled.
Collapse
Affiliation(s)
- Eric B Knudsen
- Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, California, USA
| | - Joni D Wallis
- Department of Psychology and Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, California, USA.
| |
Collapse
|
37
|
Deep imagination is a close to optimal policy for planning in large decision trees under limited resources. Sci Rep 2022; 12:10411. [PMID: 35729320 PMCID: PMC9213460 DOI: 10.1038/s41598-022-13862-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2021] [Accepted: 05/30/2022] [Indexed: 11/08/2022] Open
Abstract
Many decisions involve choosing an uncertain course of action in deep and wide decision trees, as when we plan to visit an exotic country for vacation. In these cases, exhaustive search for the best sequence of actions is not tractable due to the large number of possibilities and limited time or computational resources available to make the decision. Therefore, planning agents need to balance breadth-considering many actions in the first few tree levels-and depth-considering many levels but few actions in each of them-to allocate optimally their finite search capacity. We provide efficient analytical solutions and numerical analysis to the problem of allocating finite sampling capacity in one shot to infinitely large decision trees, both in the time discounted and undiscounted cases. We find that in general the optimal policy is to allocate few samples per level so that deep levels can be reached, thus favoring depth over breadth search. In contrast, in poor environments and at low capacity, it is best to broadly sample branches at the cost of not sampling deeply, although this policy is marginally better than deep allocations. Our results can provide a theoretical foundation for why human reasoning is pervaded by imagination-based processes.
Collapse
|
38
|
Parker NF, Baidya A, Cox J, Haetzel LM, Zhukovskaya A, Murugan M, Engelhard B, Goldman MS, Witten IB. Choice-selective sequences dominate in cortical relative to thalamic inputs to NAc to support reinforcement learning. Cell Rep 2022; 39:110756. [PMID: 35584665 PMCID: PMC9218875 DOI: 10.1016/j.celrep.2022.110756] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2019] [Revised: 02/18/2022] [Accepted: 04/07/2022] [Indexed: 11/25/2022] Open
Abstract
How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens, which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex and midline regions of the thalamus. However, little is known about whether and how representations differ across these input pathways. By comparing these inputs during a reinforcement learning task in mice, we discovered that prelimbic cortical inputs preferentially represent actions and choices, whereas midline thalamic inputs preferentially represent cues. Choice-selective activity in the prelimbic cortical inputs is organized in sequences that persist beyond the outcome. Through computational modeling, we demonstrate that these sequences can support the neural implementation of reinforcement-learning algorithms, in both a circuit model based on synaptic plasticity and one based on neural dynamics. Finally, we test and confirm a prediction of our circuit models by direct manipulation of nucleus accumbens input neurons.
Collapse
Affiliation(s)
- Nathan F Parker
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Avinash Baidya
- Center for Neuroscience, University of California, Davis, Davis, CA 95616, USA; Department of Physics and Astronomy, University of California, Davis, Davis, CA 95616, USA
| | - Julia Cox
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA; Department of Neuroscience, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
| | - Laura M Haetzel
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Anna Zhukovskaya
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Malavika Murugan
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Ben Engelhard
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Mark S Goldman
- Center for Neuroscience, University of California, Davis, Davis, CA 95616, USA; Department of Neurobiology, Physiology and Behavior, University of California, Davis, Davis, CA 95616, USA; Department of Ophthalmology and Vision Science, University of California, Davis, Davis, CA 95616, USA.
| | - Ilana B Witten
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA; Department of Psychology, Princeton University, Princeton, NJ 08544, USA.
| |
Collapse
|
39
|
Lubianiker N, Paret C, Dayan P, Hendler T. Neurofeedback through the lens of reinforcement learning. Trends Neurosci 2022; 45:579-593. [PMID: 35550813 DOI: 10.1016/j.tins.2022.03.008] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2021] [Revised: 02/11/2022] [Accepted: 03/24/2022] [Indexed: 11/29/2022]
Abstract
Despite decades of experimental and clinical practice, the neuropsychological mechanisms underlying neurofeedback (NF) training remain obscure. NF is a unique form of reinforcement learning (RL) task, during which participants are provided with rewarding feedback regarding desired changes in neural patterns. However, key RL considerations - including choices during practice, prediction errors, credit-assignment problems, or the exploration-exploitation tradeoff - have infrequently been considered in the context of NF. We offer an RL-based framework for NF, describing different internal states, actions, and rewards in common NF protocols, thus fashioning new proposals for characterizing, predicting, and hastening the course of learning. In this way we hope to advance current understanding of neural regulation via NF, and ultimately to promote its effectiveness, personalization, and clinical utility.
Collapse
Affiliation(s)
- Nitzan Lubianiker
- School of Psychological Sciences, Gershon H. Gordon Faculty of Social Sciences, Tel Aviv University, Tel Aviv, Israel; Sagol Brain Institute, Wohl Institute for Advanced Imaging, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel.
| | - Christian Paret
- School of Psychological Sciences, Gershon H. Gordon Faculty of Social Sciences, Tel Aviv University, Tel Aviv, Israel; Sagol Brain Institute, Wohl Institute for Advanced Imaging, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel; Department of Psychosomatic Medicine and Psychotherapy, Central Institute of Mental Health Mannheim, Medical Faculty Mannheim/Heidelberg University, Mannheim, Germany
| | - Peter Dayan
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany; University of Tübingen, Tübingen, Germany
| | - Talma Hendler
- School of Psychological Sciences, Gershon H. Gordon Faculty of Social Sciences, Tel Aviv University, Tel Aviv, Israel; Sagol Brain Institute, Wohl Institute for Advanced Imaging, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel; Sagol school of Neuroscience, Tel Aviv University, Tel Aviv, Israel; Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
| |
Collapse
|
40
|
Morris RW, Dezfouli A, Griffiths KR, Le Pelley ME, Balleine BW. The Neural Bases of Action-Outcome Learning in Humans. J Neurosci 2022; 42:3636-3647. [PMID: 35296548 PMCID: PMC9053851 DOI: 10.1523/jneurosci.1079-21.2022] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2021] [Revised: 02/17/2022] [Accepted: 02/22/2022] [Indexed: 11/21/2022] Open
Abstract
From an associative perspective the acquisition of new goal-directed actions requires the encoding of specific action-outcome (AO) associations and, therefore, sensitivity to the validity of an action as a predictor of a specific outcome relative to other events. Although competitive architectures have been proposed within associative learning theory to achieve this kind of identity-based selection, whether and how these architectures are implemented by the brain is still a matter of conjecture. To investigate this issue, we trained human participants to encode various AO associations while undergoing functional neuroimaging (fMRI). We then degraded one AO contingency by increasing the probability of the outcome in the absence of its associated action while keeping other AO contingencies intact. We found that this treatment selectively reduced performance of the degraded action. Furthermore, when a signal predicted the unpaired outcome, performance of the action was restored, suggesting that the degradation effect reflects competition between the action and the context for prediction of the specific outcome. We used a Kalman filter to model the contribution of different causal variables to AO learning and found that activity in the medial prefrontal cortex (mPFC) and the dorsal anterior cingulate cortex (dACC) tracked changes in the association of the action and context, respectively, with regard to the specific outcome. Furthermore, we found the mPFC participated in a network with the striatum and posterior parietal cortex to segregate the influence of the various competing predictors to establish specific AO associations.SIGNIFICANCE STATEMENT Humans and other animals learn the consequences of their actions, allowing them to control their environment in a goal-directed manner. Nevertheless, it is unknown how we parse environmental causes from the effects of our own actions to establish these specific action-outcome (AO) relationships. Here, we show that the brain learns the causal structure of the environment by segregating the unique influence of actions from other causes in the medial prefrontal and anterior cingulate cortices and, through a network of structures, including the caudate nucleus and posterior parietal cortex, establishes the distinct causal relationships from which specific AO associations are formed.
Collapse
Affiliation(s)
- Richard W Morris
- Centre for Translational Data Science, University of Sydney, Sydney, NSW 2006, Australia
| | - Amir Dezfouli
- Data61, Commonwealth Scientific and Industrial Research Organisation, Sydney, NSW 2015, Australia
| | - Kristi R Griffiths
- Brain Dynamics Centre, Westmead Institute for Medical Research, University of Sydney, Sydney, NSW 2145, Australia
| | - Mike E Le Pelley
- School of Psychology, University of New South Wales Sydney, Sydney, NSW 2052, Australia
| | - Bernard W Balleine
- School of Psychology, University of New South Wales Sydney, Sydney, NSW 2052, Australia
| |
Collapse
|
41
|
Better, worse, or different than expected: on the role of value and identity prediction errors in fear memory reactivation. Sci Rep 2022; 12:5862. [PMID: 35393469 PMCID: PMC8988102 DOI: 10.1038/s41598-022-09720-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2021] [Accepted: 03/11/2022] [Indexed: 11/08/2022] Open
Abstract
Although reconsolidation-based interventions constitute a promising new avenue to treating fear and anxieties disorders, the success of the intervention is not guaranteed. The initiation of memory reconsolidation is dependent on whether a mismatch between the experienced and predicted outcome-a prediction error (PE)-occurs during fear memory reactivation. It remains, however, elusive whether any type of PE renders fear memories susceptible to reconsolidation disruption. Here, we investigated whether a value PE, elicited by an outcome that is better or worse than expected, is necessary to make fear memories susceptible to reconsolidation disruption or whether a model-based identity PE, i.e., a PE elicited by an outcome equally aversive but different than expected, would be sufficient. Blocking beta-adrenergic receptors with propranolol HCl after reactivation did, however, not reduce the expression of fear after either type of PE. Instead, we observed intact fear memory expression 24 h after reactivation in the value-, identity- and a no-PE control group. The present results do not corroborate our earlier findings of reconsolidation disruption and point towards challenges that the field is currently facing in observing evidence for memory reconsolidation at all. We provide potential explanations for the unexpected failure of replicating reconsolidation disruption and discuss future directions.
Collapse
|
42
|
The role of causal structure in implicit evaluation. Cognition 2022; 225:105116. [PMID: 35397347 DOI: 10.1016/j.cognition.2022.105116] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2021] [Revised: 03/24/2022] [Accepted: 03/28/2022] [Indexed: 11/23/2022]
Abstract
Causal relationships, unlike mere co-occurrence, allow humans to obtain rewards and avoid punishments by intervening on their environment. Accordingly, explicit (controlled) evaluations of stimuli encountered in the environment are known to be sensitive to causal relationships above and beyond mere co-occurrence. In this project, we conduct stringent tests of whether implicit (automatic) evaluation also reflects causal relationships and begin to probe the representational mechanisms underlying such sensitivity. Participants (N = 4836) observed causal events during which two stimuli were equally contingent with positive or negative outcomes but only one of them was causally responsible for these outcomes. Across 6 studies, varying in design and amount of verbal scaffolding provided, differences in causal status consistently guided not only explicit measures of evaluation (Likert and slider scales; Bayes Factor meta-analysis: Cohen's d = 0.28, BF10 > 1046) but also their implicit counterparts (Implicit Association Tests; Bayes Factor meta-analysis: Cohen's d = 0.22, BF10 > 1029). However, unlike their explicit counterparts, implicit evaluations were not sensitive to causal relationships that had to be flexibly derived by combining disparate past experiences. Taken together, these studies suggest that implicit evaluations are sensitive to causal information. Such sensitivity seems to be mediated via precompiled, causally informed value representations rather than online computations over a causal model.
Collapse
|
43
|
Abstract
In human neuroscience, studies of cognition are rarely grounded in non-task-evoked, 'spontaneous' neural activity. Indeed, studies of spontaneous activity tend to focus predominantly on intrinsic neural patterns (for example, resting-state networks). Taking a 'representation-rich' approach bridges the gap between cognition and resting-state communities: this approach relies on decoding task-related representations from spontaneous neural activity, allowing quantification of the representational content and rich dynamics of such activity. For example, if we know the neural representation of an episodic memory, we can decode its subsequent replay during rest. We argue that such an approach advances cognitive research beyond a focus on immediate task demand and provides insight into the functional relevance of the intrinsic neural pattern (for example, the default mode network). This in turn enables a greater integration between human and animal neuroscience, facilitating experimental testing of theoretical accounts of intrinsic activity, and opening new avenues of research in psychiatry.
Collapse
|
44
|
Sensorimotor Learning in Response to Errors in Task Performance. eNeuro 2022; 9:ENEURO.0371-21.2022. [PMID: 35110383 PMCID: PMC8938978 DOI: 10.1523/eneuro.0371-21.2022] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Revised: 01/18/2022] [Accepted: 01/23/2022] [Indexed: 01/09/2023] Open
Abstract
The human sensorimotor system is sensitive to both limb-related prediction errors and task-related performance errors. Prediction error signals are believed to drive implicit refinements to motor plans. However, an understanding of the mechanisms that performance errors stimulate has remained unclear largely because their effects have not been probed in isolation from prediction errors. Diverging from past work, we induced performance errors independent of prediction errors by shifting the location of a reach target but keeping the intended and actual kinematic consequences of the motion matched. Our first two experiments revealed that rather than implicit learning, motor adjustments in response to performance errors reflect the use of deliberative, volitional strategies. Our third experiment revealed a potential dissociation of performance-error-driven strategies based on error size. Specifically, behavioral changes following large errors were consistent with goal-directed or model-based control, known to be supported by connections between prefrontal cortex and associative striatum. In contrast, motor changes following smaller performance errors carried signatures of model-free stimulus-response learning, of the kind underpinned by pathways between motor cortical areas and sensorimotor striatum. Across all experiments, we also found remarkably faster re-learning, advocating that such “savings” is associated with retrieval of previously learned strategic error compensation and may not require a history of exposure to limb-related errors.
Collapse
|
45
|
Sebold M, Chen H, Önal A, Kuitunen-Paul S, Mojtahedzadeh N, Garbusow M, Nebe S, Wittchen HU, Huys QJM, Schlagenhauf F, Rapp MA, Smolka MN, Heinz A. Stronger Prejudices Are Associated With Decreased Model-Based Control. Front Psychol 2022; 12:767022. [PMID: 35069341 PMCID: PMC8767058 DOI: 10.3389/fpsyg.2021.767022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2021] [Accepted: 11/29/2021] [Indexed: 12/01/2022] Open
Abstract
Background: Prejudices against minorities can be understood as habitually negative evaluations that are kept in spite of evidence to the contrary. Therefore, individuals with strong prejudices might be dominated by habitual or "automatic" reactions at the expense of more controlled reactions. Computational theories suggest individual differences in the balance between habitual/model-free and deliberative/model-based decision-making. Methods: 127 subjects performed the two Step task and completed the blatant and subtle prejudice scale. Results: By using analyses of choices and reaction times in combination with computational modeling, subjects with stronger blatant prejudices showed a shift away from model-based control. There was no association between these decision-making processes and subtle prejudices. Conclusion: These results support the idea that blatant prejudices toward minorities are related to a relative dominance of habitual decision-making. This finding has important implications for developing interventions that target to change prejudices across societies.
Collapse
Affiliation(s)
- Miriam Sebold
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany
- Department for Social and Preventive Medicine, University of Potsdam, Potsdam, Germany
| | - Hao Chen
- Department of Psychiatry, Neuroimaging Center, Technische Universität Dresden, Dresden, Germany
| | - Aleyna Önal
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany
| | - Sören Kuitunen-Paul
- Institute of Clinical Psychology and Psychotherapy, Technische Universität Dresden, Dresden, Germany
- Department of Child and Adolescent Psychiatry, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| | - Negin Mojtahedzadeh
- Department of Psychiatry, Neuroimaging Center, Technische Universität Dresden, Dresden, Germany
| | - Maria Garbusow
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany
| | - Stephan Nebe
- Department of Economics, Zurich Center for Neuroeconomics, University of Zurich, Zurich, Switzerland
| | - Hans-Ulrich Wittchen
- Department of Psychiatry and Psychotherapy, Ludwig-Maximilians-Universität München, Munich, Germany
| | - Quentin J M Huys
- Division of Psychiatry, University College London, London, United Kingdom
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom
| | - Florian Schlagenhauf
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Michael A Rapp
- Department for Social and Preventive Medicine, University of Potsdam, Potsdam, Germany
| | - Michael N Smolka
- Department of Psychiatry, Neuroimaging Center, Technische Universität Dresden, Dresden, Germany
| | - Andreas Heinz
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany
| |
Collapse
|
46
|
Averbeck B, O'Doherty JP. Reinforcement-learning in fronto-striatal circuits. Neuropsychopharmacology 2022; 47:147-162. [PMID: 34354249 PMCID: PMC8616931 DOI: 10.1038/s41386-021-01108-0] [Citation(s) in RCA: 34] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Revised: 07/06/2021] [Accepted: 07/09/2021] [Indexed: 01/03/2023]
Abstract
We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.
Collapse
Affiliation(s)
| | - John P O'Doherty
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA.
| |
Collapse
|
47
|
Kim D, Jeong J, Lee SW. Prefrontal solution to the bias-variance tradeoff during reinforcement learning. Cell Rep 2021; 37:110185. [PMID: 34965420 DOI: 10.1016/j.celrep.2021.110185] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2021] [Revised: 08/09/2021] [Accepted: 12/07/2021] [Indexed: 11/17/2022] Open
Abstract
Evidence that the brain combines different value learning strategies to minimize prediction error is accumulating. However, the tradeoff between bias and variance error, which imposes different constraints on each learning strategy's performance, poses a challenge for value learning. While this tradeoff specifies the requirements for optimal learning, little has been known about how the brain deals with this issue. Here, we hypothesize that the brain adaptively resolves the bias-variance tradeoff during reinforcement learning. Our theory suggests that the solution necessitates baseline correction for prediction error, which offsets the adverse effects of irreducible error on value learning. We show behavioral evidence of adaptive control using a Markov decision task with context changes. The prediction error baseline seemingly signals context changes to improve adaptability. Critically, we identify multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free reinforcement learning.
Collapse
Affiliation(s)
- Dongjae Kim
- Center for Neural Science, New York University, New York, NY, USA; Department of Psychology, New York University, New York, NY, USA
| | - Jaeseung Jeong
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; KAIST Center for Neuroscience-inspired AI, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; KI for Health Science and Technology, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; KI for Artificial Intelligence, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea.
| |
Collapse
|
48
|
Deserno L, Moran R, Michely J, Lee Y, Dayan P, Dolan RJ. Dopamine enhances model-free credit assignment through boosting of retrospective model-based inference. eLife 2021; 10:e67778. [PMID: 34882092 PMCID: PMC8758138 DOI: 10.7554/elife.67778] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2021] [Accepted: 12/08/2021] [Indexed: 11/13/2022] Open
Abstract
Dopamine is implicated in representing model-free (MF) reward prediction errors a as well as influencing model-based (MB) credit assignment and choice. Putative cooperative interactions between MB and MF systems include a guidance of MF credit assignment by MB inference. Here, we used a double-blind, placebo-controlled, within-subjects design to test an hypothesis that enhancing dopamine levels boosts the guidance of MF credit assignment by MB inference. In line with this, we found that levodopa enhanced guidance of MF credit assignment by MB inference, without impacting MF and MB influences directly. This drug effect correlated negatively with a dopamine-dependent change in purely MB credit assignment, possibly reflecting a trade-off between these two MB components of behavioural control. Our findings of a dopamine boost in MB inference guidance of MF learning highlight a novel DA influence on MB-MF cooperative interactions.
Collapse
Affiliation(s)
- Lorenz Deserno
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College LondonLondonUnited Kingdom
- The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College LondonLondonUnited Kingdom
- Department of Child and Adolescent Psychiatry, Psychotherapy and Psychosomatics, University of WürzburgWürzburgGermany
- Department of Psychiatry and Psychotherapy, Technische Universität DresdenDresdenGermany
| | - Rani Moran
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College LondonLondonUnited Kingdom
- The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College LondonLondonUnited Kingdom
| | - Jochen Michely
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College LondonLondonUnited Kingdom
- The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College LondonLondonUnited Kingdom
- Department of Psychiatry and Psychotherapy, Charité Universitätsmedizin BerlinBerlinGermany
| | - Ying Lee
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College LondonLondonUnited Kingdom
- The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College LondonLondonUnited Kingdom
- Department of Psychiatry and Psychotherapy, Technische Universität DresdenDresdenGermany
| | - Peter Dayan
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College LondonLondonUnited Kingdom
- Max Planck Institute for Biological CyberneticsTübingenGermany
- University of TübingenTübingenGermany
| | - Raymond J Dolan
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College LondonLondonUnited Kingdom
- The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College LondonLondonUnited Kingdom
| |
Collapse
|
49
|
Liakoni V, Lehmann MP, Modirshanechi A, Brea J, Lutti A, Gerstner W, Preuschoff K. Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making. Neuroimage 2021; 246:118780. [PMID: 34875383 DOI: 10.1016/j.neuroimage.2021.118780] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2021] [Revised: 08/03/2021] [Accepted: 12/04/2021] [Indexed: 11/25/2022] Open
Abstract
Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.
Collapse
Affiliation(s)
- Vasiliki Liakoni
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland.
| | - Marco P Lehmann
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Alireza Modirshanechi
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Johanni Brea
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Antoine Lutti
- Laboratoire de recherche en neuroimagerie (LREN), Department of Clinical Neurosciences, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
| | - Wulfram Gerstner
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Kerstin Preuschoff
- Geneva Finance Research Institute & Interfaculty Center for Affective Sciences, University of Geneva, Geneva, Switzerland
| |
Collapse
|
50
|
van Ments L, Treur J. Reflections on dynamics, adaptation and control: A cognitive architecture for mental models. COGN SYST RES 2021. [DOI: 10.1016/j.cogsys.2021.06.004] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|