1
|
Brown CS, Devine S, Otto AR, Bischoff-Grethe A, Wierenga CE. Greater reliance on model-free learning in adolescent anorexia nervosa: An examination of dual-system reinforcement learning. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.01.31.24302097. [PMID: 38352608 PMCID: PMC10863009 DOI: 10.1101/2024.01.31.24302097] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/19/2024]
Abstract
Alterations in learning and decision-making systems are thought to contribute to core features of anorexia nervosa (AN), a psychiatric disorder characterized by persistent dietary restriction and weight loss. Instrumental learning theory identifies a dual-system of habit and goal-directed decision-making, linked to model-free and model-based reinforcement learning algorithms. Difficulty arbitrating between these systems, resulting in an over-reliance on one strategy over the other, has been implicated in compulsivity and extreme goal pursuit, both of which are observed in AN. Characterizing alterations in model-free and model-based systems, and their neural correlates, in AN may clarify mechanisms contributing to symptom heterogeneity (e.g., binge/purge symptoms). This study tested whether adolescents with restricting AN (AN-R; n = 36) and binge/purge AN (AN-BP; n = 20) differentially utilized model-based and model-free learning systems compared to a healthy control group (HC; n = 28) during a Markov two-step decision-making task under conditions of reward and punishment. Associations between model-free and model-based learning and resting-state functional connectivity between neural regions of interest, including orbitofrontal cortex (OFC), nucleus accumbens (NAcc), putamen, and sensory motor cortex (SMC) were examined. AN-R showed higher utilization of model-free learning compared to HC for reward, but attenuated model-free and model-based learning for punishment. In AN-R only, higher model-based learning was associated with stronger OFC-to-left NAcc functional connectivity, regions linked to goal-directed behavior. Greater utilization of model-free learning for reward in AN-R may differentiate this group, particularly during adolescence, and facilitate dietary restriction by prioritizing habitual control in rewarding contexts.
Collapse
Affiliation(s)
- Carina S. Brown
- San Diego State University/University of California San Diego Joint Doctoral Program in Clinical Psychology
- Department of Psychiatry, University of California, San Diego
| | | | | | | | - Christina E. Wierenga
- San Diego State University/University of California San Diego Joint Doctoral Program in Clinical Psychology
- Department of Psychiatry, University of California, San Diego
| |
Collapse
|
2
|
Mah A, Schiereck SS, Bossio V, Constantinople CM. Distinct value computations support rapid sequential decisions. Nat Commun 2023; 14:7573. [PMID: 37989741 PMCID: PMC10663503 DOI: 10.1038/s41467-023-43250-x] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2023] [Accepted: 11/03/2023] [Indexed: 11/23/2023] Open
Abstract
The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or cache values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.
Collapse
Affiliation(s)
- Andrew Mah
- Center for Neural Science, New York University, New York, NY, 10003, USA
| | | | - Veronica Bossio
- Center for Neural Science, New York University, New York, NY, 10003, USA
- Zuckerman Institute, Columbia University, New York, NY, 10027, USA
| | | |
Collapse
|
3
|
Proietti R, Pezzulo G, Tessari A. An active inference model of hierarchical action understanding, learning and imitation. Phys Life Rev 2023; 46:92-118. [PMID: 37354642 DOI: 10.1016/j.plrev.2023.05.012] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Accepted: 05/31/2023] [Indexed: 06/26/2023]
Abstract
We advance a novel active inference model of the cognitive processing that underlies the acquisition of a hierarchical action repertoire and its use for observation, understanding and imitation. We illustrate the model in four simulations of a tennis learner who observes a teacher performing tennis shots, forms hierarchical representations of the observed actions, and imitates them. Our simulations show that the agent's oculomotor activity implements an active information sampling strategy that permits inferring the kinematic aspects of the observed movement, which lie at the lowest level of the action hierarchy. In turn, this low-level kinematic inference supports higher-level inferences about deeper aspects of the observed actions: proximal goals and intentions. Finally, the inferred action representations can steer imitative responses, but interfere with the execution of different actions. Our simulations show that hierarchical active inference provides a unified account of action observation, understanding, learning and imitation and helps explain the neurobiological underpinnings of visuomotor cognition, including the multiple routes for action understanding in the dorsal and ventral streams and mirror mechanisms.
Collapse
Affiliation(s)
| | - Giovanni Pezzulo
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy.
| | - Alessia Tessari
- Department of Psychology, University of Bologna, Italy; Alma Mater Research Institute for Human-Centered Artificial Intelligence, University of Bologna, Bologna, Italy
| |
Collapse
|
4
|
Mah A, Schiereck SS, Bossio V, Constantinople CM. Distinct value computations support rapid sequential decisions. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.03.14.532617. [PMID: 36993514 PMCID: PMC10055073 DOI: 10.1101/2023.03.14.532617] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 04/29/2023]
Abstract
The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or "cache" values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.
Collapse
Affiliation(s)
- Andrew Mah
- Center for Neural Science, New York University; New York, NY 10003
| | | | - Veronica Bossio
- Center for Neural Science, New York University; New York, NY 10003
| | | |
Collapse
|
5
|
Yamada K, Toda K. Habit formation viewed as structural change in the behavioral network. Commun Biol 2023; 6:303. [PMID: 37016036 PMCID: PMC10073220 DOI: 10.1038/s42003-023-04500-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2022] [Accepted: 01/18/2023] [Indexed: 04/06/2023] Open
Abstract
Habit formation is a process in which an action becomes involuntary. While goal-directed behavior is driven by its consequences, habits are elicited by a situation rather than its consequences. Existing theories have proposed that actions are controlled by corresponding two distinct systems. Although canonical theories based on such distinctions are starting to be challenged, there are a few theoretical frameworks that implement goal-directed behavior and habits within a single system. Here, we propose a novel theoretical framework by hypothesizing that behavior is a network composed of several responses. With this framework, we have shown that the transition of goal-directed actions to habits is caused by a change in a single network structure. Furthermore, we confirmed that the proposed network model behaves in a manner consistent with the existing experimental results reported in animal behavioral studies. Our results revealed that habit could be formed under the control of a single system rather than two distinct systems. By capturing the behavior as a single network change, this framework provides a new perspective on studying the structure of the behavior for experimental and theoretical research.
Collapse
Affiliation(s)
- Kota Yamada
- Department of Psychology, Keio University, Tokyo, Japan.
- Japan Society for Promotion of Science, Tokyo, Japan.
| | - Koji Toda
- Department of Psychology, Keio University, Tokyo, Japan
| |
Collapse
|
6
|
Barnett WH, Kuznetsov A, Lapish CC. Distinct cortico-striatal compartments drive competition between adaptive and automatized behavior. PLoS One 2023; 18:e0279841. [PMID: 36943842 PMCID: PMC10030038 DOI: 10.1371/journal.pone.0279841] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2022] [Accepted: 12/15/2022] [Indexed: 03/23/2023] Open
Abstract
Cortical and basal ganglia circuits play a crucial role in the formation of goal-directed and habitual behaviors. In this study, we investigate the cortico-striatal circuitry involved in learning and the role of this circuitry in the emergence of inflexible behaviors such as those observed in addiction. Specifically, we develop a computational model of cortico-striatal interactions that performs concurrent goal-directed and habit learning. The model accomplishes this by distinguishing learning processes in the dorsomedial striatum (DMS) that rely on reward prediction error signals as distinct from the dorsolateral striatum (DLS) where learning is supported by salience signals. These striatal subregions each operate on unique cortical input: the DMS receives input from the prefrontal cortex (PFC) which represents outcomes, and the DLS receives input from the premotor cortex which determines action selection. Following an initial learning of a two-alternative forced choice task, we subjected the model to reversal learning, reward devaluation, and learning a punished outcome. Behavior driven by stimulus-response associations in the DLS resisted goal-directed learning of new reward feedback rules despite devaluation or punishment, indicating the expression of habit. We repeated these simulations after the impairment of executive control, which was implemented as poor outcome representation in the PFC. The degraded executive control reduced the efficacy of goal-directed learning, and stimulus-response associations in the DLS were even more resistant to the learning of new reward feedback rules. In summary, this model describes how circuits of the dorsal striatum are dynamically engaged to control behavior and how the impairment of executive control by the PFC enhances inflexible behavior.
Collapse
Affiliation(s)
- William H. Barnett
- Department of Psychology, Indiana University—Purdue University Indianapolis, Indianapolis, Indiana, United States of America
| | - Alexey Kuznetsov
- Department of Mathematics, Indiana University—Purdue University Indianapolis, Indianapolis, Indiana, United States of America
| | - Christopher C. Lapish
- Department of Psychology, Indiana University—Purdue University Indianapolis, Indianapolis, Indiana, United States of America
- Stark Neurosciences Research Institute, Indiana University—Purdue University Indianapolis, Indianapolis, Indiana, United States of America
| |
Collapse
|
7
|
Taylor J, MacNamara Á, Collins D. The 3Ps: A tool for coach observation. Front Sports Act Living 2023; 4:1066378. [PMID: 36742150 PMCID: PMC9895099 DOI: 10.3389/fspor.2022.1066378] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2022] [Accepted: 12/19/2022] [Indexed: 01/21/2023] Open
Abstract
There is growing recognition of the value of "in situ" coach development practice across a variety sporting contexts. Unfortunately, however, there remains a limited number of tools available with which to observe coaching practice. In this study, we pilot and test a quasi-systematic tool for observation in the form of the 3Ps. Drawing on a range of representational perspectives, the theoretically neutral labels of "procedure", "planning", and "process" were developed for the purpose of holistic observation. In order to test the tool, a group of experienced coach development practitioners (n = 10) integrated the tool into their practice over a 12-month programme of professional development. Those participants subsequently took part in semi-structured interviews, in which they expressed a strong sense of acceptability, perceiving effectiveness and positive opportunity cost. We propose that the 3Ps tool presents a holistic and practically useful means of observing coaches' professional judgment and decision making. We also suggest future directions for the researcher who seeks to generate evidence in a naturalistic coaching context.
Collapse
Affiliation(s)
- Jamie Taylor
- School of Health and Human Performance, Faculty of Science and Health, Dublin City University, Dublin, Ireland,Grey Matters Performance Ltd., Stratford upon Avon, United Kingdom,Moray House School of Education and Sport, The University of Edinburgh, Edinburgh, United Kingdom,Insight SFI Centre for Data Analytics, Dublin City University, Glasnevin, Dublin, Ireland,Correspondence: Jamie Taylor
| | - Áine MacNamara
- School of Health and Human Performance, Faculty of Science and Health, Dublin City University, Dublin, Ireland,Grey Matters Performance Ltd., Stratford upon Avon, United Kingdom
| | - Dave Collins
- School of Health and Human Performance, Faculty of Science and Health, Dublin City University, Dublin, Ireland,Grey Matters Performance Ltd., Stratford upon Avon, United Kingdom,Moray House School of Education and Sport, The University of Edinburgh, Edinburgh, United Kingdom
| |
Collapse
|
8
|
Lee JH, Leibo JZ, An SJ, Lee SW. Importance of prefrontal meta control in human-like reinforcement learning. Front Comput Neurosci 2022; 16:1060101. [PMID: 36618272 PMCID: PMC9811824 DOI: 10.3389/fncom.2022.1060101] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2022] [Accepted: 11/30/2022] [Indexed: 12/24/2022] Open
Abstract
Recent investigation on reinforcement learning (RL) has demonstrated considerable flexibility in dealing with various problems. However, such models often experience difficulty learning seemingly easy tasks for humans. To reconcile the discrepancy, our paper is focused on the computational benefits of the brain's RL. We examine the brain's ability to combine complementary learning strategies to resolve the trade-off between prediction performance, computational costs, and time constraints. The complex need for task performance created by a volatile and/or multi-agent environment motivates the brain to continually explore an ideal combination of multiple strategies, called meta-control. Understanding these functions would allow us to build human-aligned RL models.
Collapse
Affiliation(s)
- Jee Hang Lee
- Department of Human-Centered Artificial Intelligence, Sangmyung University, Seoul, South Korea
| | | | - Su Jin An
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
- Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
- KAIST Center for Neuroscience-Inspired Artificial Intelligence, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
- KAIST Institute for Health Science and Technology, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
- KAIST Institute for Artificial Intelligence, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
| |
Collapse
|
9
|
Borghi AM, Shaki S, Fischer MH. Abstract concepts: external influences, internal constraints, and methodological issues. PSYCHOLOGICAL RESEARCH 2022; 86:2370-2388. [PMID: 35788903 PMCID: PMC9674746 DOI: 10.1007/s00426-022-01698-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
There is a longstanding and widely held misconception about the relative remoteness of abstract concepts from concrete experiences. This review examines the current evidence for external influences and internal constraints on the processing, representation, and use of abstract concepts, like truth, friendship, and number. We highlight the theoretical benefit of distinguishing between grounded and embodied cognition and then ask which roles do perception, action, language, and social interaction play in acquiring, representing and using abstract concepts. By reviewing several studies, we show that they are, against the accepted definition, not detached from perception and action. Focussing on magnitude-related concepts, we also discuss evidence for cultural influences on abstract knowledge and explore how internal processes such as inner speech, metacognition, and inner bodily signals (interoception) influence the acquisition and retrieval of abstract knowledge. Finally, we discuss some methodological developments. Specifically, we focus on the importance of studies that investigate the time course of conceptual processing and we argue that, because of the paramount role of sociality for abstract concepts, new methods are necessary to study concepts in interactive situations. We conclude that bodily, linguistic, and social constraints provide important theoretical limitations for our theories of conceptual knowledge.
Collapse
Affiliation(s)
- Anna M Borghi
- Department of Dynamic and Clinical Psychology, and Health Studies, Sapienza University of Rome, 00185, Rome, Italy.
- Institute of Cognitive Sciences and Technologies, Italian National Research Council, 00185, Rome, Italy.
| | - Samuel Shaki
- Department of Behavioral Sciences, Ariel University, 44837, Ariel, Israel
| | | |
Collapse
|
10
|
Deane G. Machines That Feel and Think: The Role of Affective Feelings and Mental Action in (Artificial) General Intelligence. ARTIFICIAL LIFE 2022; 28:289-309. [PMID: 35881678 DOI: 10.1162/artl_a_00368] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
What role do affective feelings (feelings/emotions/moods) play in adaptive behaviour? What are the implications of this for understanding and developing artificial general intelligence? Leading theoretical models of brain function are beginning to shed light on these questions. While artificial agents have excelled within narrowly circumscribed and specialised domains, domain-general intelligence has remained an elusive goal in artificial intelligence research. By contrast, humans and nonhuman animals are characterised by a capacity for flexible behaviour and general intelligence. In this article I argue that computational models of mental phenomena in predictive processing theories of the brain are starting to reveal the mechanisms underpinning domain-general intelligence in biological agents, and can inform the understanding and development of artificial general intelligence. I focus particularly on approaches to computational phenomenology in the active inference framework. Specifically, I argue that computational mechanisms of affective feelings in active inference-affective self-modelling-are revealing of how biological agents are able to achieve flexible behavioural repertoires and general intelligence. I argue that (i) affective self-modelling functions to "tune" organisms to the most tractable goals in the environmental context; and (ii) affective and agentic self-modelling is central to the capacity to perform mental actions in goal-directed imagination and creative cognition. I use this account as a basis to argue that general intelligence of the level and kind found in biological agents will likely require machines to be implemented with analogues of affective self-modelling.
Collapse
Affiliation(s)
- George Deane
- University of Edinburgh, School of Philosophy, Psychology, and Language Sciences.
| |
Collapse
|
11
|
Freezing revisited: coordinated autonomic and central optimization of threat coping. Nat Rev Neurosci 2022; 23:568-580. [PMID: 35760906 DOI: 10.1038/s41583-022-00608-2] [Citation(s) in RCA: 39] [Impact Index Per Article: 19.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/20/2022] [Indexed: 12/16/2022]
Abstract
Animals have sophisticated mechanisms for coping with danger. Freezing is a unique state that, upon threat detection, allows evidence to be gathered, response possibilities to be previsioned and preparations to be made for worst-case fight or flight. We propose that - rather than reflecting a passive fear state - the particular somatic and cognitive characteristics of freezing help to conceal overt responses, while optimizing sensory processing and action preparation. Critical for these functions are the neurotransmitters noradrenaline and acetylcholine, which modulate neural information processing and also control the sympathetic and parasympathetic branches of the autonomic nervous system. However, the interactions between autonomic systems and the brain during freezing, and the way in which they jointly coordinate responses, remain incompletely explored. We review the joint actions of these systems and offer a novel computational framework to describe their temporally harmonized integration. This reconceptualization of freezing has implications for its role in decision-making under threat and for psychopathology.
Collapse
|
12
|
Massi E, Barthélemy J, Mailly J, Dromnelle R, Canitrot J, Poniatowski E, Girard B, Khamassi M. Model-Based and Model-Free Replay Mechanisms for Reinforcement Learning in Neurorobotics. Front Neurorobot 2022; 16:864380. [PMID: 35812782 PMCID: PMC9263850 DOI: 10.3389/fnbot.2022.864380] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2022] [Accepted: 05/05/2022] [Indexed: 11/22/2022] Open
Abstract
Experience replay is widely used in AI to bootstrap reinforcement learning (RL) by enabling an agent to remember and reuse past experiences. Classical techniques include shuffled-, reversed-ordered- and prioritized-memory buffers, which have different properties and advantages depending on the nature of the data and problem. Interestingly, recent computational neuroscience work has shown that these techniques are relevant to model hippocampal reactivations recorded during rodent navigation. Nevertheless, the brain mechanisms for orchestrating hippocampal replay are still unclear. In this paper, we present recent neurorobotics research aiming to endow a navigating robot with a neuro-inspired RL architecture (including different learning strategies, such as model-based (MB) and model-free (MF), and different replay techniques). We illustrate through a series of numerical simulations how the specificities of robotic experimentation (e.g., autonomous state decomposition by the robot, noisy perception, state transition uncertainty, non-stationarity) can shed new lights on which replay techniques turn out to be more efficient in different situations. Finally, we close the loop by raising new hypotheses for neuroscience from such robotic models of hippocampal replay.
Collapse
|
13
|
Albarracin M, Demekas D, Ramstead MJD, Heins C. Epistemic Communities under Active Inference. ENTROPY (BASEL, SWITZERLAND) 2022; 24:476. [PMID: 35455140 PMCID: PMC9027706 DOI: 10.3390/e24040476] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/04/2022] [Revised: 03/11/2022] [Accepted: 03/24/2022] [Indexed: 02/04/2023]
Abstract
The spread of ideas is a fundamental concern of today's news ecology. Understanding the dynamics of the spread of information and its co-option by interested parties is of critical importance. Research on this topic has shown that individuals tend to cluster in echo-chambers and are driven by confirmation bias. In this paper, we leverage the active inference framework to provide an in silico model of confirmation bias and its effect on echo-chamber formation. We build a model based on active inference, where agents tend to sample information in order to justify their own view of reality, which eventually leads to them to have a high degree of certainty about their own beliefs. We show that, once agents have reached a certain level of certainty about their beliefs, it becomes very difficult to get them to change their views. This system of self-confirming beliefs is upheld and reinforced by the evolving relationship between an agent's beliefs and observations, which over time will continue to provide evidence for their ingrained ideas about the world. The epistemic communities that are consolidated by these shared beliefs, in turn, tend to produce perceptions of reality that reinforce those shared beliefs. We provide an active inference account of this community formation mechanism. We postulate that agents are driven by the epistemic value that they obtain from sampling or observing the behaviours of other agents. Inspired by digital social networks like Twitter, we build a generative model in which agents generate observable social claims or posts (e.g., 'tweets') while reading the socially observable claims of other agents that lend support to one of two mutually exclusive abstract topics. Agents can choose which other agent they pay attention to at each timestep, and crucially who they attend to and what they choose to read influences their beliefs about the world. Agents also assess their local network's perspective, influencing which kinds of posts they expect to see other agents making. The model was built and simulated using the freely available Python package pymdp. The proposed active inference model can reproduce the formation of echo-chambers over social networks, and gives us insight into the cognitive processes that lead to this phenomenon.
Collapse
Affiliation(s)
- Mahault Albarracin
- Department of Cognitive Computing, Université du Québec a Montreal, Montreal, QC H2K 4M1, Canada;
- VERSES Labs, Los Angeles, CA 90016, USA;
| | - Daphne Demekas
- Department of Computing, Imperial College London, London SW7 5NH, UK;
| | - Maxwell J. D. Ramstead
- VERSES Labs, Los Angeles, CA 90016, USA;
- Wellcome Centre for Human Neuroimaging, University College London, London WC1N 3AR, UK
| | - Conor Heins
- VERSES Labs, Los Angeles, CA 90016, USA;
- Department of Collective Behaviour, Max Planck Institute of Animal Behaviour, 78315 Radolfzell am Bodensee, Germany
- Centre for the Advanced Study of Collective Behaviour, University of Konstanz, 78464 Konstanz, Germany
- Department of Biology, University of Konstanz, 78457 Konstanz, Germany
| |
Collapse
|
14
|
Averbeck B, O'Doherty JP. Reinforcement-learning in fronto-striatal circuits. Neuropsychopharmacology 2022; 47:147-162. [PMID: 34354249 PMCID: PMC8616931 DOI: 10.1038/s41386-021-01108-0] [Citation(s) in RCA: 34] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Revised: 07/06/2021] [Accepted: 07/09/2021] [Indexed: 01/03/2023]
Abstract
We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.
Collapse
Affiliation(s)
| | - John P O'Doherty
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA.
| |
Collapse
|
15
|
Soltani A, Koechlin E. Computational models of adaptive behavior and prefrontal cortex. Neuropsychopharmacology 2022; 47:58-71. [PMID: 34389808 PMCID: PMC8617006 DOI: 10.1038/s41386-021-01123-1] [Citation(s) in RCA: 27] [Impact Index Per Article: 13.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/10/2021] [Revised: 07/19/2021] [Accepted: 07/20/2021] [Indexed: 02/07/2023]
Abstract
The real world is uncertain, and while ever changing, it constantly presents itself in terms of new sets of behavioral options. To attain the flexibility required to tackle these challenges successfully, most mammalian brains are equipped with certain computational abilities that rely on the prefrontal cortex (PFC). By examining learning in terms of internal models associating stimuli, actions, and outcomes, we argue here that adaptive behavior relies on specific interactions between multiple systems including: (1) selective models learning stimulus-action associations through rewards; (2) predictive models learning stimulus- and/or action-outcome associations through statistical inferences anticipating behavioral outcomes; and (3) contextual models learning external cues associated with latent states of the environment. Critically, the PFC combines these internal models by forming task sets to drive behavior and, moreover, constantly evaluates the reliability of actor task sets in predicting external contingencies to switch between task sets or create new ones. We review different models of adaptive behavior to demonstrate how their components map onto this unifying framework and specific PFC regions. Finally, we discuss how our framework may help to better understand the neural computations and the cognitive architecture of PFC regions guiding adaptive behavior.
Collapse
Affiliation(s)
- Alireza Soltani
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
| | - Etienne Koechlin
- Institut National de la Sante et de la Recherche Medicale, Universite Pierre et Marie Curie, Ecole Normale Superieure, Paris, France.
| |
Collapse
|
16
|
Yoo AH, Collins AGE. How Working Memory and Reinforcement Learning Are Intertwined: A Cognitive, Neural, and Computational Perspective. J Cogn Neurosci 2021; 34:551-568. [PMID: 34942642 DOI: 10.1162/jocn_a_01808] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Reinforcement learning and working memory are two core processes of human cognition and are often considered cognitively, neuroscientifically, and algorithmically distinct. Here, we show that the brain networks that support them actually overlap significantly and that they are less distinct cognitive processes than often assumed. We review literature demonstrating the benefits of considering each process to explain properties of the other and highlight recent work investigating their more complex interactions. We discuss how future research in both computational and cognitive sciences can benefit from one another, suggesting that a key missing piece for artificial agents to learn to behave with more human-like efficiency is taking working memory's role in learning seriously. This review highlights the risks of neglecting the interplay between different processes when studying human behavior (in particular when considering individual differences). We emphasize the importance of investigating these dynamics to build a comprehensive understanding of human cognition.
Collapse
|
17
|
Parr T, Pezzulo G. Understanding, Explanation, and Active Inference. Front Syst Neurosci 2021; 15:772641. [PMID: 34803619 PMCID: PMC8602880 DOI: 10.3389/fnsys.2021.772641] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2021] [Accepted: 10/15/2021] [Indexed: 11/13/2022] Open
Abstract
While machine learning techniques have been transformative in solving a range of problems, an important challenge is to understand why they arrive at the decisions they output. Some have argued that this necessitates augmenting machine intelligence with understanding such that, when queried, a machine is able to explain its behaviour (i.e., explainable AI). In this article, we address the issue of machine understanding from the perspective of active inference. This paradigm enables decision making based upon a model of how data are generated. The generative model contains those variables required to explain sensory data, and its inversion may be seen as an attempt to explain the causes of these data. Here we are interested in explanations of one's own actions. This implies a deep generative model that includes a model of the world, used to infer policies, and a higher-level model that attempts to predict which policies will be selected based upon a space of hypothetical (i.e., counterfactual) explanations-and which can subsequently be used to provide (retrospective) explanations about the policies pursued. We illustrate the construct validity of this notion of understanding in relation to human understanding by highlighting the similarities in computational architecture and the consequences of its dysfunction.
Collapse
Affiliation(s)
- Thomas Parr
- Wellcome Centre for Human Neuroimaging, Queen Square Institute of Neurology, University College London, London, United Kingdom
| | - Giovanni Pezzulo
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy
| |
Collapse
|
18
|
Na S, Chung D, Hula A, Perl O, Jung J, Heflin M, Blackmore S, Fiore VG, Dayan P, Gu X. Humans use forward thinking to exploit social controllability. eLife 2021; 10:64983. [PMID: 34711304 PMCID: PMC8555988 DOI: 10.7554/elife.64983] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2020] [Accepted: 09/30/2021] [Indexed: 12/27/2022] Open
Abstract
The controllability of our social environment has a profound impact on our behavior and mental health. Nevertheless, neurocomputational mechanisms underlying social controllability remain elusive. Here, 48 participants performed a task where their current choices either did (Controllable), or did not (Uncontrollable), influence partners’ future proposals. Computational modeling revealed that people engaged a mental model of forward thinking (FT; i.e., calculating the downstream effects of current actions) to estimate social controllability in both Controllable and Uncontrollable conditions. A large-scale online replication study (n=1342) supported this finding. Using functional magnetic resonance imaging (n=48), we further demonstrated that the ventromedial prefrontal cortex (vmPFC) computed the projected total values of current actions during forward planning, supporting the neural realization of the forward-thinking model. These findings demonstrate that humans use vmPFC-dependent FT to estimate and exploit social controllability, expanding the role of this neurocomputational mechanism beyond spatial and cognitive contexts.
Collapse
Affiliation(s)
- Soojung Na
- The Graduate School of Biomedical Sciences, Icahn School of Medicine at Mount Sinai, New York, United States.,Nash Family Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, United States.,Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, United States
| | - Dongil Chung
- Department of Biomedical Engineering, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea
| | - Andreas Hula
- Austrian Institute of Technology, Seibersdorf, Austria
| | - Ofer Perl
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, United States
| | - Jennifer Jung
- School of Behavioral and Brain Sciences, The University of Texas at Dallas, Richardson, United States
| | - Matthew Heflin
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, United States
| | - Sylvia Blackmore
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, United States.,Queen Square Institute of Neurology, University College London, London, United Kingdom
| | - Vincenzo G Fiore
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, United States
| | - Peter Dayan
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany.,University of Tübingen, Tübingen, Germany
| | - Xiaosi Gu
- Nash Family Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, United States.,Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, United States
| |
Collapse
|
19
|
Glück VM, Zwosta K, Wolfensteller U, Ruge H, Pittig A. Costly habitual avoidance is reduced by concurrent goal-directed approach in a modified devaluation paradigm. Behav Res Ther 2021; 146:103964. [PMID: 34547635 DOI: 10.1016/j.brat.2021.103964] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2020] [Revised: 03/31/2021] [Accepted: 09/07/2021] [Indexed: 12/24/2022]
Abstract
Avoidance habits potentially contribute to maintaining maladaptive, costly avoidance behaviors that persist in the absence of threat. However, experimental evidence about costly habitual avoidance is scarce. In two experiments, we tested whether extensively trained avoidance impairs the subsequent goal-directed approach of rewards. Healthy participants were extensively trained to avoid an aversive outcome by performing simple responses to distinct full-screen color stimuli. After the subsequent devaluation of the aversive outcome, participants received monetary rewards for correct responses to neutral object pictures, which were presented on top of the same full-screen colors. These approach responses were either compatible or incompatible with habitual avoidance responses. Notably, the full-screen colors were not relevant to inform approach responses. In Experiment 1, participants were not instructed about post-devaluation stimulus-response-reward contingencies. Accuracy was lower in habit-incompatible than in habit-compatible trials, indicating costly avoidance, whereas reaction times did not differ. In Experiment 2, contingencies were explicitly instructed. Accuracy differences disappeared, but reaction times were slower in habit-incompatible than in habit-compatible trials, indicating low-cost habitual avoidance tendencies. These findings suggest a small but consistent impact of habitual avoidance tendencies on subsequent goal-directed approach. Costly habitual responding could, however, be inhibited when competing goal-directed approach was easily realizable.
Collapse
Affiliation(s)
- Valentina M Glück
- Department of Psychology, University of Würzburg, Würzburg, Germany.
| | - Katharina Zwosta
- Department of Psychology, Technische Universität Dresden, Dresden, Germany.
| | - Uta Wolfensteller
- Department of Psychology, Technische Universität Dresden, Dresden, Germany.
| | - Hannes Ruge
- Department of Psychology, Technische Universität Dresden, Dresden, Germany.
| | - Andre Pittig
- Department of Psychology, University of Würzburg, Würzburg, Germany; Translational Psychotherapy, Department of Psychology, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany.
| |
Collapse
|
20
|
NLM-HS: Navigation Learning Model Based on a Hippocampal-Striatal Circuit for Explaining Navigation Mechanisms in Animal Brains. Brain Sci 2021; 11:brainsci11060803. [PMID: 34204482 PMCID: PMC8235547 DOI: 10.3390/brainsci11060803] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2021] [Revised: 06/04/2021] [Accepted: 06/15/2021] [Indexed: 11/17/2022] Open
Abstract
Neurophysiological studies have shown that the hippocampus, striatum, and prefrontal cortex play different roles in animal navigation, but it is still less clear how these structures work together. In this paper, we establish a navigation learning model based on the hippocampal-striatal circuit (NLM-HS), which provides a possible explanation for the navigation mechanism in the animal brain. The hippocampal model generates a cognitive map of the environment and performs goal-directed navigation by using a place cell sequence planning algorithm. The striatal model performs reward-related habitual navigation by using the classic temporal difference learning algorithm. Since the two models may produce inconsistent behavioral decisions, the prefrontal cortex model chooses the most appropriate strategies by using a strategy arbitration mechanism. The cognitive and learning mechanism of the NLM-HS works in two stages of exploration and navigation. First, the agent uses a hippocampal model to construct the cognitive map of the unknown environment. Then, the agent uses the strategy arbitration mechanism in the prefrontal cortex model to directly decide which strategy to choose. To test the validity of the NLM-HS, the classical Tolman detour experiment was reproduced. The results show that the NLM-HS not only makes agents show environmental cognition and navigation behavior similar to animals, but also makes behavioral decisions faster and achieves better adaptivity than hippocampal or striatal models alone.
Collapse
|
21
|
Abstract
We use neural reinforcement learning concepts including Pavlovian versus instrumental control, liking versus wanting, model-based versus model-free control, online versus offline learning and planning, and internal versus external actions and control to reflect on putative conflicts between short-term temptations and long-term goals.
Collapse
|
22
|
O'Doherty JP, Lee SW, Tadayonnejad R, Cockburn J, Iigaya K, Charpentier CJ. Why and how the brain weights contributions from a mixture of experts. Neurosci Biobehav Rev 2021; 123:14-23. [PMID: 33444700 DOI: 10.1016/j.neubiorev.2020.10.022] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2020] [Revised: 09/14/2020] [Accepted: 10/26/2020] [Indexed: 12/12/2022]
Abstract
It has long been suggested that human behavior reflects the contributions of multiple systems that cooperate or compete for behavioral control. Here we propose that the brain acts as a "Mixture of Experts" in which different expert systems propose strategies for action. It will be argued that the brain determines which experts should control behavior at any one moment in time by keeping track of the reliability of the predictions within each system, and by allocating control over behavior in a manner that depends on the relative reliabilities across experts. fMRI and neurostimulation studies suggest a specific contribution of the anterior prefrontal cortex in this process. Further, such a mechanism also takes into consideration the complexity of the expert, favoring simpler over more cognitively complex experts. Results from the study of different expert systems in both experiential and social learning domains hint at the possibility that this reliability-based control mechanism is domain general, exerting control over many different expert systems simultaneously in order to produce sophisticated behavior.
Collapse
Affiliation(s)
- John P O'Doherty
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA; Computation and Neural Systems Program, California Institute of Technology, Pasadena, CA, 91125, USA.
| | - Sang Wan Lee
- Department of Bio and Brain Engineering and Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science Technology (KAIST), Daejeon, 34141, Republic of Korea
| | - Reza Tadayonnejad
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA; Division of Neuromodulation, Semel Institute for Neuroscience and Behavior, University of California, Los Angeles, CA, 90095, USA
| | - Jeff Cockburn
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA
| | - Kyo Iigaya
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA
| | - Caroline J Charpentier
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA
| |
Collapse
|
23
|
Tessereau C, O’Dea R, Coombes S, Bast T. Reinforcement learning approaches to hippocampus-dependent flexible spatial navigation. Brain Neurosci Adv 2021; 5:2398212820975634. [PMID: 33954259 PMCID: PMC8042550 DOI: 10.1177/2398212820975634] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2020] [Accepted: 10/21/2020] [Indexed: 11/17/2022] Open
Abstract
Humans and non-human animals show great flexibility in spatial navigation, including the ability to return to specific locations based on as few as one single experience. To study spatial navigation in the laboratory, watermaze tasks, in which rats have to find a hidden platform in a pool of cloudy water surrounded by spatial cues, have long been used. Analogous tasks have been developed for human participants using virtual environments. Spatial learning in the watermaze is facilitated by the hippocampus. In particular, rapid, one-trial, allocentric place learning, as measured in the delayed-matching-to-place variant of the watermaze task, which requires rodents to learn repeatedly new locations in a familiar environment, is hippocampal dependent. In this article, we review some computational principles, embedded within a reinforcement learning framework, that utilise hippocampal spatial representations for navigation in watermaze tasks. We consider which key elements underlie their efficacy, and discuss their limitations in accounting for hippocampus-dependent navigation, both in terms of behavioural performance (i.e. how well do they reproduce behavioural measures of rapid place learning) and neurobiological realism (i.e. how well do they map to neurobiological substrates involved in rapid place learning). We discuss how an actor-critic architecture, enabling simultaneous assessment of the value of the current location and of the optimal direction to follow, can reproduce one-trial place learning performance as shown on watermaze and virtual delayed-matching-to-place tasks by rats and humans, respectively, if complemented with map-like place representations. The contribution of actor-critic mechanisms to delayed-matching-to-place performance is consistent with neurobiological findings implicating the striatum and hippocampo-striatal interaction in delayed-matching-to-place performance, given that the striatum has been associated with actor-critic mechanisms. Moreover, we illustrate that hierarchical computations embedded within an actor-critic architecture may help to account for aspects of flexible spatial navigation. The hierarchical reinforcement learning approach separates trajectory control via a temporal-difference error from goal selection via a goal prediction error and may account for flexible, trial-specific, navigation to familiar goal locations, as required in some arm-maze place memory tasks, although it does not capture one-trial learning of new goal locations, as observed in open field, including watermaze and virtual, delayed-matching-to-place tasks. Future models of one-shot learning of new goal locations, as observed on delayed-matching-to-place tasks, should incorporate hippocampal plasticity mechanisms that integrate new goal information with allocentric place representation, as such mechanisms are supported by substantial empirical evidence.
Collapse
Affiliation(s)
- Charline Tessereau
- School of Mathematical Sciences, University of Nottingham, Nottingham, UK
- School of Psychology, University of Nottingham, Nottingham, UK
- Neuroscience@Nottingham
| | - Reuben O’Dea
- School of Mathematical Sciences, University of Nottingham, Nottingham, UK
- Neuroscience@Nottingham
| | - Stephen Coombes
- School of Mathematical Sciences, University of Nottingham, Nottingham, UK
- Neuroscience@Nottingham
| | - Tobias Bast
- School of Psychology, University of Nottingham, Nottingham, UK
- Neuroscience@Nottingham
| |
Collapse
|
24
|
Steinke A, Lange F, Kopp B. Parallel model-based and model-free reinforcement learning for card sorting performance. Sci Rep 2020; 10:15464. [PMID: 32963297 PMCID: PMC7508815 DOI: 10.1038/s41598-020-72407-7] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2020] [Accepted: 08/31/2020] [Indexed: 12/13/2022] Open
Abstract
The Wisconsin Card Sorting Test (WCST) is considered a gold standard for the assessment of cognitive flexibility. On the WCST, repeating a sorting category following negative feedback is typically treated as indicating reduced cognitive flexibility. Therefore such responses are referred to as 'perseveration' errors. Recent research suggests that the propensity for perseveration errors is modulated by response demands: They occur less frequently when their commitment repeats the previously executed response. Here, we propose parallel reinforcement-learning models of card sorting performance, which assume that card sorting performance can be conceptualized as resulting from model-free reinforcement learning at the level of responses that occurs in parallel with model-based reinforcement learning at the categorical level. We compared parallel reinforcement-learning models with purely model-based reinforcement learning, and with the state-of-the-art attentional-updating model. We analyzed data from 375 participants who completed a computerized WCST. Parallel reinforcement-learning models showed best predictive accuracies for the majority of participants. Only parallel reinforcement-learning models accounted for the modulation of perseveration propensity by response demands. In conclusion, parallel reinforcement-learning models provide a new theoretical perspective on card sorting and it offers a suitable framework for discerning individual differences in latent processes that subserve behavioral flexibility.
Collapse
Affiliation(s)
- Alexander Steinke
- Department of Neurology, Hannover Medical School, Carl-Neuberg-Straße 1, 30625, Hannover, Germany.
| | - Florian Lange
- Behavioral Engineering Research Group, KU Leuven, Naamsestraat 69, 3000, Leuven, Belgium
| | - Bruno Kopp
- Department of Neurology, Hannover Medical School, Carl-Neuberg-Straße 1, 30625, Hannover, Germany
| |
Collapse
|
25
|
Combined model-free and model-sensitive reinforcement learning in non-human primates. PLoS Comput Biol 2020; 16:e1007944. [PMID: 32569311 PMCID: PMC7332075 DOI: 10.1371/journal.pcbi.1007944] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2020] [Revised: 07/02/2020] [Accepted: 05/12/2020] [Indexed: 11/25/2022] Open
Abstract
Contemporary reinforcement learning (RL) theory suggests that potential choices can be evaluated by strategies that may or may not be sensitive to the computational structure of tasks. A paradigmatic model-free (MF) strategy simply repeats actions that have been rewarded in the past; by contrast, model-sensitive (MS) strategies exploit richer information associated with knowledge of task dynamics. MF and MS strategies should typically be combined, because they have complementary statistical and computational strengths; however, this tradeoff between MF/MS RL has mostly only been demonstrated in humans, often with only modest numbers of trials. We trained rhesus monkeys to perform a two-stage decision task designed to elicit and discriminate the use of MF and MS methods. A descriptive analysis of choice behaviour revealed directly that the structure of the task (of MS importance) and the reward history (of MF and MS importance) significantly influenced both choice and response vigour. A detailed, trial-by-trial computational analysis confirmed that choices were made according to a combination of strategies, with a dominant influence of a particular form of model sensitivity that persisted over weeks of testing. The residuals from this model necessitated development of a new combined RL model which incorporates a particular credit assignment weighting procedure. Finally, response vigor exhibited a subtly different collection of MF and MS influences. These results provide new illumination onto RL behavioural processes in non-human primates. We routinely solve planning problems in which present decisions have consequences in the future. These pose complex computational and statistical problems and are addressed by multiple systems in the brain which use different solutions to these problems, and which may compete and cooperate. We trained two rhesus monkeys on a paradigmatic planning task that transparently reveals canonical aspects of different strategies. We performed a detailed behavioral analysis using methods of reinforcement learning on choice and reaction time to reveal conjoint influences and structural interactions of different sources of information. We show the strengths and limitations of these analyses, at the same time as we provide a novel perspective on how different learning systems interact for choice in non-human primates.
Collapse
|
26
|
Representation, abstraction, and simple-minded sophisticates. Behav Brain Sci 2020; 43:e126. [PMID: 32645812 DOI: 10.1017/s0140525x19002942] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Bayesian decision theory provides a simple formal elucidation of some of the ways that representation and representational abstraction are involved with, and exploit, both prediction and its rather distant cousin, predictive coding. Both model-free and model-based methods are involved.
Collapse
|
27
|
Khamassi M, Girard B. Modeling awake hippocampal reactivations with model-based bidirectional search. BIOLOGICAL CYBERNETICS 2020; 114:231-248. [PMID: 32065253 DOI: 10.1007/s00422-020-00817-x] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/30/2019] [Accepted: 01/21/2020] [Indexed: 06/10/2023]
Abstract
Hippocampal offline reactivations during reward-based learning, usually categorized as replay events, have been found to be important for performance improvement over time and for memory consolidation. Recent computational work has linked these phenomena to the need to transform reward information into state-action values for decision making and to propagate it to all relevant states of the environment. Nevertheless, it is still unclear whether an integrated reinforcement learning mechanism could account for the variety of awake hippocampal reactivations, including variety in order (forward and reverse reactivated trajectories) and variety in the location where they occur (reward site or decision-point). Here, we present a model-based bidirectional search model which accounts for a variety of hippocampal reactivations. The model combines forward trajectory sampling from current position and backward sampling through prioritized sweeping from states associated with large reward prediction errors until the two trajectories connect. This is repeated until stabilization of state-action values (convergence), which could explain why hippocampal reactivations drastically diminish when the animal's performance stabilizes. Simulations in a multiple T-maze task show that forward reactivations are prominently found at decision-points while backward reactivations are exclusively generated at reward sites. Finally, the model can generate imaginary trajectories that are not allowed to the agent during task performance. We raise some experimental predictions and implications for future studies of the role of the hippocampo-prefronto-striatal network in learning.
Collapse
Affiliation(s)
- Mehdi Khamassi
- Institute of Intelligent Systems and Robotics (ISIR), Sorbonne Université and CNRS (Centre National de la Recherche Scientifique), 75005, Paris, France.
| | - Benoît Girard
- Institute of Intelligent Systems and Robotics (ISIR), Sorbonne Université and CNRS (Centre National de la Recherche Scientifique), 75005, Paris, France
| |
Collapse
|
28
|
O’Reilly RC, Nair A, Russin JL, Herd SA. How Sequential Interactive Processing Within Frontostriatal Loops Supports a Continuum of Habitual to Controlled Processing. Front Psychol 2020; 11:380. [PMID: 32210892 PMCID: PMC7076192 DOI: 10.3389/fpsyg.2020.00380] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2019] [Accepted: 02/18/2020] [Indexed: 11/13/2022] Open
Abstract
We address the distinction between habitual/automatic vs. goal-directed/controlled behavior, from the perspective of a computational model of the frontostriatal loops. The model exhibits a continuum of behavior between these poles, as a function of the interactive dynamics among different functionally-specialized brain areas, operating iteratively over multiple sequential steps, and having multiple nested loops of similar decision making circuits. This framework blurs the lines between these traditional distinctions in many ways. For example, although habitual actions have traditionally been considered purely automatic, the outer loop must first decide to allow such habitual actions to proceed. Furthermore, because the part of the brain that generates proposed action plans is common across habitual and controlled/goal-directed behavior, the key differences are instead in how many iterations of sequential decision-making are taken, and to what extent various forms of predictive (model-based) processes are engaged. At the core of every iterative step in our model, the basal ganglia provides a "model-free" dopamine-trained Go/NoGo evaluation of the entire distributed plan/goal/evaluation/prediction state. This evaluation serves as the fulcrum of serializing otherwise parallel neural processing. Goal-based inputs to the nominally model-free basal ganglia system are among several ways in which the popular model-based vs. model-free framework may not capture the most behaviorally and neurally relevant distinctions in this area.
Collapse
Affiliation(s)
- Randall C. O’Reilly
- Computational Cognitive Neuroscience Lab, Department of Psychology, Computer Science, and Center for Neuroscience, University of California, Davis, Davis, CA, United States
- eCortex, Inc., Boulder, CO, United States
| | | | - Jacob L. Russin
- Computational Cognitive Neuroscience Lab, Department of Psychology, Computer Science, and Center for Neuroscience, University of California, Davis, Davis, CA, United States
| | - Seth A. Herd
- Computational Cognitive Neuroscience Lab, Department of Psychology, Computer Science, and Center for Neuroscience, University of California, Davis, Davis, CA, United States
- eCortex, Inc., Boulder, CO, United States
| |
Collapse
|
29
|
Balleine BW, Dezfouli A. Hierarchical Action Control: Adaptive Collaboration Between Actions and Habits. Front Psychol 2019; 10:2735. [PMID: 31920796 PMCID: PMC6917637 DOI: 10.3389/fpsyg.2019.02735] [Citation(s) in RCA: 34] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2019] [Accepted: 11/19/2019] [Indexed: 12/25/2022] Open
Abstract
It is now commonly accepted that instrumental actions can reflect goal-directed control; i.e., they can show sensitivity to changes in the relationship to and the value of their consequences. With overtraining, stress, neurodegeneration, psychiatric conditions, or after exposure to various drugs of abuse, goal-directed control declines and instrumental actions are performed independently of their consequences. Although this latter insensitivity has been argued to reflect the development of habitual control, the lack of a positive definition of habits has rendered this conclusion controversial. Here we consider various alternative definitions of habit, including recent suggestions they reflect chunked action sequences, to derive criteria with which to categorize responses as habitual. We consider various theories regarding the interaction between goal-directed and habitual controllers and propose a collaborative model based on their hierarchical integration. We argue that this model is consistent with the available data, can be instantiated both at an associative level and computationally and generates interesting predictions regarding the influence of this collaborative integration on behavior.
Collapse
Affiliation(s)
- Bernard W Balleine
- Decision Neuroscience Laboratory, School of Psychology, University of New South Wales Sydney, Sydney, NSW, Australia
| | - Amir Dezfouli
- Data 61, Commonwealth Scientific and Industrial Research Organisation, Sydney, NSW, Australia
| |
Collapse
|
30
|
Abstract
A popular distinction in the human and animal learning literature is between deliberate (or willed) and habitual (or automatic) modes of control. Extensive evidence indicates that, after sufficient learning, living organisms develop behavioural habits that permit them saving computational resources. Furthermore, humans and other animals are able to transfer control from deliberate to habitual modes (and vice versa), trading off efficiently flexibility and parsimony - an ability that is currently unparalleled by artificial control systems. Here, we discuss a computational implementation of habit formation, and the transfer of control from deliberate to habitual modes (and vice versa) within Active Inference: a computational framework that merges aspects of cybernetic theory and of Bayesian inference. To model habit formation, we endow an Active Inference agent with a mechanism to "cache" (or memorize) policy probabilities from previous trials, and reuse them to skip - in part or in full - the inferential steps of deliberative processing. We exploit the fact that the relative quality of policies, conditioned upon hidden states, is constant over trials; provided that contingencies and prior preferences do not change. This means the only quantity that can change policy selection is the prior distribution over the initial state - where this prior is based upon the posterior beliefs from previous trials. Thus, an agent that caches the quality (or the probability) of policies can safely reuse cached values to save on cognitive and computational resources - unless contingencies change. Our simulations illustrate the computational benefits, but also the limits, of three caching schemes under Active Inference. They suggest that key aspects of habitual behaviour - such as perseveration - can be explained in terms of caching policy probabilities. Furthermore, they suggest that there may be many kinds (or stages) of habitual behaviour, each associated with a different caching scheme; for example, caching associated or not associated with contextual estimation. These schemes are more or less impervious to contextual and contingency changes.
Collapse
Affiliation(s)
- D Maisto
- Institute for High Performance Computing and Networking, National Research Council, Via P. Castellino, 111, Naples 80131, Italy
| | - K Friston
- The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College London, London, UK
| | - G Pezzulo
- Institute of Cognitive Sciences and Technologies, National Research Council, Via San Martino della Battaglia 44, Rome 00185, Italy
| |
Collapse
|
31
|
Gottwald S, Braun DA. Bounded Rational Decision-Making from Elementary Computations That Reduce Uncertainty. ENTROPY 2019; 21:e21040375. [PMID: 33267089 PMCID: PMC7514859 DOI: 10.3390/e21040375] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/19/2019] [Revised: 04/02/2019] [Accepted: 04/04/2019] [Indexed: 11/20/2022]
Abstract
In its most basic form, decision-making can be viewed as a computational process that progressively eliminates alternatives, thereby reducing uncertainty. Such processes are generally costly, meaning that the amount of uncertainty that can be reduced is limited by the amount of available computational resources. Here, we introduce the notion of elementary computation based on a fundamental principle for probability transfers that reduce uncertainty. Elementary computations can be considered as the inverse of Pigou–Dalton transfers applied to probability distributions, closely related to the concepts of majorization, T-transforms, and generalized entropies that induce a preorder on the space of probability distributions. Consequently, we can define resource cost functions that are order-preserving and therefore monotonic with respect to the uncertainty reduction. This leads to a comprehensive notion of decision-making processes with limited resources. Along the way, we prove several new results on majorization theory, as well as on entropy and divergence measures.
Collapse
|
32
|
Schach S, Gottwald S, Braun DA. Quantifying Motor Task Performance by Bounded Rational Decision Theory. Front Neurosci 2018; 12:932. [PMID: 30618561 PMCID: PMC6302104 DOI: 10.3389/fnins.2018.00932] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2018] [Accepted: 11/27/2018] [Indexed: 01/22/2023] Open
Abstract
Expected utility models are often used as a normative baseline for human performance in motor tasks. However, this baseline ignores computational costs that are incurred when searching for the optimal strategy. In contrast, bounded rational decision-theory provides a normative baseline that takes computational effort into account, as it describes optimal behavior of an agent with limited information-processing capacity to change a prior motor strategy (before information-processing) into a posterior strategy (after information-processing). Here, we devised a pointing task where subjects had restricted reaction and movement time. In particular, we manipulated the permissible reaction time as a proxy for the amount of computation allowed for planning the movements. Moreover, we tested three different distributions over the target locations to induce different prior strategies that would influence the amount of required information-processing. We found that movement endpoint precision generally decreases with limited planning time and that non-uniform prior probabilities allow for more precise movements toward high-probability targets. Considering these constraints in a bounded rational decision model, we found that subjects were generally close to bounded optimal. We conclude that bounded rational decision theory may be a promising normative framework to analyze human sensorimotor performance.
Collapse
Affiliation(s)
- Sonja Schach
- Faculty of Engingeering, Computer Science and Psychology, Institute of Neural Information Processing, Ulm University, Ulm, Germany
| | - Sebastian Gottwald
- Faculty of Engingeering, Computer Science and Psychology, Institute of Neural Information Processing, Ulm University, Ulm, Germany
| | - Daniel A Braun
- Faculty of Engingeering, Computer Science and Psychology, Institute of Neural Information Processing, Ulm University, Ulm, Germany
| |
Collapse
|
33
|
Cazé R, Khamassi M, Aubin L, Girard B. Hippocampal replays under the scrutiny of reinforcement learning models. J Neurophysiol 2018; 120:2877-2896. [DOI: 10.1152/jn.00145.2018] [Citation(s) in RCA: 24] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023] Open
Abstract
Multiple in vivo studies have shown that place cells from the hippocampus replay previously experienced trajectories. These replays are commonly considered to mainly reflect memory consolidation processes. Some data, however, have highlighted a functional link between replays and reinforcement learning (RL). This theory, extensively used in machine learning, has introduced efficient algorithms and can explain various behavioral and physiological measures from different brain regions. RL algorithms could constitute a mechanistic description of replays and explain how replays can reduce the number of iterations required to explore the environment during learning. We review the main findings concerning the different hippocampal replay types and the possible associated RL models (either model-based, model-free, or hybrid model types). We conclude by tying these frameworks together. We illustrate the link between data and RL through a series of model simulations. This review, at the frontier between informatics and biology, paves the way for future work on replays.
Collapse
Affiliation(s)
- Romain Cazé
- Institute of Intelligent Systems and Robotics, Sorbonne Université, CNRS, Paris, France
| | - Mehdi Khamassi
- Institute of Intelligent Systems and Robotics, Sorbonne Université, CNRS, Paris, France
| | - Lise Aubin
- Institute of Intelligent Systems and Robotics, Sorbonne Université, CNRS, Paris, France
| | - Benoît Girard
- Institute of Intelligent Systems and Robotics, Sorbonne Université, CNRS, Paris, France
| |
Collapse
|
34
|
Zénon A, Solopchuk O, Pezzulo G. An information-theoretic perspective on the costs of cognition. Neuropsychologia 2018; 123:5-18. [PMID: 30268880 DOI: 10.1016/j.neuropsychologia.2018.09.013] [Citation(s) in RCA: 54] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2017] [Revised: 08/10/2018] [Accepted: 09/19/2018] [Indexed: 01/06/2023]
Abstract
In statistics and machine learning, model accuracy is traded off with complexity, which can be viewed as the amount of information extracted from the data. Here, we discuss how cognitive costs can be expressed in terms of similar information costs, i.e. as a function of the amount of information required to update a person's prior knowledge (or internal model) to effectively solve a task. We then examine the theoretical consequences that ensue from this assumption. This framework naturally explains why some tasks - for example, unfamiliar or dual tasks - are costly and permits to quantify these costs using information-theoretic measures. Finally, we discuss brain implementation of this principle and show that subjective cognitive costs can originate either from local or global capacity limitations on information processing or from increased rate of metabolic alterations. These views shed light on the potential adaptive value of cost-avoidance mechanisms.
Collapse
Affiliation(s)
- Alexandre Zénon
- Institut de Neuroscience Cognitive et Intégrative d'Aquitaine, Université de Bordeaux, France; Institute of Neuroscience, Université Catholique de Louvain, Brussels, Belgium.
| | - Oleg Solopchuk
- Institut de Neuroscience Cognitive et Intégrative d'Aquitaine, Université de Bordeaux, France; Institute of Neuroscience, Université Catholique de Louvain, Brussels, Belgium
| | - Giovanni Pezzulo
- Institute of Cognitive Sciences and Technologies, National Research Council, Via San Martino della Battaglia 44, 00185 Rome, Italy
| |
Collapse
|
35
|
Stoianov IP, Pennartz CMA, Lansink CS, Pezzulo G. Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis. PLoS Comput Biol 2018; 14:e1006316. [PMID: 30222746 PMCID: PMC6160242 DOI: 10.1371/journal.pcbi.1006316] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2017] [Revised: 09/27/2018] [Accepted: 06/20/2018] [Indexed: 12/26/2022] Open
Abstract
While the neurobiology of simple and habitual choices is relatively well known, our current understanding of goal-directed choices and planning in the brain is still limited. Theoretical work suggests that goal-directed computations can be productively associated to model-based (reinforcement learning) computations, yet a detailed mapping between computational processes and neuronal circuits remains to be fully established. Here we report a computational analysis that aligns Bayesian nonparametrics and model-based reinforcement learning (MB-RL) to the functioning of the hippocampus (HC) and the ventral striatum (vStr)-a neuronal circuit that increasingly recognized to be an appropriate model system to understand goal-directed (spatial) decisions and planning mechanisms in the brain. We test the MB-RL agent in a contextual conditioning task that depends on intact hippocampus and ventral striatal (shell) function and show that it solves the task while showing key behavioral and neuronal signatures of the HC-vStr circuit. Our simulations also explore the benefits of biological forms of look-ahead prediction (forward sweeps) during both learning and control. This article thus contributes to fill the gap between our current understanding of computational algorithms and biological realizations of (model-based) reinforcement learning.
Collapse
Affiliation(s)
- Ivilin Peev Stoianov
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy
| | - Cyriel M. A. Pennartz
- University of Amsterdam, Swammerdam Institute for Life Sciences–Center for Neuroscience Amsterdam, The Netherlands
| | - Carien S. Lansink
- University of Amsterdam, Swammerdam Institute for Life Sciences–Center for Neuroscience Amsterdam, The Netherlands
| | - Giovani Pezzulo
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy
| |
Collapse
|
36
|
A Multilevel Computational Characterization of Endophenotypes in Addiction. eNeuro 2018; 5:eN-TNC-0151-18. [PMID: 30073199 PMCID: PMC6071202 DOI: 10.1523/eneuro.0151-18.2018] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2018] [Revised: 06/06/2018] [Accepted: 06/12/2018] [Indexed: 02/06/2023] Open
Abstract
Addiction is characterized by a profound intersubject (phenotypic) variability in the expression of addictive symptomatology and propensity to relapse following treatment. However, laboratory investigations have primarily focused on common neural substrates in addiction and have not yet been able to identify mechanisms that can account for the multifaceted phenotypic behaviors reported in the literature. To fill this knowledge gap theoretically, here we simulated phenotypic variations in addiction symptomology and responses to putative treatments, using both a neural model, based on cortico-striatal circuit dynamics, and an algorithmic model of reinforcement learning (RL). These simulations rely on the widely accepted assumption that both the ventral, model-based, goal-directed system and the dorsal, model-free, habitual system are vulnerable to extra-physiologic dopamine reinforcements triggered by addictive rewards. We found that endophenotypic differences in the balance between the two circuit or control systems resulted in an inverted-U shape in optimal choice behavior. Specifically, greater unbalance led to a higher likelihood of developing addiction and more severe drug-taking behaviors. Furthermore, endophenotypes with opposite asymmetrical biases among cortico-striatal circuits expressed similar addiction behaviors, but responded differently to simulated treatments, suggesting personalized treatment development could rely on endophenotypic rather than phenotypic differentiations. We propose our simulated results, confirmed across neural and algorithmic levels of analysis, inform on a fundamental and, to date, neglected quantitative method to characterize clinical heterogeneity in addiction.
Collapse
|
37
|
Kool W, Gershman SJ, Cushman FA. Planning Complexity Registers as a Cost in Metacontrol. J Cogn Neurosci 2018; 30:1391-1404. [PMID: 29668390 DOI: 10.1162/jocn_a_01263] [Citation(s) in RCA: 26] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Decision-making algorithms face a basic tradeoff between accuracy and effort (i.e., computational demands). It is widely agreed that humans can choose between multiple decision-making processes that embody different solutions to this tradeoff: Some are computationally cheap but inaccurate, whereas others are computationally expensive but accurate. Recent progress in understanding this tradeoff has been catalyzed by formalizing it in terms of model-free (i.e., habitual) versus model-based (i.e., planning) approaches to reinforcement learning. Intuitively, if two tasks offer the same rewards for accuracy but one of them is much more demanding, we might expect people to rely on habit more in the difficult task: Devoting significant computation to achieve slight marginal accuracy gains would not be "worth it." We test and verify this prediction in a sequential reinforcement learning task. Because our paradigm is amenable to formal analysis, it contributes to the development of a computational model of how people balance the costs and benefits of different decision-making processes in a task-specific manner; in other words, how we decide when hard thinking is worth it.
Collapse
|
38
|
Pauli WM, Cockburn J, Pool ER, Pérez OD, O’Doherty JP. Computational approaches to habits in a model-free world. Curr Opin Behav Sci 2018. [DOI: 10.1016/j.cobeha.2017.12.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
39
|
Dollé L, Chavarriaga R, Guillot A, Khamassi M. Interactions of spatial strategies producing generalization gradient and blocking: A computational approach. PLoS Comput Biol 2018; 14:e1006092. [PMID: 29630600 PMCID: PMC5908205 DOI: 10.1371/journal.pcbi.1006092] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2017] [Revised: 04/19/2018] [Accepted: 03/15/2018] [Indexed: 12/16/2022] Open
Abstract
We present a computational model of spatial navigation comprising different learning mechanisms in mammals, i.e., associative, cognitive mapping and parallel systems. This model is able to reproduce a large number of experimental results in different variants of the Morris water maze task, including standard associative phenomena (spatial generalization gradient and blocking), as well as navigation based on cognitive mapping. Furthermore, we show that competitive and cooperative patterns between different navigation strategies in the model allow to explain previous apparently contradictory results supporting either associative or cognitive mechanisms for spatial learning. The key computational mechanism to reconcile experimental results showing different influences of distal and proximal cues on the behavior, different learning times, and different abilities of individuals to alternatively perform spatial and response strategies, relies in the dynamic coordination of navigation strategies, whose performance is evaluated online with a common currency through a modular approach. We provide a set of concrete experimental predictions to further test the computational model. Overall, this computational work sheds new light on inter-individual differences in navigation learning, and provides a formal and mechanistic approach to test various theories of spatial cognition in mammals.
Collapse
Affiliation(s)
- Laurent Dollé
- Institute of Intelligent Systems and Robotics, Sorbonne Université, CNRS, F-75005 Paris, France
| | - Ricardo Chavarriaga
- Defitech Chair in Brain-Machine Interface, Center for Neuroprosthetics, Institute of Bioengineering and School of Engineering, EPFL, Geneva, Switzerland
| | - Agnès Guillot
- Institute of Intelligent Systems and Robotics, Sorbonne Université, CNRS, F-75005 Paris, France
| | - Mehdi Khamassi
- Institute of Intelligent Systems and Robotics, Sorbonne Université, CNRS, F-75005 Paris, France
| |
Collapse
|
40
|
Abstract
The future harbours the potential for myriad threats to the fitness of organisms, and many species prepare accordingly based on indicators of hazards. Here, we distinguish between defensive responses on the basis of sensed cues and those based on autocues generated by mental simulations of the future in humans. Whereas sensed threat cues usually induce specific responses with reference to particular features of the environment or generalized responses to protect against diffuse threats, autocues generated by mental simulations of the future enable strategic preparation for hazards that may not require an immediate response. The overlap of these mechanisms makes defence effective and versatile, yet can manifest as contemporary anxiety disorders in humans.
Collapse
Affiliation(s)
- Beyon Miloyan
- Department of Mental Health, Bloomberg School of Public Health, Johns Hopkins University, USA and School of Psychology and Health Sciences, Faculty of Health, Federation University, Australia
| | | | | |
Collapse
|
41
|
Martinelli C, Rigoli F, Averbeck B, Shergill SS. The value of novelty in schizophrenia. Schizophr Res 2018; 192:287-293. [PMID: 28495493 PMCID: PMC5890442 DOI: 10.1016/j.schres.2017.05.007] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/06/2016] [Revised: 04/27/2017] [Accepted: 05/06/2017] [Indexed: 11/15/2022]
Abstract
Influential models of schizophrenia suggest that patients experience incoming stimuli as excessively novel and motivating, with important consequences for hallucinatory experience and delusional belief. However, whether schizophrenia patients exhibit excessive novelty value and whether this interferes with adaptive behaviour has not yet been formally tested. Here, we employed a three-armed bandit task to investigate this hypothesis. Schizophrenia patients and healthy controls were first familiarised with a group of images and then asked to repeatedly choose between familiar and unfamiliar images associated with different monetary reward probabilities. By fitting a reinforcement-learning model we were able to estimate the values attributed to familiar and unfamiliar images when first presented in the context of the decision-making task. In line with our hypothesis, we found increased preference for newly introduced images (irrespective of whether these were familiar or unfamiliar) in patients compared to healthy controls and this to correlate with severity of hallucinatory experience. In addition, we found a correlation between value assigned to novel images and task performance, suggesting that excessive novelty value may interfere with optimal learning in patients, putatively through the disruption of the mechanisms regulating exploration versus exploitation. Our results suggest excessive novelty value in patients, whereby even previously seen stimuli acquire higher value as the result of their exposure in a novel context - a form of 'hyper novelty' which may explain why patients are often attracted by familiar stimuli experienced as new.
Collapse
Affiliation(s)
- Cristina Martinelli
- Department of Psychosis Studies, Institute of Psychiatry, Psychology & Neuroscience, King's College London, De Crespigny Park, SE5 8AF London, United Kingdom.
| | - Francesco Rigoli
- Wellcome Trust Centre for Neuroimaging, University College London, 12 Queen's Square, WC1N 3BG London, United Kingdom
| | - Bruno Averbeck
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institute of Health, Bethesda, MD 20892-4415, USA
| | - Sukhwinder S Shergill
- Department of Psychosis Studies, Institute of Psychiatry, Psychology & Neuroscience, King's College London, De Crespigny Park, SE5 8AF London, United Kingdom
| |
Collapse
|
42
|
Lloyd K, Dayan P. Interrupting behaviour: Minimizing decision costs via temporal commitment and low-level interrupts. PLoS Comput Biol 2018; 14:e1005916. [PMID: 29338004 PMCID: PMC5786335 DOI: 10.1371/journal.pcbi.1005916] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2017] [Revised: 01/26/2018] [Accepted: 12/05/2017] [Indexed: 11/22/2022] Open
Abstract
Ideal decision-makers should constantly assess all sources of information about opportunities and threats, and be able to redetermine their choices promptly in the face of change. However, perpetual monitoring and reassessment impose inordinate sensing and computational costs, making them impractical for animals and machines alike. The obvious alternative of committing for extended periods of time to limited sensory strategies associated with particular courses of action can be dangerous and wasteful. Here, we explore the intermediate possibility of making provisional temporal commitments whilst admitting interruption based on limited broader observation. We simulate foraging under threat of predation to elucidate the benefits of such a scheme. We relate our results to diseases of distractibility and roving attention, and consider mechanistic substrates such as noradrenergic neuromodulation. Animals should ideally be able to monitor all relevant aspects of their environment constantly and be ever prepared to alter their course of action in the face of unexpected change. However, the impractically high costs of continual monitoring and deliberation mean that a more realistic strategy is required. Here, we explore a solution in which an animal makes provisional commitments to a temporally-extended action while maintaining the ability to interrupt this behaviour prematurely on the basis of more limited sensing. We demonstrate the benefits of such a scheme through the example of foraging under predation risk, and propose a simple mechanism for implementing interruption. We suggest possible relationships between these results and neural substrates, particularly norepinephrine, and also highlight potential relevance to diseases of distractibility.
Collapse
Affiliation(s)
- Kevin Lloyd
- Gatsby Computational Neuroscience Unit, London, United Kingdom
- * E-mail:
| | - Peter Dayan
- Gatsby Computational Neuroscience Unit, London, United Kingdom
| |
Collapse
|
43
|
Russek EM, Momennejad I, Botvinick MM, Gershman SJ, Daw ND. Predictive representations can link model-based reinforcement learning to model-free mechanisms. PLoS Comput Biol 2017; 13:e1005768. [PMID: 28945743 PMCID: PMC5628940 DOI: 10.1371/journal.pcbi.1005768] [Citation(s) in RCA: 127] [Impact Index Per Article: 18.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2016] [Revised: 10/05/2017] [Accepted: 09/04/2017] [Indexed: 11/19/2022] Open
Abstract
Humans and animals are capable of evaluating actions by considering their long-run future rewards through a process described using model-based reinforcement learning (RL) algorithms. The mechanisms by which neural circuits perform the computations prescribed by model-based RL remain largely unknown; however, multiple lines of evidence suggest that neural circuits supporting model-based behavior are structurally homologous to and overlapping with those thought to carry out model-free temporal difference (TD) learning. Here, we lay out a family of approaches by which model-based computation may be built upon a core of TD learning. The foundation of this framework is the successor representation, a predictive state representation that, when combined with TD learning of value predictions, can produce a subset of the behaviors associated with model-based learning, while requiring less decision-time computation than dynamic programming. Using simulations, we delineate the precise behavioral capabilities enabled by evaluating actions using this approach, and compare them to those demonstrated by biological organisms. We then introduce two new algorithms that build upon the successor representation while progressively mitigating its limitations. Because this framework can account for the full range of observed putatively model-based behaviors while still utilizing a core TD framework, we suggest that it represents a neurally plausible family of mechanisms for model-based evaluation.
Collapse
Affiliation(s)
- Evan M. Russek
- Center for Neural Science, New York University, New York, NY, United States of America
| | - Ida Momennejad
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, United States of America
| | - Matthew M. Botvinick
- DeepMind, London, United Kingdom and Gatsby Computational Neuroscience Unit, University College London, United Kingdom
| | - Samuel J. Gershman
- Department of Psychology and Center for Brain Science, Harvard University, Cambridge, MA, United States of America
| | - Nathaniel D. Daw
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, United States of America
| |
Collapse
|
44
|
Kool W, Gershman SJ, Cushman FA. Cost-Benefit Arbitration Between Multiple Reinforcement-Learning Systems. Psychol Sci 2017; 28:1321-1333. [DOI: 10.1177/0956797617708288] [Citation(s) in RCA: 109] [Impact Index Per Article: 15.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
Abstract
Human behavior is sometimes determined by habit and other times by goal-directed planning. Modern reinforcement-learning theories formalize this distinction as a competition between a computationally cheap but inaccurate model-free system that gives rise to habits and a computationally expensive but accurate model-based system that implements planning. It is unclear, however, how people choose to allocate control between these systems. Here, we propose that arbitration occurs by comparing each system’s task-specific costs and benefits. To investigate this proposal, we conducted two experiments showing that people increase model-based control when it achieves greater accuracy than model-free control, and especially when the rewards of accurate performance are amplified. In contrast, they are insensitive to reward amplification when model-based and model-free control yield equivalent accuracy. This suggests that humans adaptively balance habitual and planned action through on-line cost-benefit analysis.
Collapse
Affiliation(s)
- Wouter Kool
- Department of Psychology, Harvard University
| | - Samuel J. Gershman
- Department of Psychology, Harvard University
- Center for Brain Science, Harvard University
| | | |
Collapse
|
45
|
Gershman SJ, Zhou J, Kommers C. Imaginative Reinforcement Learning: Computational Principles and Neural Mechanisms. J Cogn Neurosci 2017; 29:2103-2113. [PMID: 28707569 DOI: 10.1162/jocn_a_01170] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/24/2023]
Abstract
Imagination enables us not only to transcend reality but also to learn about it. In the context of reinforcement learning, an agent can rationally update its value estimates by simulating an internal model of the environment, provided that the model is accurate. In a series of sequential decision-making experiments, we investigated the impact of imaginative simulation on subsequent decisions. We found that imagination can cause people to pursue imagined paths, even when these paths are suboptimal. This bias is systematically related to participants' optimism about how much reward they expect to receive along imagined paths; providing feedback strongly attenuates the effect. The imagination effect can be captured by a reinforcement learning model that includes a bonus added onto imagined rewards. Using fMRI, we show that a network of regions associated with valuation is predictive of the imagination effect. These results suggest that imagination, although a powerful tool for learning, is also susceptible to motivational biases.
Collapse
|
46
|
|
47
|
Vyazovskiy VV, Walton ME, Peirson SN, Bannerman DM. Sleep homeostasis, habits and habituation. Curr Opin Neurobiol 2017; 44:202-211. [PMID: 28575718 DOI: 10.1016/j.conb.2017.05.002] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2016] [Revised: 04/06/2017] [Accepted: 05/01/2017] [Indexed: 02/08/2023]
Abstract
The importance of sleep for behavioural performance during waking is long-established, but the underlying reasons and mechanisms remain elusive. Waking and sleep are associated with changes in the levels of GluA1 AMPAR subunit in synaptic membranes, while studies using genetically-modified mice have identified an important role for GluA1-dependent synaptic plasticity in a non-associative form of memory that underlies short-term habituation to recently experienced stimuli. Here we posit that sleep may play a role in dishabituation, which restores attentional capacity and maximises the readiness of the animal for learning and goal-directed behaviour during subsequent wakefulness. Furthermore we suggest that sleep disturbance may fundamentally change the nature of behaviour, making it more model-free and habitual as a result of reduced attentional capacity.
Collapse
Affiliation(s)
- Vladyslav V Vyazovskiy
- Department of Physiology, Anatomy and Genetics, University of Oxford, Parks Road, Oxford, OX1 3PT, United Kingdom; Sleep and Circadian Neuroscience Institute, Oxford Molecular Pathology Institute, Sir William Dunn School of Pathology, South Parks Road, Oxford OX1 3RE, United Kingdom.
| | - Mark E Walton
- Department of Experimental Psychology, University of Oxford,South Parks Road, Oxford OX1 3UD, United Kingdom
| | - Stuart N Peirson
- Sleep and Circadian Neuroscience Institute, Oxford Molecular Pathology Institute, Sir William Dunn School of Pathology, South Parks Road, Oxford OX1 3RE, United Kingdom
| | - David M Bannerman
- Sleep and Circadian Neuroscience Institute, Oxford Molecular Pathology Institute, Sir William Dunn School of Pathology, South Parks Road, Oxford OX1 3RE, United Kingdom; Department of Experimental Psychology, University of Oxford,South Parks Road, Oxford OX1 3UD, United Kingdom
| |
Collapse
|
48
|
Fatigue modulates dopamine availability and promotes flexible choice reversals during decision making. Sci Rep 2017; 7:535. [PMID: 28373651 PMCID: PMC5428685 DOI: 10.1038/s41598-017-00561-6] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2016] [Accepted: 03/03/2017] [Indexed: 01/28/2023] Open
Abstract
During decisions, animals balance goal achievement and effort management. Despite physical exercise and fatigue significantly affecting the levels of effort that an animal exerts to obtain a reward, their role in effort-based choice and the underlying neurochemistry are incompletely known. In particular, it is unclear whether fatigue influences decision (cost-benefit) strategies flexibly or only post-decision action execution and learning. To answer this question, we trained mice on a T-maze task in which they chose between a high-cost, high-reward arm (HR), which included a barrier, and a low-cost, low-reward arm (LR), with no barrier. The animals were parametrically fatigued immediately before the behavioural tasks by running on a treadmill. We report a sharp choice reversal, from the HR to LR arm, at 80% of their peak workload (PW), which was temporary and specific, as the mice returned to choose the HC when the animals were successively tested at 60% PW or in a two-barrier task. These rapid reversals are signatures of flexible choice. We also observed increased subcortical dopamine levels in fatigued mice: a marker of individual bias to use model-based control in humans. Our results indicate that fatigue levels can be incorporated in flexible cost-benefits computations that improve foraging efficiency.
Collapse
|
49
|
Abstract
The nature and neural implementation of emotions is the subject of vigorous debate. Here, we use Bayesian decision theory to address key complexities in this field and conceptualize emotions in terms of their relationship to survival-relevant behavioural choices. Decision theory indicates which behaviours are optimal in a given situation; however, the calculations required are radically intractable. We therefore conjecture that the brain uses a range of pre-programmed algorithms that provide approximate solutions. These solutions seem to produce specific behavioural manifestations of emotions and can also be associated with core affective dimensions. We identify principles according to which these algorithms are implemented in the brain and illustrate our approach by considering decision making in the face of proximal threat.
Collapse
|
50
|
Rigoli F, Pezzulo G, Dolan R, Friston K. A Goal-Directed Bayesian Framework for Categorization. Front Psychol 2017; 8:408. [PMID: 28382008 PMCID: PMC5360703 DOI: 10.3389/fpsyg.2017.00408] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2016] [Accepted: 03/06/2017] [Indexed: 12/03/2022] Open
Abstract
Categorization is a fundamental ability for efficient behavioral control. It allows organisms to remember the correct responses to categorical cues and not for every stimulus encountered (hence eluding computational cost or complexity), and to generalize appropriate responses to novel stimuli dependant on category assignment. Assuming the brain performs Bayesian inference, based on a generative model of the external world and future goals, we propose a computational model of categorization in which important properties emerge. These properties comprise the ability to infer latent causes of sensory experience, a hierarchical organization of latent causes, and an explicit inclusion of context and action representations. Crucially, these aspects derive from considering the environmental statistics that are relevant to achieve goals, and from the fundamental Bayesian principle that any generative model should be preferred over alternative models based on an accuracy-complexity trade-off. Our account is a step toward elucidating computational principles of categorization and its role within the Bayesian brain hypothesis.
Collapse
Affiliation(s)
- Francesco Rigoli
- The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College London London, UK
| | - Giovanni Pezzulo
- Institute of Cognitive Sciences and Technologies - National Research Council Rome, Italy
| | - Raymond Dolan
- The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College LondonLondon, UK; Max Planck UCL Centre for Computational Psychiatry and Ageing ResearchLondon, UK
| | - Karl Friston
- The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College London London, UK
| |
Collapse
|