1
|
Qi S, Cross L, Wise T, Sui X, O'Doherty J, Mobbs D. The Role of the Medial Prefrontal Cortex in Spatial Margin of Safety Calculations. J Neurosci 2024; 44:e1162222024. [PMID: 38997158 DOI: 10.1523/jneurosci.1162-22.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2022] [Revised: 05/05/2023] [Accepted: 07/03/2024] [Indexed: 07/14/2024] Open
Abstract
Naturalistic observations show that animals pre-empt danger by moving to locations that increase their success in avoiding future threats. To test this in humans, we created a spatial margin of safety (MOS) decision task that quantifies pre-emptive avoidance by measuring the distance subjects place themselves to safety when facing different threats whose attack locations vary in predictability. Behavioral results show that human participants place themselves closer to safe locations when facing threats that attack in spatial locations with more outliers. Using both univariate and multivariate pattern analysis (MVPA) on fMRI data collected during a 2 h session on participants of both sexes, we demonstrate a dissociable role for the vmPFC in MOS-related decision-making. MVPA results revealed that the posterior vmPFC encoded for more unpredictable threats with univariate analyses showing a functional coupling with the amygdala and hippocampus. Conversely, the anterior vmPFC was more active for the more predictable attacks and showed coupling with the striatum. Our findings converge in showing that during pre-emptive danger, the anterior vmPFC may provide a safety signal, possibly via foreseeable outcomes, while the posterior vmPFC drives unpredictable danger signals.
Collapse
Affiliation(s)
- Song Qi
- Department of Humanities and Social Sciences and Computation, California Institute of Technology, Pasadena, California 91125
| | - Logan Cross
- Department of Humanities and Social Sciences and Computation, California Institute of Technology, Pasadena, California 91125
- Neural Systems Program at the California Institute of Technology, Pasadena, California 91125
| | - Toby Wise
- Department of Humanities and Social Sciences and Computation, California Institute of Technology, Pasadena, California 91125
| | - Xin Sui
- Department of Humanities and Social Sciences and Computation, California Institute of Technology, Pasadena, California 91125
| | - John O'Doherty
- Department of Humanities and Social Sciences and Computation, California Institute of Technology, Pasadena, California 91125
- Neural Systems Program at the California Institute of Technology, Pasadena, California 91125
| | - Dean Mobbs
- Department of Humanities and Social Sciences and Computation, California Institute of Technology, Pasadena, California 91125
- Neural Systems Program at the California Institute of Technology, Pasadena, California 91125
| |
Collapse
|
2
|
Eppinger B, Ruel A, Bolenz F. Diminished State Space Theory of Human Aging. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2023:17456916231204811. [PMID: 37931229 DOI: 10.1177/17456916231204811] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2023]
Abstract
Many new technologies, such as smartphones, computers, or public-access systems (like ticket-vending machines), are a challenge for older adults. One feature that these technologies have in common is that they involve underlying, partially observable, structures (state spaces) that determine the actions that are necessary to reach a certain goal (e.g., to move from one menu to another, to change a function, or to activate a new service). In this work we provide a theoretical, neurocomputational account to explain these behavioral difficulties in older adults. Based on recent findings from age-comparative computational- and cognitive-neuroscience studies, we propose that age-related impairments in complex goal-directed behavior result from an underlying deficit in the representation of state spaces of cognitive tasks. Furthermore, we suggest that these age-related deficits in adaptive decision-making are due to impoverished neural representations in the orbitofrontal cortex and hippocampus.
Collapse
Affiliation(s)
- Ben Eppinger
- Institute of Psychology, University of Greifswald
- Department of Psychology, Concordia University
- PERFORM Centre, Concordia University
- Faculty of Psychology, Technische Universität Dresden
| | - Alexa Ruel
- Department of Psychology, Concordia University
- PERFORM Centre, Concordia University
- Institute of Psychology, University of Hamburg
| | - Florian Bolenz
- Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany
- Science of Intelligence/Cluster of Excellence, Technical University of Berlin
| |
Collapse
|
3
|
Topel S, Ma I, Sleutels J, van Steenbergen H, de Bruijn ERA, van Duijvenvoorde ACK. Expecting the unexpected: a review of learning under uncertainty across development. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2023:10.3758/s13415-023-01098-0. [PMID: 37237092 PMCID: PMC10390612 DOI: 10.3758/s13415-023-01098-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 03/28/2023] [Indexed: 05/28/2023]
Abstract
Many of our decisions take place under uncertainty. To successfully navigate the environment, individuals need to estimate the degree of uncertainty and adapt their behaviors accordingly by learning from experiences. However, uncertainty is a broad construct and distinct types of uncertainty may differentially influence our learning. We provide a semi-systematic review to illustrate cognitive and neurobiological processes involved in learning under two types of uncertainty: learning in environments with stochastic outcomes, and with volatile outcomes. We specifically reviewed studies (N = 26 studies) that included an adolescent population, because adolescence is a period in life characterized by heightened exploration and learning, as well as heightened uncertainty due to experiencing many new, often social, environments. Until now, reviews have not comprehensively compared learning under distinct types of uncertainties in this age range. Our main findings show that although the overall developmental patterns were mixed, most studies indicate that learning from stochastic outcomes, as indicated by increased accuracy in performance, improved with age. We also found that adolescents tended to have an advantage compared with adults and children when learning from volatile outcomes. We discuss potential mechanisms explaining these age-related differences and conclude by outlining future research directions.
Collapse
Affiliation(s)
- Selin Topel
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands.
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands.
| | - Ili Ma
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Jan Sleutels
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden University, Institute for Philosophy, Leiden, The Netherlands
| | - Henk van Steenbergen
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Ellen R A de Bruijn
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Anna C K van Duijvenvoorde
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| |
Collapse
|
4
|
Bakst L, McGuire JT. Experience-driven recalibration of learning from surprising events. Cognition 2023; 232:105343. [PMID: 36481590 PMCID: PMC9851993 DOI: 10.1016/j.cognition.2022.105343] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2022] [Revised: 10/13/2022] [Accepted: 11/21/2022] [Indexed: 12/12/2022]
Abstract
Different environments favor different patterns of adaptive learning. A surprising event that in one context would accelerate belief updating might, in another context, be downweighted as a meaningless outlier. Here, we investigated whether people would spontaneously regulate the influence of surprise on learning in response to event-by-event experiential feedback. Across two experiments, we examined whether participants performing a perceptual judgment task under spatial uncertainty (n = 29, n = 63) adapted their patterns of predictive gaze according to the informativeness or uninformativeness of surprising events in their current environment. Uninstructed predictive eye movements exhibited a form of metalearning in which surprise came to modulate event-by-event learning rates in opposite directions across contexts. Participants later appropriately readjusted their patterns of adaptive learning when the statistics of the environment underwent an unsignaled reversal. Although significant adjustments occurred in both directions, performance was consistently superior in environments in which surprising events reflected meaningful change, potentially reflecting a bias towards interpreting surprise as informative and/or difficulty ignoring salient outliers. Our results provide evidence for spontaneous, context-appropriate recalibration of the role of surprise in adaptive learning.
Collapse
Affiliation(s)
- Leah Bakst
- Department of Psychological & Brain Sciences, Boston University, 64 Cummington Mall, Boston, MA 02215, USA; Center for Systems Neuroscience, Boston University, 610 Commonwealth Avenue, Boston, MA 02215, USA.
| | - Joseph T McGuire
- Department of Psychological & Brain Sciences, Boston University, 64 Cummington Mall, Boston, MA 02215, USA; Center for Systems Neuroscience, Boston University, 610 Commonwealth Avenue, Boston, MA 02215, USA.
| |
Collapse
|
5
|
Kirschner H, Fischer AG, Ullsperger M. Feedback-related EEG dynamics separately reflect decision parameters, biases, and future choices. Neuroimage 2022; 259:119437. [PMID: 35788041 DOI: 10.1016/j.neuroimage.2022.119437] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2022] [Revised: 06/17/2022] [Accepted: 06/30/2022] [Indexed: 11/17/2022] Open
Abstract
Optimal decision making in complex environments requires dynamic learning from unexpected events. To speed up learning, we should heavily weight information that indicates state-action-outcome contingency changes and ignore uninformative fluctuations in the environment. Often, however, unrelated information is hard to ignore and can potentially bias our learning. Here we used computational modelling and EEG to investigate learning behaviour in a modified probabilistic choice task that introduced two task-irrelevant factors that were uninformative for optimal task performance, but nevertheless could potentially bias learning: pay-out magnitudes were varied randomly and, occasionally, feedback presentation was enhanced by visual surprise. We found that participants' overall good learning performance was biased by distinct effects of these non-normative factors. On the neural level, these parameters are represented in a dynamic and spatiotemporally dissociable sequence of EEG activity. Later in feedback processing the different streams converged on a central to centroparietal positivity reflecting a signal that is interpreted by downstream learning processes that adjust future behaviour.
Collapse
Affiliation(s)
- Hans Kirschner
- Institute of Psychology, Otto-von-Guericke University, D-39106 Magdeburg, Germany.
| | - Adrian G Fischer
- Department of Education and Psychology, Freie Universität Berlin, D-14195 Berlin, Germany
| | - Markus Ullsperger
- Institute of Psychology, Otto-von-Guericke University, D-39106 Magdeburg, Germany; Center for Behavioral Brain Sciences, D-39106 Magdeburg, Germany
| |
Collapse
|
6
|
Bujold PM, Seak LCU, Schultz W, Ferrari-Toniolo S. Comparing utility functions between risky and riskless choice in rhesus monkeys. Anim Cogn 2022; 25:385-399. [PMID: 34568979 PMCID: PMC8940808 DOI: 10.1007/s10071-021-01560-x] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2021] [Revised: 08/10/2021] [Accepted: 09/14/2021] [Indexed: 11/29/2022]
Abstract
Decisions can be risky or riskless, depending on the outcomes of the choice. Expected utility theory describes risky choices as a utility maximization process: we choose the option with the highest subjective value (utility), which we compute considering both the option's value and its associated risk. According to the random utility maximization framework, riskless choices could also be based on a utility measure. Neuronal mechanisms of utility-based choice may thus be common to both risky and riskless choices. This assumption would require the existence of a utility function that accounts for both risky and riskless decisions. Here, we investigated whether the choice behavior of two macaque monkeys in risky and riskless decisions could be described by a common underlying utility function. We found that the utility functions elicited in the two choice scenarios were different from each other, even after taking into account the contribution of subjective probability weighting. Our results suggest that distinct utility representations exist for risky and riskless choices, which could reflect distinct neuronal representations of the utility quantities, or distinct brain mechanisms for risky and riskless choices. The different utility functions should be taken into account in neuronal investigations of utility-based choice.
Collapse
Affiliation(s)
- Philipe M. Bujold
- Department of Physiology, Development and Neuroscience, University of Cambridge, Cambridge, CB2 3DY UK
| | - Leo Chi U. Seak
- Department of Physiology, Development and Neuroscience, University of Cambridge, Cambridge, CB2 3DY UK
| | - Wolfram Schultz
- Department of Physiology, Development and Neuroscience, University of Cambridge, Cambridge, CB2 3DY UK
| | - Simone Ferrari-Toniolo
- Department of Physiology, Development and Neuroscience, University of Cambridge, Cambridge, CB2 3DY UK
| |
Collapse
|
7
|
Liakoni V, Lehmann MP, Modirshanechi A, Brea J, Lutti A, Gerstner W, Preuschoff K. Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making. Neuroimage 2021; 246:118780. [PMID: 34875383 DOI: 10.1016/j.neuroimage.2021.118780] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2021] [Revised: 08/03/2021] [Accepted: 12/04/2021] [Indexed: 11/25/2022] Open
Abstract
Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.
Collapse
Affiliation(s)
- Vasiliki Liakoni
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland.
| | - Marco P Lehmann
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Alireza Modirshanechi
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Johanni Brea
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Antoine Lutti
- Laboratoire de recherche en neuroimagerie (LREN), Department of Clinical Neurosciences, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
| | - Wulfram Gerstner
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Kerstin Preuschoff
- Geneva Finance Research Institute & Interfaculty Center for Affective Sciences, University of Geneva, Geneva, Switzerland
| |
Collapse
|
8
|
Yu LQ, Wilson RC, Nassar MR. Adaptive learning is structure learning in time. Neurosci Biobehav Rev 2021; 128:270-281. [PMID: 34144114 PMCID: PMC8422504 DOI: 10.1016/j.neubiorev.2021.06.024] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2020] [Revised: 04/19/2021] [Accepted: 06/11/2021] [Indexed: 10/21/2022]
Abstract
People use information flexibly. They often combine multiple sources of relevant information over time in order to inform decisions with little or no interference from intervening irrelevant sources. They adjust the degree to which they use new information over time rationally in accordance with environmental statistics and their own uncertainty. They can even use information gained in one situation to solve a problem in a very different one. Learning flexibly rests on the ability to infer the context at a given time, and therefore knowing which pieces of information to combine and which to separate. We review the psychological and neural mechanisms behind adaptive learning and structure learning to outline how people pool together relevant information, demarcate contexts, prevent interference between information collected in different contexts, and transfer information from one context to another. By examining all of these processes through the lens of optimal inference we bridge concepts from multiple fields to provide a unified multi-system view of how the brain exploits structure in time to optimize learning.
Collapse
Affiliation(s)
- Linda Q Yu
- Carney Institute for Brain Sciences, Brown University, 164 Angell Street, Providence, RI, 02912, USA.
| | - Robert C Wilson
- Department of Psychology, University of Arizona, Tucson, AZ, 85721, USA
| | - Matthew R Nassar
- Carney Institute for Brain Sciences, Brown University, 164 Angell Street, Providence, RI, 02912, USA
| |
Collapse
|
9
|
Nassar MR, Waltz JA, Albrecht MA, Gold JM, Frank MJ. All or nothing belief updating in patients with schizophrenia reduces precision and flexibility of beliefs. Brain 2021; 144:1013-1029. [PMID: 33434284 DOI: 10.1093/brain/awaa453] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2020] [Revised: 10/09/2020] [Accepted: 10/14/2020] [Indexed: 01/11/2023] Open
Abstract
Schizophrenia is characterized by abnormal perceptions and beliefs, but the computational mechanisms through which these abnormalities emerge remain unclear. One prominent hypothesis asserts that such abnormalities result from overly precise representations of prior knowledge, which in turn lead beliefs to become insensitive to feedback. In contrast, another prominent hypothesis asserts that such abnormalities result from a tendency to interpret prediction errors as indicating meaningful change, leading to the assignment of aberrant salience to noisy or misleading information. Here we examine behaviour of patients and control subjects in a behavioural paradigm capable of adjudicating between these competing hypotheses and characterizing belief updates directly on individual trials. We show that patients are more prone to completely ignoring new information and perseverating on previous responses, but when they do update, tend to do so completely. This updating strategy limits the integration of information over time, reducing both the flexibility and precision of beliefs and provides a potential explanation for how patients could simultaneously show over-sensitivity and under-sensitivity to feedback in different paradigms.
Collapse
Affiliation(s)
- Matthew R Nassar
- Robert J. and Nancy D. Carney Institute for Brain Science, Brown University, Providence RI 02912-1821, USA.,Department of Neuroscience, Brown University, Providence RI 02912-1821, USA
| | - James A Waltz
- Maryland Psychiatric Research Center, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Matthew A Albrecht
- School of Public Health, Curtin University, Perth, Western Australia, Australia
| | - James M Gold
- Maryland Psychiatric Research Center, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Michael J Frank
- Robert J. and Nancy D. Carney Institute for Brain Science, Brown University, Providence RI 02912-1821, USA.,Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence RI 02912-1821, USA
| |
Collapse
|
10
|
Liu M, Dong W, Qin S, Verguts T, Chen Q. Electrophysiological Signatures of Hierarchical Learning. Cereb Cortex 2021; 32:626-639. [PMID: 34339505 DOI: 10.1093/cercor/bhab245] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2021] [Revised: 06/26/2021] [Accepted: 06/27/2021] [Indexed: 11/13/2022] Open
Abstract
Human perception and learning is thought to rely on a hierarchical generative model that is continuously updated via precision-weighted prediction errors (pwPEs). However, the neural basis of such cognitive process and how it unfolds during decision-making remain poorly understood. To investigate this question, we combined a hierarchical Bayesian model (i.e., Hierarchical Gaussian Filter [HGF]) with electroencephalography (EEG), while participants performed a probabilistic reversal learning task in alternatingly stable and volatile environments. Behaviorally, the HGF fitted significantly better than two control, nonhierarchical, models. Neurally, low-level and high-level pwPEs were independently encoded by the P300 component. Low-level pwPEs were reflected in the theta (4-8 Hz) frequency band, but high-level pwPEs were not. Furthermore, the expressions of high-level pwPEs were stronger for participants with better HGF fit. These results indicate that the brain employs hierarchical learning and encodes both low- and high-level learning signals separately and adaptively.
Collapse
Affiliation(s)
- Meng Liu
- Key Laboratory of Brain, Cognition and Education Sciences (South China Normal University), Ministry of Education, 510631 Guangzhou, China.,School of Psychology, South China Normal University, 510631 Guangzhou, China.,Center for Studies of Psychological Application, South China Normal University, 510631 Guangzhou, China.,Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, 510631 Guangzhou, China
| | - Wenshan Dong
- Key Laboratory of Brain, Cognition and Education Sciences (South China Normal University), Ministry of Education, 510631 Guangzhou, China.,School of Psychology, South China Normal University, 510631 Guangzhou, China.,Center for Studies of Psychological Application, South China Normal University, 510631 Guangzhou, China.,Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, 510631 Guangzhou, China
| | - Shaozheng Qin
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, 100875 Beijing, China
| | - Tom Verguts
- Department of Experimental Psychology, Ghent University, B-9000 Ghent, Belgium
| | - Qi Chen
- Key Laboratory of Brain, Cognition and Education Sciences (South China Normal University), Ministry of Education, 510631 Guangzhou, China.,School of Psychology, South China Normal University, 510631 Guangzhou, China.,Center for Studies of Psychological Application, South China Normal University, 510631 Guangzhou, China.,Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, 510631 Guangzhou, China
| |
Collapse
|
11
|
Bossaerts P. How Neurobiology Elucidates the Role of Emotions in Financial Decision-Making. Front Psychol 2021; 12:697375. [PMID: 34349708 PMCID: PMC8326835 DOI: 10.3389/fpsyg.2021.697375] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2021] [Accepted: 06/09/2021] [Indexed: 11/13/2022] Open
Abstract
Over the last 15 years, a revolution has been taking place in neuroscience, whereby models and methods of economics have led to deeper insights into the neurobiological foundations of human decision-making. These have revealed a number of widespread mis-conceptions, among others, about the role of emotions. Furthermore, the findings suggest that a purely behavior-based approach to studying decisions may miss crucial features of human choice long appreciated in biology, such as Pavlovian approach. The findings could help economists formalize elusive concepts such as intuition, as I show here for financial “trading intuition.”
Collapse
Affiliation(s)
- Peter Bossaerts
- Faculty of Business and Economics, University of Melbourne, Parkville, VIC, Australia.,Faculty of Economics, University of Cambridge, Cambridge, United Kingdom.,Geneva School of Economics and Management, University of Geneva, Geneva, Switzerland
| |
Collapse
|
12
|
Nassar MR, Troiani V. The stability flexibility tradeoff and the dark side of detail. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2021; 21:607-623. [PMID: 33236296 PMCID: PMC8141540 DOI: 10.3758/s13415-020-00848-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 11/04/2020] [Indexed: 12/26/2022]
Abstract
Learning in dynamic environments requires integrating over stable fluctuations to minimize the impact of noise (stability) but rapidly responding in the face of fundamental changes (flexibility). Achieving one of these goals often requires sacrificing the other to some degree, producing a stability-flexibility tradeoff. Individuals navigate this tradeoff in different ways; some people learn rapidly (emphasizing flexibility) and others rely more heavily on historical information (emphasizing stability). Despite the prominence of such individual differences in learning tasks, the degree to which they relate to broader characteristics of real-world behavior or pathologies has not been well explored. We relate individual differences in learning behavior to self-report measures thought to capture collectively the characteristics of the Autism spectrum. We show that young adults who learn most slowly tend to integrate more effective samples into their beliefs about the world making them more robust to noise (more stability) but are more likely to integrate information from previous contexts (less flexibility). We show that individuals who report paying more attention to detail tend to use high flexibility and low stability information processing strategies. We demonstrate the robustness of this inverse relationship between attention to detail and formation of stable beliefs in a heterogeneous population of children that includes a high proportion of Autism diagnoses. Together, our results highlight that attention to detail reflects an information processing policy that comes with a substantial downside, namely the ability to integrate data to overcome environmental noise.
Collapse
Affiliation(s)
- Matthew R Nassar
- Department of Neuroscience; Carney Institute for Brain Science, Brown University, Providence, RI, 02912-1821, USA.
| | - Vanessa Troiani
- Geisinger-Bucknell Autism & Developmental Medicine Institute, Lewisburg, PA, USA
| |
Collapse
|
13
|
Rothenhoefer KM, Hong T, Alikaya A, Stauffer WR. Rare rewards amplify dopamine responses. Nat Neurosci 2021; 24:465-469. [PMID: 33686298 PMCID: PMC9373731 DOI: 10.1038/s41593-021-00807-7] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2019] [Accepted: 01/20/2021] [Indexed: 01/02/2023]
Abstract
Dopamine prediction error responses are essential components of universal learning mechanisms. However, it is unknown whether individual dopamine neurons reflect the shape of reward distributions. Here, we used symmetrical distributions with differently weighted tails to investigate how the frequency of rewards and reward prediction errors influence dopamine signals. Rare rewards amplified dopamine responses, even when conventional prediction errors were identical, indicating a mechanism for learning the complexities of real-world incentives.
Collapse
Affiliation(s)
- Kathryn M Rothenhoefer
- Center for Neuroscience, University of Pittsburgh, Pittsburgh, PA, USA
- Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA
- Systems Neuroscience Center, University of Pittsburgh, Pittsburgh, PA, USA
- The Brain Institute, University of Pittsburgh, Pittsburgh, PA, USA
| | - Tao Hong
- Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA
- Systems Neuroscience Center, University of Pittsburgh, Pittsburgh, PA, USA
- The Brain Institute, University of Pittsburgh, Pittsburgh, PA, USA
- Program in Neural Computation, Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Aydin Alikaya
- Center for Neuroscience, University of Pittsburgh, Pittsburgh, PA, USA
- Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA
- Systems Neuroscience Center, University of Pittsburgh, Pittsburgh, PA, USA
- The Brain Institute, University of Pittsburgh, Pittsburgh, PA, USA
| | - William R Stauffer
- Center for Neuroscience, University of Pittsburgh, Pittsburgh, PA, USA.
- Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA.
- Systems Neuroscience Center, University of Pittsburgh, Pittsburgh, PA, USA.
- The Brain Institute, University of Pittsburgh, Pittsburgh, PA, USA.
| |
Collapse
|
14
|
Liakoni V, Modirshanechi A, Gerstner W, Brea J. Learning in Volatile Environments With the Bayes Factor Surprise. Neural Comput 2021; 33:269-340. [PMID: 33400898 DOI: 10.1162/neco_a_01352] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
Abstract
Surprise-based learning allows agents to rapidly adapt to nonstationary stochastic environments characterized by sudden changes. We show that exact Bayesian inference in a hierarchical model gives rise to a surprise-modulated trade-off between forgetting old observations and integrating them with the new ones. The modulation depends on a probability ratio, which we call the Bayes Factor Surprise, that tests the prior belief against the current belief. We demonstrate that in several existing approximate algorithms, the Bayes Factor Surprise modulates the rate of adaptation to new observations. We derive three novel surprise-based algorithms, one in the family of particle filters, one in the family of variational learning, and one in the family of message passing, that have constant scaling in observation sequence length and particularly simple update dynamics for any distribution in the exponential family. Empirical results show that these surprise-based algorithms estimate parameters better than alternative approximate approaches and reach levels of performance comparable to computationally more expensive algorithms. The Bayes Factor Surprise is related to but different from the Shannon Surprise. In two hypothetical experiments, we make testable predictions for physiological indicators that dissociate the Bayes Factor Surprise from the Shannon Surprise. The theoretical insight of casting various approaches as surprise-based learning, as well as the proposed online algorithms, may be applied to the analysis of animal and human behavior and to reinforcement learning in nonstationary environments.
Collapse
Affiliation(s)
- Vasiliki Liakoni
- École Polytechnique Fédérale de Lausanne, School of Computer and Communication Sciences and School of Life Sciences, 1015 Lausanne, Switzerland
| | - Alireza Modirshanechi
- École Polytechnique Fédérale de Lausanne, School of Computer and Communication Sciences and School of Life Sciences, 1015 Lausanne, Switzerland
| | - Wulfram Gerstner
- École Polytechnique Fédérale de Lausanne, School of Computer and Communication Sciences and School of Life Sciences, 1015 Lausanne, Switzerland
| | - Johanni Brea
- École Polytechnique Fédérale de Lausanne, School of Computer and Communication Sciences and School of Life Sciences, 1015 Lausanne, Switzerland
| |
Collapse
|
15
|
Kao CH, Lee S, Gold JI, Kable JW. Neural encoding of task-dependent errors during adaptive learning. eLife 2020; 9:58809. [PMID: 33074104 PMCID: PMC7584453 DOI: 10.7554/elife.58809] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2020] [Accepted: 10/15/2020] [Indexed: 01/20/2023] Open
Abstract
Effective learning requires using errors in a task-dependent manner, for example adjusting to errors that result from unpredicted environmental changes but ignoring errors that result from environmental stochasticity. Where and how the brain represents errors in a task-dependent manner and uses them to guide behavior are not well understood. We imaged the brains of human participants performing a predictive-inference task with two conditions that had different sources of errors. Their performance was sensitive to this difference, including more choice switches after fundamental changes versus stochastic fluctuations in reward contingencies. Using multi-voxel pattern classification, we identified task-dependent representations of error magnitude and past errors in posterior parietal cortex. These representations were distinct from representations of the resulting behavioral adjustments in dorsomedial frontal, anterior cingulate, and orbitofrontal cortex. The results provide new insights into how the human brain represents errors in a task-dependent manner and guides subsequent adaptive behavior.
Collapse
Affiliation(s)
- Chang-Hao Kao
- Department of Psychology, University of Pennsylvania, Philadelphia, United States
| | - Sangil Lee
- Department of Psychology, University of Pennsylvania, Philadelphia, United States
| | - Joshua I Gold
- Department of Neuroscience, University of Pennsylvania, Philadelphia, United States
| | - Joseph W Kable
- Department of Psychology, University of Pennsylvania, Philadelphia, United States
| |
Collapse
|
16
|
Bakst L, McGuire JT. Eye movements reflect adaptive predictions and predictive precision. J Exp Psychol Gen 2020; 150:915-929. [PMID: 33048566 DOI: 10.1037/xge0000977] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Successful decision-making depends on the ability to form predictions about uncertain future events. Existing evidence suggests predictive representations are not limited to point estimates but also include information about the associated level of predictive uncertainty. Estimates of predictive uncertainty have an important role in governing the rate at which beliefs are updated in response to new observations. It is not yet known, however, whether the same form of uncertainty-modulated learning occurs naturally and spontaneously when there is no task requirement to express predictions explicitly. Here, we used a gaze-based predictive inference paradigm to show that (a) predictive inference manifested in spontaneous gaze dynamics, (b) feedback-driven updating of spontaneous gaze-based predictions reflected adaptation to environmental statistics, and (c) anticipatory gaze variability tracked predictive uncertainty in an event-by-event manner. Our results demonstrate that sophisticated predictive inference can occur spontaneously and that oculomotor behavior can provide a multidimensional readout of internal predictive beliefs. (PsycInfo Database Record (c) 2021 APA, all rights reserved).
Collapse
Affiliation(s)
- Leah Bakst
- Department of Psychological and Brain Sciences, Boston University
| | - Joseph T McGuire
- Department of Psychological and Brain Sciences, Boston University
| |
Collapse
|
17
|
Nassar MR, Bruckner R, Frank MJ. Statistical context dictates the relationship between feedback-related EEG signals and learning. eLife 2019; 8:e46975. [PMID: 31433294 PMCID: PMC6716947 DOI: 10.7554/elife.46975] [Citation(s) in RCA: 32] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2019] [Accepted: 08/12/2019] [Indexed: 12/18/2022] Open
Abstract
Learning should be adjusted according to the surprise associated with observed outcomes but calibrated according to statistical context. For example, when occasional changepoints are expected, surprising outcomes should be weighted heavily to speed learning. In contrast, when uninformative outliers are expected to occur occasionally, surprising outcomes should be less influential. Here we dissociate surprising outcomes from the degree to which they demand learning using a predictive inference task and computational modeling. We show that the P300, a stimulus-locked electrophysiological response previously associated with adjustments in learning behavior, does so conditionally on the source of surprise. Larger P300 signals predicted greater learning in a changing context, but less learning in a context where surprise was indicative of a one-off outlier (oddball). Our results suggest that the P300 provides a surprise signal that is interpreted by downstream learning processes differentially according to statistical context in order to appropriately calibrate learning across complex environments.
Collapse
Affiliation(s)
- Matthew R Nassar
- Robert J. & Nancy D. Carney Institute for Brain ScienceBrown UniversityProvidenceUnited States
- Department of NeuroscienceBrown UniversityProvidenceUnited States
| | - Rasmus Bruckner
- Department of Education and PsychologyFreie Universität BerlinBerlinGermany
- Center for Lifespan PsychologyMax Planck Institute for Human DevelopmentBerlinGermany
- International Max Planck Research School on the Life Course (LIFE)BerlinGermany
| | - Michael J Frank
- Robert J. & Nancy D. Carney Institute for Brain ScienceBrown UniversityProvidenceUnited States
- Department of Cognitive, Linguistic, and Psychological SciencesBrown UniversityProvidenceUnited States
| |
Collapse
|
18
|
Bossaerts P. Formalizing the Function of Anterior Insula in Rapid Adaptation. Front Integr Neurosci 2018; 12:61. [PMID: 30568581 PMCID: PMC6290341 DOI: 10.3389/fnint.2018.00061] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2018] [Accepted: 11/20/2018] [Indexed: 02/04/2023] Open
Abstract
Anterior insula (aIns) is thought to play a crucial role in rapid adaptation in an ever-changing environment. Mathematically, it is known to track risk and surprise. Modern theories of learning, however, assign a dominant role to signed prediction errors (PEs), not to risk and surprise. Risk and surprise only enter to the extent that they modulate the learning rate, in an attempt to approximate Bayesian learning. Even without such modulation, adaptation is still possible, albeit slow. Here, I propose a new theory of learning, reference-model based learning (RMBL), where risk and surprise are central, and PEs play a secondary, though still crucial, role. The primary goal is to bring outcomes in line with expectations in the reference model (RM). Learning is modulated by how large the PEs are relative to model anticipation, i.e., to surprise as defined by the RM. In a target location prediction task where participants were continuously required to adapt, choices appeared to be closer with to RMBL predictions than to Bayesian learning. aIns reaction to surprise was more acute in the more difficult treatment, consistent with its hypothesized role in metacognition. I discuss links with related theories, such as Active Inference, Actor-Critic Models and Reference-Model Based Adaptive Control.
Collapse
Affiliation(s)
- Peter Bossaerts
- Department of Finance, Faculty of Business and Economics, The University of Melbourne, Melbourne, VIC, Australia.,The Florey Institute of Neuroscience and Mental Health, Parkville, VIC, Australia
| |
Collapse
|
19
|
Zylberberg A, Fetsch CR, Shadlen MN. The influence of evidence volatility on choice, reaction time and confidence in a perceptual decision. eLife 2016; 5. [PMID: 27787198 PMCID: PMC5083065 DOI: 10.7554/elife.17688] [Citation(s) in RCA: 59] [Impact Index Per Article: 7.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2016] [Accepted: 09/29/2016] [Indexed: 12/25/2022] Open
Abstract
Many decisions are thought to arise via the accumulation of noisy evidence to a threshold or bound. In perception, the mechanism explains the effect of stimulus strength, characterized by signal-to-noise ratio, on decision speed, accuracy and confidence. It also makes intriguing predictions about the noise itself. An increase in noise should lead to faster decisions, reduced accuracy and, paradoxically, higher confidence. To test these predictions, we introduce a novel sensory manipulation that mimics the addition of unbiased noise to motion-selective regions of visual cortex, which we verified with neuronal recordings from macaque areas MT/MST. For both humans and monkeys, increasing the noise induced faster decisions and greater confidence over a range of stimuli for which accuracy was minimally impaired. The magnitude of the effects was in agreement with predictions of a bounded evidence accumulation model. DOI:http://dx.doi.org/10.7554/eLife.17688.001 Many of our decisions are made on the basis of imperfect or ‘noisy’ information. A longstanding goal in neuroscience is to work out how such noise affects three aspects of decision-making: the accuracy (or appropriateness) of a choice, the speed at which the choice is made, and the decision-maker’s confidence that they have chosen correctly. One theory of decision-making is that the brain simultaneously accumulates evidence for each of the options it is considering, until one option exceeds a threshold and is declared the ‘winner’. This theory is known as bounded evidence accumulation. It predicts that increasing the noisiness of the available information decreases the accuracy of decisions made in response. Counterintuitively, it also predicts that such an increase in noise speeds up decision-making and increases confidence levels. Zylberberg et al. have now tested these predictions experimentally by getting human volunteers and monkeys to perform a series of trials where they had to decide whether a set of randomly moving dots moved to the left or to the right overall. Using a newly developed method, the noisiness of the dot motion could be changed between trials. The effectiveness of this technique was confirmed by recording the activity of neurons in the region of the monkey brain that processes visual motion information. After each trial, the humans rated their confidence in their decision. By comparison, the monkeys could indicate that they were not confident in a decision by opting for a guaranteed small reward on certain trials (instead of the larger reward they received when they correctly indicated the direction of motion of the dots). In both humans and monkeys, increasing the noisiness associated with the movement of the dots led to faster and more confident decision-making, just as the bounded evidence accumulation framework predicts. Furthermore, the results presented by Zylberberg et al. suggest that the brain does not always gauge how reliable evidence is in order to fine-tune decisions. Now that the role of noise in decision-making is better understood, future experiments could attempt to reveal how artificial manipulations of the brain contribute both information and noise to a decision. Other experiments might ascertain when the brain can learn that noisy information should invite slower, more cautious decisions. DOI:http://dx.doi.org/10.7554/eLife.17688.002
Collapse
Affiliation(s)
- Ariel Zylberberg
- Kavli Institute, Department of Neuroscience, Howard Hughes Medical Institute, Columbia University, New York, United States.,Zuckerman Mind Brain Behavior Institute, Department of Neuroscience, Howard Hughes Medical Institute, Columbia University, New York, United States
| | - Christopher R Fetsch
- Kavli Institute, Department of Neuroscience, Howard Hughes Medical Institute, Columbia University, New York, United States.,Zuckerman Mind Brain Behavior Institute, Department of Neuroscience, Howard Hughes Medical Institute, Columbia University, New York, United States
| | - Michael N Shadlen
- Kavli Institute, Department of Neuroscience, Howard Hughes Medical Institute, Columbia University, New York, United States.,Zuckerman Mind Brain Behavior Institute, Department of Neuroscience, Howard Hughes Medical Institute, Columbia University, New York, United States
| |
Collapse
|