1
|
Ger Y, Shahar M, Shahar N. Using recurrent neural network to estimate irreducible stochasticity in human choice behavior. eLife 2024; 13:RP90082. [PMID: 39240757 PMCID: PMC11379453 DOI: 10.7554/elife.90082] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/08/2024] Open
Abstract
Theoretical computational models are widely used to describe latent cognitive processes. However, these models do not equally explain data across participants, with some individuals showing a bigger predictive gap than others. In the current study, we examined the use of theory-independent models, specifically recurrent neural networks (RNNs), to classify the source of a predictive gap in the observed data of a single individual. This approach aims to identify whether the low predictability of behavioral data is mainly due to noisy decision-making or misspecification of the theoretical model. First, we used computer simulation in the context of reinforcement learning to demonstrate that RNNs can be used to identify model misspecification in simulated agents with varying degrees of behavioral noise. Specifically, both prediction performance and the number of RNN training epochs (i.e., the point of early stopping) can be used to estimate the amount of stochasticity in the data. Second, we applied our approach to an empirical dataset where the actions of low IQ participants, compared with high IQ participants, showed lower predictability by a well-known theoretical model (i.e., Daw's hybrid model for the two-step task). Both the predictive gap and the point of early stopping of the RNN suggested that model misspecification is similar across individuals. This led us to a provisional conclusion that low IQ subjects are mostly noisier compared to their high IQ peers, rather than being more misspecified by the theoretical model. We discuss the implications and limitations of this approach, considering the growing literature in both theoretical and data-driven computational modeling in decision-making science.
Collapse
Affiliation(s)
- Yoav Ger
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
| | - Moni Shahar
- TAD, Center of AI & Data Science, Tel Aviv University, Tel Aviv, Israel
| | - Nitzan Shahar
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel
| |
Collapse
|
2
|
Lin W, Dolan RJ. Decision-Making, Pro-variance Biases and Mood-Related Traits. COMPUTATIONAL PSYCHIATRY (CAMBRIDGE, MASS.) 2024; 8:142-158. [PMID: 39184228 PMCID: PMC11342847 DOI: 10.5334/cpsy.114] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/25/2024] [Accepted: 07/19/2024] [Indexed: 08/27/2024]
Abstract
In value-based decision-making there is wide behavioural variability in how individuals respond to uncertainty. Maladaptive responses to uncertainty have been linked to a vulnerability to mental illness, for example, between risk aversion and affective disorders. Here, we examine individual differences in risk sensitivity when subjects confront options drawn from different value distributions, where these embody the same or different means and variances. In simulations, we show that a model that learns a distribution using Bayes' rule and reads out different parts of the distribution under the influence of a risk-sensitive parameter (Conditional Value at Risk, CVaR) predicts how likely an agent is to prefer a broader over a narrow distribution (pro-variance bias/risk-seeking) under the same overall means. Using empirical data, we show that CVaR estimates correlate with participants' pro-variance biases better than a range of alternative parameters derived from other models. Importantly, across two independent samples, CVaR estimates and participants' pro-variance bias negatively correlated with trait rumination, a common trait in depression and anxiety. We conclude that a Bayesian-CVaR model captures individual differences in sensitivity to variance in value distributions and task-independent trait dispositions linked to affective disorders.
Collapse
Affiliation(s)
- Wanjun Lin
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, University College London, London WC1B 5EH, UK
| | - Raymond J. Dolan
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, University College London, London WC1B 5EH, UK
- Welcome Centre for Human Neuroimaging, University College London, London WC1N 3BG, UK
| |
Collapse
|
3
|
Higashi H. Dynamics of visual attention in exploration and exploitation for reward-guided adjustment tasks. Conscious Cogn 2024; 123:103724. [PMID: 38996747 DOI: 10.1016/j.concog.2024.103724] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2024] [Revised: 06/24/2024] [Accepted: 06/26/2024] [Indexed: 07/14/2024]
Abstract
The learning process encompasses exploration and exploitation phases. While reinforcement learning models have revealed functional and neuroscientific distinctions between these phases, knowledge regarding how they affect visual attention while observing the external environment is limited. This study sought to elucidate the interplay between these learning phases and visual attention allocation using visual adjustment tasks combined with a two-armed bandit problem tailored to detect serial effects only when attention is dispersed across both arms. Per our findings, human participants exhibited a distinct serial effect only during the exploration phase, suggesting enhanced attention to the visual stimulus associated with the non-target arm. Remarkably, although rewards did not motivate attention dispersion in our task, during the exploration phase, individuals engaged in active observation and searched for targets to observe. This behavior highlights a unique information-seeking process in exploration that is distinct from exploitation.
Collapse
Affiliation(s)
- Hiroshi Higashi
- Graduate School of Engineering, Osaka University, Suita, Osaka, Japan.
| |
Collapse
|
4
|
Cinotti F, Coutureau E, Khamassi M, Marchand AR, Girard B. Regulation of reinforcement learning parameters captures long-term changes in rat behaviour. Eur J Neurosci 2024; 60:4469-4490. [PMID: 38923238 DOI: 10.1111/ejn.16449] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2023] [Revised: 05/14/2024] [Accepted: 06/05/2024] [Indexed: 06/28/2024]
Abstract
In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.
Collapse
Affiliation(s)
- François Cinotti
- Institut des Systèmes Intelligents et de Robotique, Sorbonne Université, CNRS, Paris, France
- University of Reading, School of Psychology and Clinical Language Sciences, Whiteknights, Reading, UK
| | | | - Mehdi Khamassi
- Institut des Systèmes Intelligents et de Robotique, Sorbonne Université, CNRS, Paris, France
| | | | - Benoît Girard
- Institut des Systèmes Intelligents et de Robotique, Sorbonne Université, CNRS, Paris, France
| |
Collapse
|
5
|
Grujic N, Polania R, Burdakov D. Neurobehavioral meaning of pupil size. Neuron 2024:S0896-6273(24)00406-9. [PMID: 38925124 DOI: 10.1016/j.neuron.2024.05.029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2023] [Revised: 03/22/2024] [Accepted: 05/31/2024] [Indexed: 06/28/2024]
Abstract
Pupil size is a widely used metric of brain state. It is one of the few signals originating from the brain that can be readily monitored with low-cost devices in basic science, clinical, and home settings. It is, therefore, important to investigate and generate well-defined theories related to specific interpretations of this metric. What exactly does it tell us about the brain? Pupils constrict in response to light and dilate during darkness, but the brain also controls pupil size irrespective of luminosity. Pupil size fluctuations resulting from ongoing "brain states" are used as a metric of arousal, but what is pupil-linked arousal and how should it be interpreted in neural, cognitive, and computational terms? Here, we discuss some recent findings related to these issues. We identify open questions and propose how to answer them through a combination of well-defined tasks, neurocomputational models, and neurophysiological probing of the interconnected loops of causes and consequences of pupil size.
Collapse
Affiliation(s)
- Nikola Grujic
- Neurobehavioural Dynamics Lab, ETH Zürich, Department of Health Sciences and Technology, Schorenstrasse 16, 8603 Schwerzenbach, Switzerland.
| | - Rafael Polania
- Decision Neuroscience Lab, ETH Zürich, Department of Health Sciences and Technology, Winterthurstrasse 190, 8057 Zürich, Switzerland
| | - Denis Burdakov
- Neurobehavioural Dynamics Lab, ETH Zürich, Department of Health Sciences and Technology, Schorenstrasse 16, 8603 Schwerzenbach, Switzerland.
| |
Collapse
|
6
|
Shen B, Wilson J, Nguyen D, Glimcher PW, Louie K. Origins of noise in both improving and degrading decision making. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.26.586597. [PMID: 38915616 PMCID: PMC11195060 DOI: 10.1101/2024.03.26.586597] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/26/2024]
Abstract
Noise is a fundamental problem for information processing in neural systems. In decision-making, noise is assumed to have a primary role in errors and stochastic choice behavior. However, little is known about how noise arising from different sources contributes to value coding and choice behaviors, especially when it interacts with neural computation. Here we examine how noise arising early versus late in the choice process differentially impacts context-dependent choice behavior. We found in model simulations that early and late noise predict opposing context effects: under early noise, contextual information enhances choice accuracy; while under late noise, context degrades choice accuracy. Furthermore, we verified these opposing predictions in experimental human choice behavior. Manipulating early and late noise - by inducing uncertainty in option values and controlling time pressure - produced dissociable positive and negative context effects. These findings reconcile controversial experimental findings in the literature reporting either context-driven impairments or improvements in choice performance, suggesting a unified mechanism for context-dependent choice. More broadly, these findings highlight how different sources of noise can interact with neural computations to differentially modulate behavior. Significance The current study addresses the role of noise origin in decision-making, reconciling controversies around how decision-making is impacted by context. We demonstrate that different types of noise - either arising early during evaluation or late during option comparison - leads to distinct results: with early noise, context enhances choice accuracy, while with late noise, context impairs it. Understanding these dynamics offers potential strategies for improving decision-making in noisy environments and refining existing neural computation models. Overall, our findings advance our understanding of how neural systems handle noise in essential cognitive tasks, suggest a beneficial role for contextual modulation under certain conditions, and highlight the profound implications of noise structure in decision-making.
Collapse
|
7
|
Kobayashi K, Kable JW. Neural mechanisms of information seeking. Neuron 2024; 112:1741-1756. [PMID: 38703774 DOI: 10.1016/j.neuron.2024.04.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2023] [Revised: 01/30/2024] [Accepted: 04/08/2024] [Indexed: 05/06/2024]
Abstract
We ubiquitously seek information to make better decisions. Particularly in the modern age, when more information is available at our fingertips than ever, the information we choose to collect determines the quality of our decisions. Decision neuroscience has long adopted empirical approaches where the information available to decision-makers is fully controlled by the researchers, leaving neural mechanisms of information seeking less understood. Although information seeking has long been studied in the context of the exploration-exploitation trade-off, recent studies have widened the scope to investigate more overt information seeking in a way distinct from other decision processes. Insights gained from these studies, accumulated over the last few years, raise the possibility that information seeking is driven by the reward system signaling the subjective value of information. In this piece, we review findings from the recent studies, highlighting the conceptual and empirical relationships between distinct literatures, and discuss future research directions necessary to establish a more comprehensive understanding of how individuals seek information as a part of value-based decision-making.
Collapse
Affiliation(s)
- Kenji Kobayashi
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, USA.
| | - Joseph W Kable
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, USA.
| |
Collapse
|
8
|
Gilmour W, Mackenzie G, Feile M, Tayler-Grint L, Suveges S, Macfarlane JA, Macleod AD, Marshall V, Grunwald IQ, Steele JD, Gilbertson T. Impaired value-based decision-making in Parkinson's disease apathy. Brain 2024; 147:1362-1376. [PMID: 38305691 PMCID: PMC10994558 DOI: 10.1093/brain/awae025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2023] [Revised: 12/07/2023] [Accepted: 01/13/2024] [Indexed: 02/03/2024] Open
Abstract
Apathy is a common and disabling complication of Parkinson's disease characterized by reduced goal-directed behaviour. Several studies have reported dysfunction within prefrontal cortical regions and projections from brainstem nuclei whose neuromodulators include dopamine, serotonin and noradrenaline. Work in animal and human neuroscience have confirmed contributions of these neuromodulators on aspects of motivated decision-making. Specifically, these neuromodulators have overlapping contributions to encoding the value of decisions, and influence whether to explore alternative courses of action or persist in an existing strategy to achieve a rewarding goal. Building upon this work, we hypothesized that apathy in Parkinson's disease should be associated with an impairment in value-based learning. Using a four-armed restless bandit reinforcement learning task, we studied decision-making in 75 volunteers; 53 patients with Parkinson's disease, with and without clinical apathy, and 22 age-matched healthy control subjects. Patients with apathy exhibited impaired ability to choose the highest value bandit. Task performance predicted an individual patient's apathy severity measured using the Lille Apathy Rating Scale (R = -0.46, P < 0.001). Computational modelling of the patient's choices confirmed the apathy group made decisions that were indifferent to the learnt value of the options, consistent with previous reports of reward insensitivity. Further analysis demonstrated a shift away from exploiting the highest value option and a reduction in perseveration, which also correlated with apathy scores (R = -0.5, P < 0.001). We went on to acquire functional MRI in 59 volunteers; a group of 19 patients with and 20 without apathy and 20 age-matched controls performing the Restless Bandit Task. Analysis of the functional MRI signal at the point of reward feedback confirmed diminished signal within ventromedial prefrontal cortex in Parkinson's disease, which was more marked in apathy, but not predictive of their individual apathy severity. Using a model-based categorization of choice type, decisions to explore lower value bandits in the apathy group activated prefrontal cortex to a similar degree to the age-matched controls. In contrast, Parkinson's patients without apathy demonstrated significantly increased activation across a distributed thalamo-cortical network. Enhanced activity in the thalamus predicted individual apathy severity across both patient groups and exhibited functional connectivity with dorsal anterior cingulate cortex and anterior insula. Given that task performance in patients without apathy was no different to the age-matched control subjects, we interpret the recruitment of this network as a possible compensatory mechanism, which compensates against symptomatic manifestation of apathy in Parkinson's disease.
Collapse
Affiliation(s)
- William Gilmour
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
- Department of Neurology, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
| | - Graeme Mackenzie
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
- Department of Neurology, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
| | - Mathias Feile
- Rehabilitation Psychiatry, Murray Royal Hospital, Perth PH2 7BH, UK
| | | | - Szabolcs Suveges
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
| | - Jennifer A Macfarlane
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
- Medical Physics, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
- SINAPSE, University of Glasgow, Imaging Centre of Excellence, Level 2, Queen Elizabeth University Hospital, Glasgow G51 4TF, Scotland, UK
| | - Angus D Macleod
- Institute of Applied Health Sciences, School of Medicine, University of Aberdeen, Foresterhill, Aberdeen AB24 2ZD, UK
- Department of Neurology, Aberdeen Royal Infirmary, Foresterhill, Aberdeen AB24 2ZD, UK
| | - Vicky Marshall
- Institute of Neurological Sciences, Queen Elizabeth University Hospital, Glasgow G51 4TF, UK
| | - Iris Q Grunwald
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
| | - J Douglas Steele
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
| | - Tom Gilbertson
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
- Department of Neurology, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
| |
Collapse
|
9
|
Claydon J, James WRG, Clarke ADF, Hunt AR. The role of framing, agency and uncertainty in a focus-divide dilemma. Mem Cognit 2024; 52:574-594. [PMID: 37922110 PMCID: PMC11021327 DOI: 10.3758/s13421-023-01484-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/13/2023] [Indexed: 11/05/2023]
Abstract
How to prioritise multiple objectives is a common dilemma of daily life. A simple and effective decision rule is to focus resources when the tasks are difficult, and divide when tasks are easy. Nonetheless, in experimental paradigms of this dilemma, participants make highly variable and suboptimal strategic decisions when asked to allocate resources to two competing goals that vary in difficulty. We developed a new version in which participants had to choose where to park a fire truck between houses of varying distances apart. Unlike in the previous versions of the dilemma, participants approached the optimal strategy in this task. Three key differences between the fire truck version and previous versions of the task were investigated: (1) Framing (whether the objectives are familiar or abstract), by comparing a group who placed cartoon trucks between houses to a group performing the same task with abstract shapes; (2) Agency (how much of the task is under the participants' direct control), by comparing groups who controlled the movement of the truck to those who did not; (3) Uncertainty, by adding variability to the driving speed of the truck to make success or failure on a given trial more difficult to predict. Framing and agency did not influence strategic decisions. When adding variability to outcomes, however, decisions shifted away from optimal. The results suggest choices become more variable when the outcome is less certain, consistent with exploration of response alternatives triggered by an inability to predict success.
Collapse
Affiliation(s)
- Justin Claydon
- School of Psychology, University of Aberdeen, Aberdeen, AB24 3UB, UK.
| | - Warren R G James
- School of Medical Sciences, University of Aberdeen, Aberdeen, UK
| | | | - Amelia R Hunt
- School of Psychology, University of Aberdeen, Aberdeen, AB24 3UB, UK
| |
Collapse
|
10
|
Gregorová K, Eldar E, Deserno L, Reiter AMF. A cognitive-computational account of mood swings in adolescence. Trends Cogn Sci 2024; 28:290-303. [PMID: 38503636 DOI: 10.1016/j.tics.2024.02.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Revised: 02/06/2024] [Accepted: 02/12/2024] [Indexed: 03/21/2024]
Abstract
Teenagers have a reputation for being fickle, in both their choices and their moods. This variability may help adolescents as they begin to independently navigate novel environments. Recently, however, adolescent moodiness has also been linked to psychopathology. Here, we consider adolescents' mood swings from a novel computational perspective, grounded in reinforcement learning (RL). This model proposes that mood is determined by surprises about outcomes in the environment, and how much we learn from these surprises. It additionally suggests that mood biases learning and choice in a bidirectional manner. Integrating independent lines of research, we sketch a cognitive-computational account of how adolescents' mood, learning, and choice dynamics influence each other, with implications for normative and psychopathological development.
Collapse
Affiliation(s)
- Klára Gregorová
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital, Würzburg 97080, Germany; Department of Psychology, Julius-Maximilians-Universität, Würzburg 97070, Germany; German Center of Prevention Research on Mental Health, Würzburg 97080, Germany
| | - Eran Eldar
- Department of Psychology, Hebrew University of Jerusalem, Jerusalem 9190501, Israel; Department of Cognitive & Brain Sciences, Hebrew University of Jerusalem, Jerusalem 9190501, Israel
| | - Lorenz Deserno
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital, Würzburg 97080, Germany; Department of Psychology, Julius-Maximilians-Universität, Würzburg 97070, Germany; Department of Cognitive & Brain Sciences, Hebrew University of Jerusalem, Jerusalem 9190501, Israel; Department of Psychiatry and Psychotherapy, Technical University of Dresden, Dresden 01069, Germany
| | - Andrea M F Reiter
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital, Würzburg 97080, Germany; Department of Psychology, Julius-Maximilians-Universität, Würzburg 97070, Germany; German Center of Prevention Research on Mental Health, Würzburg 97080, Germany; Collaborative Research Centre 940 Volition and Cognitive Control, Technical University of Dresden, Dresden 01069, Germany.
| |
Collapse
|
11
|
Paunov A, L'Hôtellier M, Guo D, He Z, Yu A, Meyniel F. Multiple and subject-specific roles of uncertainty in reward-guided decision-making. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.27.587016. [PMID: 38585958 PMCID: PMC10996615 DOI: 10.1101/2024.03.27.587016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/09/2024]
Abstract
Decision-making in noisy, changing, and partially observable environments entails a basic tradeoff between immediate reward and longer-term information gain, known as the exploration-exploitation dilemma. Computationally, an effective way to balance this tradeoff is by leveraging uncertainty to guide exploration. Yet, in humans, empirical findings are mixed, from suggesting uncertainty-seeking to indifference and avoidance. In a novel bandit task that better captures uncertainty-driven behavior, we find multiple roles for uncertainty in human choices. First, stable and psychologically meaningful individual differences in uncertainty preferences actually range from seeking to avoidance, which can manifest as null group-level effects. Second, uncertainty modulates the use of basic decision heuristics that imperfectly exploit immediate rewards: a repetition bias and win-stay-lose-shift heuristic. These heuristics interact with uncertainty, favoring heuristic choices under higher uncertainty. These results, highlighting the rich and varied structure of reward-based choice, are a step to understanding its functional basis and dysfunction in psychopathology.
Collapse
Affiliation(s)
- Alexander Paunov
- INSERM-CEA Cognitive Neuroimaging Unit (UNICOG), NeuroSpin Center, CEA Paris-Saclay, Gif-sur-Yvette, France Université de Paris, Paris, France
- Institut de Neuromodulation, GHU Paris, Psychiatrie et Neurosciences, Centre Hospitalier Sainte-Anne, Pôle Hospitalo-universitaire 15, Université Paris Cité, Paris, France
| | - Maëva L'Hôtellier
- INSERM-CEA Cognitive Neuroimaging Unit (UNICOG), NeuroSpin Center, CEA Paris-Saclay, Gif-sur-Yvette, France Université de Paris, Paris, France
| | - Dalin Guo
- Department of Cognitive Science, University of California San Diego, San Diego, CA, USA
| | - Zoe He
- Department of Cognitive Science, University of California San Diego, San Diego, CA, USA
| | - Angela Yu
- Department of Cognitive Science, University of California San Diego, San Diego, CA, USA
- Centre for Cognitive Science & Hessian AI Center, Technical University of Darmstadt, Germany
| | - Florent Meyniel
- INSERM-CEA Cognitive Neuroimaging Unit (UNICOG), NeuroSpin Center, CEA Paris-Saclay, Gif-sur-Yvette, France Université de Paris, Paris, France
- Institut de Neuromodulation, GHU Paris, Psychiatrie et Neurosciences, Centre Hospitalier Sainte-Anne, Pôle Hospitalo-universitaire 15, Université Paris Cité, Paris, France
| |
Collapse
|
12
|
Aberg KC, Paz R. The neurobehavioral correlates of exploration without learning: Trading off value for explicit, prospective, and variable information gains. Cell Rep 2024; 43:113880. [PMID: 38416639 DOI: 10.1016/j.celrep.2024.113880] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Revised: 01/10/2024] [Accepted: 02/13/2024] [Indexed: 03/01/2024] Open
Abstract
Exploration is typically motivated by gaining information, with previous research showing that potential information gains drive a "directed" type of exploration. Yet, this research usually studies exploration in the context of learning paradigms and does not directly manipulate multiple levels of information gain. Here, we present a task that isolates learning from decision-making and controls the magnitude of prospective information gains. As predicted, participants explore more with larger future information gains. Both value gains and information gains, at a trial-by-trial level, engage the ventromedial prefrontal cortex (vmPFC), the ventral striatum (VStr), the amygdala, the dorsal anterior cingulate cortex (dACC), and the anterior insula (aINS). Moreover, individual sensitivities to value gains and information gains modulate the vmPFC, dACC, and aINS, but the amygdala and VStr are modulated only by individual sensitivities to information gains. Overall, we identify the neural circuitry of information-based exploration and its relationship with inter-individual exploration biases.
Collapse
Affiliation(s)
- Kristoffer C Aberg
- Department of Brain Sciences, Weizmann Institute of Science, Rehovot 76100, Israel.
| | - Rony Paz
- Department of Brain Sciences, Weizmann Institute of Science, Rehovot 76100, Israel
| |
Collapse
|
13
|
Prat-Carrabin A, Meyniel F, Azeredo da Silveira R. Resource-rational account of sequential effects in human prediction. eLife 2024; 13:e81256. [PMID: 38224341 PMCID: PMC10789490 DOI: 10.7554/elife.81256] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Accepted: 12/11/2023] [Indexed: 01/16/2024] Open
Abstract
An abundant literature reports on 'sequential effects' observed when humans make predictions on the basis of stochastic sequences of stimuli. Such sequential effects represent departures from an optimal, Bayesian process. A prominent explanation posits that humans are adapted to changing environments, and erroneously assume non-stationarity of the environment, even if the latter is static. As a result, their predictions fluctuate over time. We propose a different explanation in which sub-optimal and fluctuating predictions result from cognitive constraints (or costs), under which humans however behave rationally. We devise a framework of costly inference, in which we develop two classes of models that differ by the nature of the constraints at play: in one case the precision of beliefs comes at a cost, resulting in an exponential forgetting of past observations, while in the other beliefs with high predictive power are favored. To compare model predictions to human behavior, we carry out a prediction task that uses binary random stimuli, with probabilities ranging from 0.05 to 0.95. Although in this task the environment is static and the Bayesian belief converges, subjects' predictions fluctuate and are biased toward the recent stimulus history. Both classes of models capture this 'attractive effect', but they depart in their characterization of higher-order effects. Only the precision-cost model reproduces a 'repulsive effect', observed in the data, in which predictions are biased away from stimuli presented in more distant trials. Our experimental results reveal systematic modulations in sequential effects, which our theoretical approach accounts for in terms of rationality under cognitive constraints.
Collapse
Affiliation(s)
- Arthur Prat-Carrabin
- Department of Economics, Columbia UniversityNew YorkUnited States
- Laboratoire de Physique de l’École Normale Supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université de ParisParisFrance
| | - Florent Meyniel
- Cognitive Neuroimaging Unit, Institut National de la Santé et de la Recherche Médicale, Commissariat à l’Energie Atomique et aux Energies Alternatives, Centre National de la Recherche Scientifique, Université Paris-Saclay, NeuroSpin centerGif-sur-YvetteFrance
- Institut de neuromodulation, GHU Paris, Psychiatrie et Neurosciences, Centre Hospitalier Sainte-Anne, Pôle Hospitalo-Universitaire 15, Université Paris CitéParisFrance
| | - Rava Azeredo da Silveira
- Laboratoire de Physique de l’École Normale Supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université de ParisParisFrance
- Institute of Molecular and Clinical Ophthalmology BaselBaselSwitzerland
- Faculty of Science, University of BaselBaselSwitzerland
| |
Collapse
|
14
|
Algermissen J, Swart JC, Scheeringa R, Cools R, den Ouden HEM. Prefrontal signals precede striatal signals for biased credit assignment in motivational learning biases. Nat Commun 2024; 15:19. [PMID: 38168089 PMCID: PMC10762147 DOI: 10.1038/s41467-023-44632-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2021] [Accepted: 12/22/2023] [Indexed: 01/05/2024] Open
Abstract
Actions are biased by the outcomes they can produce: Humans are more likely to show action under reward prospect, but hold back under punishment prospect. Such motivational biases derive not only from biased response selection, but also from biased learning: humans tend to attribute rewards to their own actions, but are reluctant to attribute punishments to having held back. The neural origin of these biases is unclear. Specifically, it remains open whether motivational biases arise primarily from the architecture of subcortical regions or also reflect cortical influences, the latter being typically associated with increased behavioral flexibility and control beyond stereotyped behaviors. Simultaneous EEG-fMRI allowed us to track which regions encoded biased prediction errors in which order. Biased prediction errors occurred in cortical regions (dorsal anterior and posterior cingulate cortices) before subcortical regions (striatum). These results highlight that biased learning is not a mere feature of the basal ganglia, but arises through prefrontal cortical contributions, revealing motivational biases to be a potentially flexible, sophisticated mechanism.
Collapse
Affiliation(s)
- Johannes Algermissen
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.
| | - Jennifer C Swart
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - René Scheeringa
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
- Erwin L. Hahn Institute for Magnetic Resonance Imaging, University of Duisburg-Essen, Essen, Germany
| | - Roshan Cools
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
- Department of Psychiatry, Radboud University Medical Centre, Nijmegen, The Netherlands
| | - Hanneke E M den Ouden
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.
| |
Collapse
|
15
|
Mikus N, Eisenegger C, Mathys C, Clark L, Müller U, Robbins TW, Lamm C, Naef M. Blocking D2/D3 dopamine receptors in male participants increases volatility of beliefs when learning to trust others. Nat Commun 2023; 14:4049. [PMID: 37422466 PMCID: PMC10329681 DOI: 10.1038/s41467-023-39823-5] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2022] [Accepted: 06/29/2023] [Indexed: 07/10/2023] Open
Abstract
The ability to learn about other people is crucial for human social functioning. Dopamine has been proposed to regulate the precision of beliefs, but direct behavioural evidence of this is lacking. In this study, we investigate how a high dose of the D2/D3 dopamine receptor antagonist sulpiride impacts learning about other people's prosocial attitudes in a repeated Trust game. Using a Bayesian model of belief updating, we show that in a sample of 76 male participants sulpiride increases the volatility of beliefs, which leads to higher precision weights on prediction errors. This effect is driven by participants with genetically conferred higher dopamine availability (Taq1a polymorphism) and remains even after controlling for working memory performance. Higher precision weights are reflected in higher reciprocal behaviour in the repeated Trust game but not in single-round Trust games. Our data provide evidence that the D2 receptors are pivotal in regulating prediction error-driven belief updating in a social context.
Collapse
Affiliation(s)
- Nace Mikus
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria.
- Interacting Minds Centre, Aarhus University, Aarhus, Denmark.
| | - Christoph Eisenegger
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
- Behavioural and Clinical Neuroscience Institute and Department of Psychology, University of Cambridge, Cambridge, UK
| | - Christoph Mathys
- Interacting Minds Centre, Aarhus University, Aarhus, Denmark
- Translational Neuromodeling Unit (TNU), Institute for Biomedical Engineering, University of Zurich and ETH Zurich, Zurich, Switzerland
- Scuola Internazionale Superiore di Studi Avanzati (SISSA), Trieste, Italy
| | - Luke Clark
- Centre for Gambling Research at UBC, Department of Psychology, University of British, Columbia, Vancouver, BC, Canada
- Djavad Mowafaghian Centre for Brain Health, University of British Columbia, Vancouver, BC, Canada
| | - Ulrich Müller
- Behavioural and Clinical Neuroscience Institute and Department of Psychology, University of Cambridge, Cambridge, UK
- Adult Neurodevelopmental Services, Health & Community Services, Government of Jersey, St Helier, Jersey
| | - Trevor W Robbins
- Behavioural and Clinical Neuroscience Institute and Department of Psychology, University of Cambridge, Cambridge, UK
| | - Claus Lamm
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria.
| | - Michael Naef
- Department of Economics, University of Durham, Durham, UK.
| |
Collapse
|
16
|
Chen CS, Mueller D, Knep E, Ebitz RB, Grissom NM. Dopamine and norepinephrine differentially mediate the exploration-exploitation tradeoff. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.01.09.523322. [PMID: 36711959 PMCID: PMC9881999 DOI: 10.1101/2023.01.09.523322] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/11/2023]
Abstract
The catecholamines dopamine (DA) and norepinephrine (NE) have been repeatedly implicated in neuropsychiatric vulnerability, in part via their roles in mediating the decision making processes. Although the two neuromodulators share a synthesis pathway and are co-activated under states of arousal, they engage in distinct circuits and roles in modulating neural activity across the brain. However, in the computational neuroscience literature, they have been assigned similar roles in modulating the latent cognitive processes of decision making, in particular the exploration-exploitation tradeoff. Revealing how each neuromodulator contributes to this explore-exploit process will be important in guiding mechanistic hypotheses emerging from computational psychiatric approaches. To understand the differences and overlaps of the roles of these two catecholamine systems in regulating exploration and exploitation, a direct comparison using the same dynamic decision making task is needed. Here, we ran mice in a restless two-armed bandit task, which encourages both exploration and exploitation. We systemically administered a nonselective DA receptor antagonist (flupenthixol), a nonselective DA receptor agonist (apomorphine), a NE beta-receptor antagonist (propranolol), and a NE beta-receptor agonist (isoproterenol), and examined changes in exploration within subjects across sessions. We found a bidirectional modulatory effect of dopamine receptor activity on the level of exploration. Increasing dopamine activity decreased exploration and decreasing dopamine activity increased exploration. Beta-noradrenergic receptor activity also modulated exploration, but the modulatory effect was mediated by sex. Reinforcement learning model parameters suggested that dopamine modulation affected exploration via decision noise and norepinephrine modulation affected exploration via outcome sensitivity. Together, these findings suggested that the mechanisms that govern the transition between exploration and exploitation are sensitive to changes in both catecholamine functions and revealed differential roles for NE and DA in mediating exploration.
Collapse
|
17
|
Topel S, Ma I, Sleutels J, van Steenbergen H, de Bruijn ERA, van Duijvenvoorde ACK. Expecting the unexpected: a review of learning under uncertainty across development. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2023:10.3758/s13415-023-01098-0. [PMID: 37237092 PMCID: PMC10390612 DOI: 10.3758/s13415-023-01098-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 03/28/2023] [Indexed: 05/28/2023]
Abstract
Many of our decisions take place under uncertainty. To successfully navigate the environment, individuals need to estimate the degree of uncertainty and adapt their behaviors accordingly by learning from experiences. However, uncertainty is a broad construct and distinct types of uncertainty may differentially influence our learning. We provide a semi-systematic review to illustrate cognitive and neurobiological processes involved in learning under two types of uncertainty: learning in environments with stochastic outcomes, and with volatile outcomes. We specifically reviewed studies (N = 26 studies) that included an adolescent population, because adolescence is a period in life characterized by heightened exploration and learning, as well as heightened uncertainty due to experiencing many new, often social, environments. Until now, reviews have not comprehensively compared learning under distinct types of uncertainties in this age range. Our main findings show that although the overall developmental patterns were mixed, most studies indicate that learning from stochastic outcomes, as indicated by increased accuracy in performance, improved with age. We also found that adolescents tended to have an advantage compared with adults and children when learning from volatile outcomes. We discuss potential mechanisms explaining these age-related differences and conclude by outlining future research directions.
Collapse
Affiliation(s)
- Selin Topel
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands.
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands.
| | - Ili Ma
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Jan Sleutels
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden University, Institute for Philosophy, Leiden, The Netherlands
| | - Henk van Steenbergen
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Ellen R A de Bruijn
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Anna C K van Duijvenvoorde
- Leiden University, Institute of Psychology, Wassenaarseweg 52, 2333, AK, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| |
Collapse
|
18
|
Gibbs-Dean T, Katthagen T, Tsenkova I, Ali R, Liang X, Spencer T, Diederen K. Belief updating in psychosis, depression and anxiety disorders: A systematic review across computational modelling approaches. Neurosci Biobehav Rev 2023; 147:105087. [PMID: 36791933 DOI: 10.1016/j.neubiorev.2023.105087] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2022] [Revised: 01/31/2023] [Accepted: 02/09/2023] [Indexed: 02/15/2023]
Abstract
Alterations in belief updating are proposed to underpin symptoms of psychiatric illness, including psychosis, depression, and anxiety. Key parameters underlying belief updating can be captured using computational modelling techniques, aiding the identification of unique and shared deficits, and improving diagnosis and treatment. We systematically reviewed research that applied computational modelling to probabilistic tasks measuring belief updating in stable and volatile (changing) environments, across clinical and subclinical psychosis (n = 17), anxiety (n = 9), depression (n = 9) and transdiagnostic samples (n = 9). Depression disorders related to abnormal belief updating in response to the valence of rewards, evidenced in both stable and volatile environments. Whereas psychosis and anxiety disorders were associated with difficulties adapting to changing contingencies specifically, indicating an inflexibility and/or insensitivity to environmental volatility. Higher-order learning models revealed additional difficulties in the estimation of overall environmental volatility across psychosis disorders, showing increased updating to irrelevant information. These findings stress the importance of investigating belief updating in transdiagnostic samples, using homogeneous experimental and computational modelling approaches.
Collapse
Affiliation(s)
- Toni Gibbs-Dean
- Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King's College London, UK.
| | - Teresa Katthagen
- Department of Psychiatry and Neuroscience CCM, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Germany
| | - Iveta Tsenkova
- Psychological Medicine, Institute of Psychiatry, Psychology and neuroscience, King's College London, UK
| | - Rubbia Ali
- Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King's College London, UK
| | - Xinyi Liang
- Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King's College London, UK
| | - Thomas Spencer
- Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King's College London, UK
| | - Kelly Diederen
- Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King's College London, UK
| |
Collapse
|
19
|
Lee JK, Rouault M, Wyart V. Adaptive tuning of human learning and choice variability to unexpected uncertainty. SCIENCE ADVANCES 2023; 9:eadd0501. [PMID: 36989365 PMCID: PMC10058239 DOI: 10.1126/sciadv.add0501] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/17/2022] [Accepted: 02/28/2023] [Indexed: 06/19/2023]
Abstract
Human value-based decisions are notably variable under uncertainty. This variability is known to arise from two distinct sources: variable choices aimed at exploring available options and imprecise learning of option values due to limited cognitive resources. However, whether these two sources of decision variability are tuned to their specific costs and benefits remains unclear. To address this question, we compared the effects of expected and unexpected uncertainty on decision-making in the same reinforcement learning task. Across two large behavioral datasets, we found that humans choose more variably between options but simultaneously learn less imprecisely their values in response to unexpected uncertainty. Using simulations of learning agents, we demonstrate that these opposite adjustments reflect adaptive tuning of exploration and learning precision to the structure of uncertainty. Together, these findings indicate that humans regulate not only how much they explore uncertain options but also how precisely they learn the values of these options.
Collapse
Affiliation(s)
- Junseok K. Lee
- Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France
- Département d’Études Cognitives, École Normale Supérieure, Université PSL, Paris, France
| | - Marion Rouault
- Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France
- Département d’Études Cognitives, École Normale Supérieure, Université PSL, Paris, France
| | - Valentin Wyart
- Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France
- Département d’Études Cognitives, École Normale Supérieure, Université PSL, Paris, France
- Institut du Psychotraumatisme de l’Enfant et de l’Adolescent, Conseil Départemental Yvelines et Hauts-de-Seine, Versailles, France
| |
Collapse
|
20
|
Nissan N, Hertz U, Shahar N, Gabay Y. Distinct reinforcement learning profiles distinguish between language and attentional neurodevelopmental disorders. BEHAVIORAL AND BRAIN FUNCTIONS : BBF 2023; 19:6. [PMID: 36941632 PMCID: PMC10029183 DOI: 10.1186/s12993-023-00207-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Subscribe] [Scholar Register] [Received: 07/31/2022] [Accepted: 01/26/2023] [Indexed: 03/23/2023]
Abstract
BACKGROUND Theoretical models posit abnormalities in cortico-striatal pathways in two of the most common neurodevelopmental disorders (Developmental dyslexia, DD, and Attention deficit hyperactive disorder, ADHD), but it is still unclear what distinct cortico-striatal dysfunction might distinguish language disorders from others that exhibit very different symptomatology. Although impairments in tasks that depend on the cortico-striatal network, including reinforcement learning (RL), have been implicated in both disorders, there has been little attempt to dissociate between different types of RL or to compare learning processes in these two types of disorders. The present study builds upon prior research indicating the existence of two learning manifestations of RL and evaluates whether these processes can be differentiated in language and attention deficit disorders. We used a two-step RL task shown to dissociate model-based from model-free learning in human learners. RESULTS Our results show that, relative to neurotypicals, DD individuals showed an impairment in model-free but not in model-based learning, whereas in ADHD the ability to use both model-free and model-based learning strategies was significantly compromised. CONCLUSIONS Thus, learning impairments in DD may be linked to a selective deficit in the ability to form action-outcome associations based on previous history, whereas in ADHD some learning deficits may be related to an incapacity to pursue rewards based on the tasks' structure. Our results indicate how different patterns of learning deficits may underlie different disorders, and how computation-minded experimental approaches can differentiate between them.
Collapse
Affiliation(s)
- Noyli Nissan
- Department of Special Education, University of Haifa, Haifa, Israel
- Edmond J. Safra Brain Research Center for the Study of Learning Disabilities, University of Haifa, 199 Abba Khoushy Ave, Haifa, Israel
| | - Uri Hertz
- Department of Cognitive Sciences, University of Haifa, Haifa, Israel
| | - Nitzan Shahar
- The School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
| | - Yafit Gabay
- Department of Special Education, University of Haifa, Haifa, Israel.
- Edmond J. Safra Brain Research Center for the Study of Learning Disabilities, University of Haifa, 199 Abba Khoushy Ave, Haifa, Israel.
| |
Collapse
|
21
|
Wieland L, Ebrahimi C, Katthagen T, Panitz M, Luettgau L, Heinz A, Schlagenhauf F, Sjoerds Z. Acute stress alters probabilistic reversal learning in healthy male adults. Eur J Neurosci 2023; 57:824-839. [PMID: 36656136 DOI: 10.1111/ejn.15916] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2022] [Revised: 01/12/2023] [Accepted: 01/14/2023] [Indexed: 01/20/2023]
Abstract
Behavioural adaptation is a fundamental cognitive ability, ensuring survival by allowing for flexible adjustment to changing environments. In laboratory settings, behavioural adaptation can be measured with reversal learning paradigms requiring agents to adjust reward learning to stimulus-action-outcome contingency changes. Stress is found to alter flexibility of reward learning, but effect directionality is mixed across studies. Here, we used model-based functional MRI (fMRI) in a within-subjects design to investigate the effect of acute psychosocial stress on flexible behavioural adaptation. Healthy male volunteers (n = 28) did a reversal learning task during fMRI in two sessions, once after the Trier Social Stress Test (TSST), a validated psychosocial stress induction method, and once after a control condition. Stress effects on choice behaviour were investigated using multilevel generalized linear models and computational models describing different learning processes that potentially generated the data. Computational models were fitted using a hierarchical Bayesian approach, and model-derived reward prediction errors (RPE) were used as fMRI regressors. We found that acute psychosocial stress slightly increased correct response rates. Model comparison revealed that double-update learning with altered choice temperature under stress best explained the observed behaviour. In the brain, model-derived RPEs were correlated with BOLD signals in striatum and ventromedial prefrontal cortex (vmPFC). Striatal RPE signals for win trials were stronger during stress compared with the control condition. Our study suggests that acute psychosocial stress could enhance reversal learning and RPE brain responses in healthy male participants and provides a starting point to explore these effects further in a more diverse population.
Collapse
Affiliation(s)
- Lara Wieland
- Department of Psychiatry and Neurosciences, CCM, Charité-Universitätsmedizin Berlin, Berlin, Germany.,Einstein Center for Neurosciences Berlin, Charité-Universitätsmedizin Berlin, Berlin, Germany.,Bernstein Center for Computational Neuroscience, Berlin, Germany
| | - Claudia Ebrahimi
- Department of Psychiatry and Neurosciences, CCM, Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Teresa Katthagen
- Department of Psychiatry and Neurosciences, CCM, Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Martin Panitz
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Lennart Luettgau
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,Max Planck University College London Centre for Computational Psychiatry and Ageing Research, London, UK
| | - Andreas Heinz
- Department of Psychiatry and Neurosciences, CCM, Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Florian Schlagenhauf
- Department of Psychiatry and Neurosciences, CCM, Charité-Universitätsmedizin Berlin, Berlin, Germany.,Einstein Center for Neurosciences Berlin, Charité-Universitätsmedizin Berlin, Berlin, Germany.,Bernstein Center for Computational Neuroscience, Berlin, Germany.,Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Zsuzsika Sjoerds
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,Cognitive Psychology Unit, Institute of Psychology & Leiden Institute for Brain and Cognition, Leiden University, Leiden, the Netherlands
| |
Collapse
|
22
|
de A Marcelino AL, Gray O, Al-Fatly B, Gilmour W, Douglas Steele J, Kühn AA, Gilbertson T. Pallidal neuromodulation of the explore/exploit trade-off in decision-making. eLife 2023; 12:79642. [PMID: 36727860 PMCID: PMC9940911 DOI: 10.7554/elife.79642] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2022] [Accepted: 02/01/2023] [Indexed: 02/03/2023] Open
Abstract
Every decision that we make involves a conflict between exploiting our current knowledge of an action's value or exploring alternative courses of action that might lead to a better, or worse outcome. The sub-cortical nuclei that make up the basal ganglia have been proposed as a neural circuit that may contribute to resolving this explore-exploit 'dilemma'. To test this hypothesis, we examined the effects of neuromodulating the basal ganglia's output nucleus, the globus pallidus interna, in patients who had undergone deep brain stimulation (DBS) for isolated dystonia. Neuromodulation enhanced the number of exploratory choices to the lower value option in a two-armed bandit probabilistic reversal-learning task. Enhanced exploration was explained by a reduction in the rate of evidence accumulation (drift rate) in a reinforcement learning drift diffusion model. We estimated the functional connectivity profile between the stimulating DBS electrode and the rest of the brain using a normative functional connectome derived from heathy controls. Variation in the extent of neuromodulation induced exploration between patients was associated with functional connectivity from the stimulation electrode site to a distributed brain functional network. We conclude that the basal ganglia's output nucleus, the globus pallidus interna, can adaptively modify decision choice when faced with the dilemma to explore or exploit.
Collapse
Affiliation(s)
- Ana Luisa de A Marcelino
- Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Movement Disorder and Neuromodulation Unit, Department of Neurology, Charité Campus MitteBerlinGermany
- Berlin Institute of Health at Charité – Universitätsmedizin Berlin, Core Facility GenomicsBerlinGermany
| | - Owen Gray
- Division of Imaging Science and Technology, Medical School, University of DundeeDundeeUnited Kingdom
| | - Bassam Al-Fatly
- Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Movement Disorder and Neuromodulation Unit, Department of Neurology, Charité Campus MitteBerlinGermany
| | - William Gilmour
- Division of Imaging Science and Technology, Medical School, University of DundeeDundeeUnited Kingdom
| | - J Douglas Steele
- Division of Imaging Science and Technology, Medical School, University of DundeeDundeeUnited Kingdom
| | - Andrea A Kühn
- Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Movement Disorder and Neuromodulation Unit, Department of Neurology, Charité Campus MitteBerlinGermany
- Berlin Institute of Health at Charité – Universitätsmedizin Berlin, Core Facility GenomicsBerlinGermany
- Berlin School of Mind and Brain, Charité - University Medicine BerlinBerlinGermany
- NeuroCure, Charité - University Medicine BerlinBerlinGermany
- DZNE, German Centre for Degenerative DiseasesBerlinGermany
| | - Tom Gilbertson
- Division of Imaging Science and Technology, Medical School, University of DundeeDundeeUnited Kingdom
- Department of Neurology, Ninewells Hospital & Medical SchoolDundeeUnited Kingdom
| |
Collapse
|
23
|
Knowledge generalization and the costs of multitasking. Nat Rev Neurosci 2023; 24:98-112. [PMID: 36347942 DOI: 10.1038/s41583-022-00653-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/12/2022] [Indexed: 11/10/2022]
Abstract
Humans are able to rapidly perform novel tasks, but show pervasive performance costs when attempting to do two things at once. Traditionally, empirical and theoretical investigations into the sources of such multitasking interference have largely focused on multitasking in isolation to other cognitive functions, characterizing the conditions that give rise to performance decrements. Here we instead ask whether multitasking costs are linked to the system's capacity for knowledge generalization, as is required to perform novel tasks. We show how interrogation of the neurophysiological circuitry underlying these two facets of cognition yields further insights for both. Specifically, we demonstrate how a system that rapidly generalizes knowledge may induce multitasking costs owing to sharing of task contingencies between contexts in neural representations encoded in frontoparietal and striatal brain regions. We discuss neurophysiological insights suggesting that prolonged learning segregates such representations by refining the brain's model of task-relevant contingencies, thereby reducing information sharing between contexts and improving multitasking performance while reducing flexibility and generalization. These proposed neural mechanisms explain why the brain shows rapid task understanding, multitasking limitations and practice effects. In short, multitasking limits are the price we pay for behavioural flexibility.
Collapse
|
24
|
Jahn CI, Grohn J, Cuell S, Emberton A, Bouret S, Walton ME, Kolling N, Sallet J. Neural responses in macaque prefrontal cortex are linked to strategic exploration. PLoS Biol 2023; 21:e3001985. [PMID: 36716348 PMCID: PMC9910800 DOI: 10.1371/journal.pbio.3001985] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2022] [Revised: 02/09/2023] [Accepted: 01/03/2023] [Indexed: 02/01/2023] Open
Abstract
Humans have been shown to strategically explore. They can identify situations in which gathering information about distant and uncertain options is beneficial for the future. Because primates rely on scarce resources when they forage, they are also thought to strategically explore, but whether they use the same strategies as humans and the neural bases of strategic exploration in monkeys are largely unknown. We designed a sequential choice task to investigate whether monkeys mobilize strategic exploration based on whether information can improve subsequent choice, but also to ask the novel question about whether monkeys adjust their exploratory choices based on the contingency between choice and information, by sometimes providing the counterfactual feedback about the unchosen option. We show that monkeys decreased their reliance on expected value when exploration could be beneficial, but this was not mediated by changes in the effect of uncertainty on choices. We found strategic exploratory signals in anterior and mid-cingulate cortex (ACC/MCC) and dorsolateral prefrontal cortex (dlPFC). This network was most active when a low value option was chosen, which suggests a role in counteracting expected value signals, when exploration away from value should to be considered. Such strategic exploration was abolished when the counterfactual feedback was available. Learning from counterfactual outcome was associated with the recruitment of a different circuit centered on the medial orbitofrontal cortex (OFC), where we showed that monkeys represent chosen and unchosen reward prediction errors. Overall, our study shows how ACC/MCC-dlPFC and OFC circuits together could support exploitation of available information to the fullest and drive behavior towards finding more information through exploration when it is beneficial.
Collapse
Affiliation(s)
- Caroline I. Jahn
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
- Motivation, Brain and Behavior Team, Institut du Cerveau et de la Moelle Epinière, Paris, France
- Sorbonne Paris Cité universités, Université Paris Descartes, Frontières du Vivant, Paris, France
- * E-mail: (CIJ); (JG); (NK); (JS)
| | - Jan Grohn
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
- * E-mail: (CIJ); (JG); (NK); (JS)
| | - Steven Cuell
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
| | - Andrew Emberton
- Biomedical Science Services, University of Oxford, Oxford, United Kingdom
| | - Sebastien Bouret
- Motivation, Brain and Behavior Team, Institut du Cerveau et de la Moelle Epinière, Paris, France
| | - Mark E. Walton
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
| | - Nils Kolling
- Wellcome Centre for Integrative Neuroimaging, OBHA, University of Oxford, Headington, United Kingdom
- Univ Lyon, Université Lyon 1, Inserm, Stem Cell and Brain Research Institute U1208, Bron, France
- * E-mail: (CIJ); (JG); (NK); (JS)
| | - Jérôme Sallet
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
- Univ Lyon, Université Lyon 1, Inserm, Stem Cell and Brain Research Institute U1208, Bron, France
- * E-mail: (CIJ); (JG); (NK); (JS)
| |
Collapse
|
25
|
Drevet J, Drugowitsch J, Wyart V. Efficient stabilization of imprecise statistical inference through conditional belief updating. Nat Hum Behav 2022; 6:1691-1704. [PMID: 36138224 DOI: 10.1038/s41562-022-01445-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2021] [Accepted: 08/11/2022] [Indexed: 01/14/2023]
Abstract
Statistical inference is the optimal process for forming and maintaining accurate beliefs about uncertain environments. However, human inference comes with costs due to its associated biases and limited precision. Indeed, biased or imprecise inference can trigger variable beliefs and unwarranted changes in behaviour. Here, by studying decisions in a sequential categorization task based on noisy visual stimuli, we obtained converging evidence that humans reduce the variability of their beliefs by updating them only when the reliability of incoming sensory information is judged as sufficiently strong. Instead of integrating the evidence provided by all stimuli, participants actively discarded as much as a third of stimuli. This conditional belief updating strategy shows good test-retest reliability, correlates with perceptual confidence and explains human behaviour better than previously described strategies. This seemingly suboptimal strategy not only reduces the costs of imprecise computations but also, counterintuitively, increases the accuracy of resulting decisions.
Collapse
Affiliation(s)
- Julie Drevet
- Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France.
- Département d'Études Cognitives, École Normale Supérieure, Université PSL, Paris, France.
| | - Jan Drugowitsch
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Valentin Wyart
- Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France.
- Département d'Études Cognitives, École Normale Supérieure, Université PSL, Paris, France.
| |
Collapse
|
26
|
Ashinoff BK, Buck J, Woodford M, Horga G. The effects of base rate neglect on sequential belief updating and real-world beliefs. PLoS Comput Biol 2022; 18:e1010796. [PMID: 36548395 PMCID: PMC9831339 DOI: 10.1371/journal.pcbi.1010796] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2022] [Revised: 01/10/2023] [Accepted: 12/06/2022] [Indexed: 12/24/2022] Open
Abstract
Base-rate neglect is a pervasive bias in judgment that is conceptualized as underweighting of prior information and can have serious consequences in real-world scenarios. This bias is thought to reflect variability in inferential processes but empirical support for a cohesive theory of base-rate neglect with sufficient explanatory power to account for longer-term and real-world beliefs is lacking. A Bayesian formalization of base-rate neglect in the context of sequential belief updating predicts that belief trajectories should exhibit dynamic patterns of dependence on the order in which evidence is presented and its consistency with prior beliefs. To test this, we developed a novel 'urn-and-beads' task that systematically manipulated the order of colored bead sequences and elicited beliefs via an incentive-compatible procedure. Our results in two independent online studies confirmed the predictions of the sequential base-rate neglect model: people exhibited beliefs that are more influenced by recent evidence and by evidence inconsistent with prior beliefs. We further found support for a noisy-sampling inference model whereby base-rate neglect results from rational discounting of noisy internal representations of prior beliefs. Finally, we found that model-derived indices of base-rate neglect-including noisier prior representation-correlated with propensity for unusual beliefs outside the laboratory. Our work supports the relevance of Bayesian accounts of sequential base-rate neglect to real-world beliefs and hints at strategies to minimize deleterious consequences of this pervasive bias.
Collapse
Affiliation(s)
- Brandon K. Ashinoff
- Department of Psychiatry, Columbia University, New York, NY, United States of America
- New York State Psychiatric Institute (NYSPI), New York, NY, United States of America
| | - Justin Buck
- Department of Psychiatry, Columbia University, New York, NY, United States of America
- New York State Psychiatric Institute (NYSPI), New York, NY, United States of America
- Department of Neuroscience, Columbia University, New York, NY, United States of America
| | - Michael Woodford
- Department of Economics, Columbia University, New York, NY, United States of America
| | - Guillermo Horga
- Department of Psychiatry, Columbia University, New York, NY, United States of America
- New York State Psychiatric Institute (NYSPI), New York, NY, United States of America
| |
Collapse
|
27
|
Training diversity promotes absolute-value-guided choice. PLoS Comput Biol 2022; 18:e1010664. [DOI: 10.1371/journal.pcbi.1010664] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2022] [Revised: 11/21/2022] [Accepted: 10/18/2022] [Indexed: 11/06/2022] Open
Abstract
Many decision-making studies have demonstrated that humans learn either expected values or relative preferences among choice options, yet little is known about what environmental conditions promote one strategy over the other. Here, we test the novel hypothesis that humans adapt the degree to which they form absolute values to the diversity of the learning environment. Since absolute values generalize better to new sets of options, we predicted that the more options a person learns about the more likely they would be to form absolute values. To test this, we designed a multi-day learning experiment comprising twenty learning sessions in which subjects chose among pairs of images each associated with a different probability of reward. We assessed the degree to which subjects formed absolute values and relative preferences by asking them to choose between images they learned about in separate sessions. We found that concurrently learning about more images within a session enhanced absolute-value, and suppressed relative-preference, learning. Conversely, cumulatively pitting each image against a larger number of other images across multiple sessions did not impact the form of learning. These results show that the way humans encode preferences is adapted to the diversity of experiences offered by the immediate learning context.
Collapse
|
28
|
Dubois M, Bowler A, Moses-Payne ME, Habicht J, Moran R, Steinbeis N, Hauser TU. Exploration heuristics decrease during youth. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2022; 22:969-983. [PMID: 35589910 PMCID: PMC9458685 DOI: 10.3758/s13415-022-01009-9] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Accepted: 04/22/2022] [Indexed: 01/01/2023]
Abstract
Deciding between exploring new avenues and exploiting known choices is central to learning, and this exploration-exploitation trade-off changes during development. Exploration is not a unitary concept, and humans deploy multiple distinct mechanisms, but little is known about their specific emergence during development. Using a previously validated task in adults, changes in exploration mechanisms were investigated between childhood (8-9 y/o, N = 26; 16 females), early (12-13 y/o, N = 38; 21 females), and late adolescence (16-17 y/o, N = 33; 19 females) in ethnically and socially diverse schools from disadvantaged areas. We find an increased usage of a computationally light exploration heuristic in younger groups, effectively accommodating their limited neurocognitive resources. Moreover, this heuristic was associated with self-reported, attention-deficit/hyperactivity disorder symptoms in this population-based sample. This study enriches our mechanistic understanding about how exploration strategies mature during development.
Collapse
Affiliation(s)
- Magda Dubois
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, WC1B 5EH, London, UK.
- Wellcome Centre for Human Neuroimaging, University College London, WC1N 3BG, London, UK.
| | - Aislinn Bowler
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, WC1B 5EH, London, UK
- Wellcome Centre for Human Neuroimaging, University College London, WC1N 3BG, London, UK
- Centre for Brain and Cognitive Development, Birkbeck, University of London, WC1E 7HX, London, UK
| | - Madeleine E Moses-Payne
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, WC1B 5EH, London, UK
- Wellcome Centre for Human Neuroimaging, University College London, WC1N 3BG, London, UK
- UCL Institute of Cognitive Neuroscience, WC1N 3AZ, London, UK
| | - Johanna Habicht
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, WC1B 5EH, London, UK
- Wellcome Centre for Human Neuroimaging, University College London, WC1N 3BG, London, UK
| | - Rani Moran
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, WC1B 5EH, London, UK
- Wellcome Centre for Human Neuroimaging, University College London, WC1N 3BG, London, UK
| | - Nikolaus Steinbeis
- Division of Psychology and Language Sciences, University College London, WC1H 0AP, London, UK
| | - Tobias U Hauser
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, WC1B 5EH, London, UK
- Wellcome Centre for Human Neuroimaging, University College London, WC1N 3BG, London, UK
| |
Collapse
|
29
|
Causal surgery under a Markov blanket. Behav Brain Sci 2022; 45:e218. [PMID: 36172770 DOI: 10.1017/s0140525x22000218] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Bruineberg et al. provide compelling clarity on the roles Markov blankets could (and perhaps should) play in the study of life and mind. However, here we draw attention to a further role blankets might play: as a hypothesis about cognition itself. People and other animals may use blanket-like representations to model the boundary between themselves and their worlds.
Collapse
|
30
|
Rouault M, Weiss A, Lee JK, Drugowitsch J, Chambon V, Wyart V. Controllability boosts neural and cognitive signatures of changes-of-mind in uncertain environments. eLife 2022; 11:75038. [PMID: 36097814 PMCID: PMC9470160 DOI: 10.7554/elife.75038] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2021] [Accepted: 07/19/2022] [Indexed: 11/13/2022] Open
Abstract
In uncertain environments, seeking information about alternative choice options is essential for adaptive learning and decision-making. However, information seeking is usually confounded with changes-of-mind about the reliability of the preferred option. Here, we exploited the fact that information seeking requires control over which option to sample to isolate its behavioral and neurophysiological signatures. We found that changes-of-mind occurring with control require more evidence against the current option, are associated with reduced confidence, but are nevertheless more likely to be confirmed on the next decision. Multimodal neurophysiological recordings showed that these changes-of-mind are preceded by stronger activation of the dorsal attention network in magnetoencephalography, and followed by increased pupil-linked arousal during the presentation of decision outcomes. Together, these findings indicate that information seeking increases the saliency of evidence perceived as the direct consequence of one's own actions.
Collapse
Affiliation(s)
- Marion Rouault
- Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France.,Institut Jean Nicod, Centre National de la Recherche Scientifique (CNRS), Paris, France.,Département d'Études Cognitives, École Normale Supérieure, Université Paris Sciences et Lettres (PSL University), Paris, France
| | - Aurélien Weiss
- Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France.,Département d'Études Cognitives, École Normale Supérieure, Université Paris Sciences et Lettres (PSL University), Paris, France.,Université de Paris, Paris, France
| | - Junseok K Lee
- Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France.,Département d'Études Cognitives, École Normale Supérieure, Université Paris Sciences et Lettres (PSL University), Paris, France
| | - Jan Drugowitsch
- Department of Neurobiology, Harvard Medical School, Boston, United States
| | - Valerian Chambon
- Institut Jean Nicod, Centre National de la Recherche Scientifique (CNRS), Paris, France.,Département d'Études Cognitives, École Normale Supérieure, Université Paris Sciences et Lettres (PSL University), Paris, France
| | - Valentin Wyart
- Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France.,Département d'Études Cognitives, École Normale Supérieure, Université Paris Sciences et Lettres (PSL University), Paris, France
| |
Collapse
|
31
|
Rojas GR, Curry-Pochy LS, Chen CS, Heller AT, Grissom NM. Sequential delay and probability discounting tasks in mice reveal anchoring effects partially attributable to decision noise. Behav Brain Res 2022; 431:113951. [PMID: 35661751 PMCID: PMC9844124 DOI: 10.1016/j.bbr.2022.113951] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2021] [Revised: 05/20/2022] [Accepted: 05/29/2022] [Indexed: 01/19/2023]
Abstract
Delay discounting and probability discounting decision making tasks in rodent models have high translational potential. However, it is unclear whether the discounted value of the large reward option is the main contributor to variability in animals' choices in either task, which may limit translation to humans. Male and female mice underwent sessions of delay and probability discounting in sequence to assess how choice behavior adapts over experience with each task. To control for "anchoring" (persistent choices based on the initial delay or probability), mice experienced "Worsening" schedules where the large reward was offered under initially favorable conditions that became less favorable during testing, followed by "Improving" schedules where the large reward was offered under initially unfavorable conditions that improved over a session. During delay discounting, both male and female mice showed elimination of anchoring effects over training. In probability discounting, both sexes of mice continued to show some anchoring even after months of training. One possibility is that "noisy", exploratory choices could contribute to these persistent anchoring effects, rather than constant fluctuations in value discounting. We fit choice behavior in individual animals using models that included both a value-based discounting parameter and a decision noise parameter that captured variability in choices deviating from value maximization. Changes in anchoring behavior over time were tracked by changes in both the value and decision noise parameters in delay discounting, but by the decision noise parameter in probability discounting. Exploratory decision making was also reflected in choice response times that tracked the degree of conflict caused by both uncertainty and temporal cost, but was not linked with differences in locomotor activity reflecting chamber exploration. Thus, variable discounting behavior in mice can result from changes in exploration of the decision options rather than changes in reward valuation.
Collapse
|
32
|
Abstract
Deciding whether to forgo a good choice in favour of exploring a potentially more rewarding alternative is one of the most challenging arbitrations both in human reasoning and in artificial intelligence. Humans show substantial variability in their exploration, and theoretical (but only limited empirical) work has suggested that excessive exploration is a critical mechanism underlying the psychiatric dimension of impulsivity. In this registered report, we put these theories to test using large online samples, dimensional analyses, and computational modelling. Capitalising on recent advances in disentangling distinct human exploration strategies, we not only demonstrate that impulsivity is associated with a specific form of exploration—value-free random exploration—but also explore links between exploration and other psychiatric dimensions. The Stage 1 protocol for this Registered Report was accepted in principle on 19/03/2021. The protocol, as accepted by the journal, can be found at 10.6084/m9.figshare.14346506.v1. Deciding between known rewarding options and exploring novel avenues is central to decision making. Humans show variability in their exploration. Here, the authors show that impulsivity is associated to an increased usage of a cognitively cheap (and sometimes sub-optimal) exploration strategy.
Collapse
|
33
|
Poli F, Meyer M, Mars RB, Hunnius S. Contributions of expected learning progress and perceptual novelty to curiosity-driven exploration. Cognition 2022; 225:105119. [PMID: 35421742 PMCID: PMC9194910 DOI: 10.1016/j.cognition.2022.105119] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2021] [Revised: 03/31/2022] [Accepted: 04/01/2022] [Indexed: 11/30/2022]
Abstract
Exploration is curiosity-driven when it relies on the intrinsic motivation to know rather than on extrinsic rewards. Recent evidence shows that artificial agents perform better on a variety of tasks when their learning is curiosity-driven, and humans often engage in curiosity-driven learning when sampling information from the environment. However, which mechanisms underlie curiosity is still unclear. Here, we let participants freely explore different unknown environments that contained learnable sequences of events with varying degrees of noise and volatility. A hierarchical reinforcement learning model captured how participants were learning in these different kinds of unknown environments, and it also tracked the errors they expected to make and the learning opportunities they were planning to seek. With this computational approach, we show that participants' exploratory behavior is guided by learning progress and perceptual novelty. Moreover, we demonstrate an overall tendency of participants to avoid extreme forms of uncertainty. These findings elucidate the cognitive mechanisms that underlie curiosity-driven exploration of unknown environments. Implications of this novel way of quantifying curiosity within a reinforcement learning framework are discussed.
Collapse
Affiliation(s)
- Francesco Poli
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands.
| | - Marlene Meyer
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands
| | - Rogier B Mars
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands; Wellcome Centre for Integrative Neuroimaging, Centre for Functional MRI of the Brain (FMRIB), Nuffield Department of Clinical Neurosciences, John Radcliffe Hospital, University of Oxford, Oxford, UK
| | - Sabine Hunnius
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands
| |
Collapse
|
34
|
Król M, Król ME. Great Minds Think Alike? Spatial Search Processes Can Be More Idiosyncratic When Guided by More Accurate Information. Cogn Sci 2022; 46:e13132. [PMID: 35411964 PMCID: PMC9286361 DOI: 10.1111/cogs.13132] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2021] [Revised: 03/17/2022] [Accepted: 03/21/2022] [Indexed: 11/29/2022]
Abstract
Existing research demonstrates that pre‐decisional information sampling strategies are often stable within a given person while varying greatly across people. However, it remains largely unknown what drives these individual differences, that is, why in some circumstances we collect information more idiosyncratically. In this brief report, we present a pre‐registered online study of spatial search. Using a novel technique that combines machine‐learning dimension reduction and sequence alignment algorithms, we quantify the extent to which the shape and temporal properties of a search trajectory are idiosyncratic. We show that this metric increases (trajectories become more idiosyncratic) when a person is better informed about the likely location of the search target, while poorly informed individuals seem more likely to resort to default search routines determined bottom‐up by the properties of the search field. This shows that when many people independently attempt to solve a task in a similar way, they are not necessarily “onto something.”
Collapse
Affiliation(s)
- Michal Król
- School of Business and Law, Universitetet i Agder
| | - Magdalena E Król
- Wroclaw Faculty of Psychology, SWPS University of Social Sciences and Humanities
| |
Collapse
|
35
|
Grujic N, Brus J, Burdakov D, Polania R. Rational inattention in mice. SCIENCE ADVANCES 2022; 8:eabj8935. [PMID: 35245128 PMCID: PMC8896787 DOI: 10.1126/sciadv.abj8935] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2023]
Abstract
Behavior exhibited by humans and other organisms is generally inconsistent and biased and, thus, is often labeled irrational. However, the origins of this seemingly suboptimal behavior remain elusive. We developed a behavioral task and normative framework to reveal how organisms should allocate their limited processing resources such that sensory precision and its related metabolic investment are balanced to guarantee maximal utility. We found that mice act as rational inattentive agents by adaptively allocating their sensory resources in a way that maximizes reward consumption in previously unexperienced stimulus-reward association environments. Unexpectedly, perception of commonly occurring stimuli was relatively imprecise; however, this apparent statistical fallacy implies "awareness" and efficient adaptation to their neurocognitive limitations. Arousal systems carry reward distribution information of sensory signals, and distributional reinforcement learning mechanisms regulate sensory precision via top-down normalization. These findings reveal how organisms efficiently perceive and adapt to previously unexperienced environmental contexts within the constraints imposed by neurobiology.
Collapse
Affiliation(s)
- Nikola Grujic
- Institute for Neuroscience, Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland
- Neuroscience Center Zürich, Zurich, Switzerland
| | - Jeroen Brus
- Neuroscience Center Zürich, Zurich, Switzerland
- Decision Neuroscience Lab, Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland
| | - Denis Burdakov
- Institute for Neuroscience, Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland
- Neuroscience Center Zürich, Zurich, Switzerland
- Corresponding author. (R.P.); (D.B.)
| | - Rafael Polania
- Neuroscience Center Zürich, Zurich, Switzerland
- Decision Neuroscience Lab, Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland
- Corresponding author. (R.P.); (D.B.)
| |
Collapse
|
36
|
A neural and behavioral trade-off between value and uncertainty underlies exploratory decisions in normative anxiety. Mol Psychiatry 2022; 27:1573-1587. [PMID: 34725456 DOI: 10.1038/s41380-021-01363-z] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Revised: 10/10/2021] [Accepted: 10/14/2021] [Indexed: 11/08/2022]
Abstract
Exploration reduces uncertainty about the environment and improves the quality of future decisions, but at the cost of provisional uncertain and suboptimal outcomes. Although anxiety promotes intolerance to uncertainty, it remains unclear whether and by which mechanisms anxiety relates to exploratory decision-making. We use a dynamic three-armed-bandit task and find that higher trait-anxiety is associated with increased exploration, which in turn harms overall performance. We identify two distinct behavioral sources: first, decisions made by anxious individuals are guided toward reduction of uncertainty; and second, decisions are less guided by immediate value gains. These findings are similar in both loss and gain domains, and further demonstrate that an affective trait relates to exploration and results in an inverse-U-shaped relationship between anxiety and overall performance. Additional imaging data (fMRI) suggests that normative anxiety correlates negatively with the representation of expected-value in the dorsal-anterior-cingulate-cortex, and in contrast, positively with the representation of uncertainty in the anterior-insula. We conclude that a trade-off between value-gains and uncertainty-reduction entails maladaptive decision-making in individuals with higher normal-range anxiety.
Collapse
|
37
|
Shamay-Tsoory SG, Hertz U. Adaptive Empathy: A Model for Learning Empathic Responses in Response to Feedback. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2022; 17:1008-1023. [PMID: 35050819 DOI: 10.1177/17456916211031926] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2023]
Abstract
Empathy is usually deployed in social interactions. Nevertheless, common measures and examinations of empathy study this construct in isolation from the person in distress. In this article we seek to extend the field of examination to include both empathizer and target to determine whether and how empathic responses are affected by feedback and learned through interaction. Building on computational approaches in feedback-based adaptations (e.g., no feedback, model-free and model-based learning), we propose a framework for understanding how empathic responses are learned on the basis of feedback. In this framework, adaptive empathy, defined as the ability to adapt one's empathic responses, is a central aspect of empathic skills and can provide a new dimension to the evaluation and investigation of empathy. By extending existing neural models of empathy, we suggest that adaptive empathy may be mediated by interactions between the neural circuits associated with valuation, shared distress, observation-execution, and mentalizing. Finally, we propose that adaptive empathy should be considered a prominent facet of empathic capabilities with the potential to explain empathic behavior in health and psychopathology.
Collapse
Affiliation(s)
- Simone G Shamay-Tsoory
- Department of Psychology, University of Haifa.,Integrated Brain and Behavior Research Center (IBBRC), University of Haifa
| | - Uri Hertz
- Integrated Brain and Behavior Research Center (IBBRC), University of Haifa.,Department of Cognitive Sciences, University of Haifa
| |
Collapse
|
38
|
Lefebvre G, Summerfield C, Bogacz R. A Normative Account of Confirmation Bias During Reinforcement Learning. Neural Comput 2022; 34:307-337. [PMID: 34758486 PMCID: PMC7612695 DOI: 10.1162/neco_a_01455] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2021] [Accepted: 07/26/2021] [Indexed: 11/04/2022]
Abstract
Reinforcement learning involves updating estimates of the value of states and actions on the basis of experience. Previous work has shown that in humans, reinforcement learning exhibits a confirmatory bias: when the value of a chosen option is being updated, estimates are revised more radically following positive than negative reward prediction errors, but the converse is observed when updating the unchosen option value estimate. Here, we simulate performance on a multi-arm bandit task to examine the consequences of a confirmatory bias for reward harvesting. We report a paradoxical finding: that confirmatory biases allow the agent to maximize reward relative to an unbiased updating rule. This principle holds over a wide range of experimental settings and is most influential when decisions are corrupted by noise. We show that this occurs because on average, confirmatory biases lead to overestimating the value of more valuable bandits and underestimating the value of less valuable bandits, rendering decisions overall more robust in the face of noise. Our results show how apparently suboptimal learning rules can in fact be reward maximizing if decisions are made with finite computational precision.
Collapse
Affiliation(s)
- Germain Lefebvre
- MRC Brain Network Dynamics Unit, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford OX3 9DU, U.K.
| | | | - Rafal Bogacz
- MRC Brain Network Dynamics Unit, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford OX3 9DU, U.K.
| |
Collapse
|
39
|
Smith R, Taylor S, Wilson RC, Chuning AE, Persich MR, Wang S, Killgore WDS. Lower Levels of Directed Exploration and Reflective Thinking Are Associated With Greater Anxiety and Depression. Front Psychiatry 2022; 12:782136. [PMID: 35126200 PMCID: PMC8808291 DOI: 10.3389/fpsyt.2021.782136] [Citation(s) in RCA: 15] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/23/2021] [Accepted: 12/07/2021] [Indexed: 01/15/2023] Open
Abstract
Anxiety and depression are often associated with strong beliefs that entering specific situations will lead to aversive outcomes - even when these situations are objectively safe and avoiding them reduces well-being. A possible mechanism underlying this maladaptive avoidance behavior is a failure to reflect on: (1) appropriate levels of uncertainty about the situation, and (2) how this uncertainty could be reduced by seeking further information (i.e., exploration). To test this hypothesis, we asked a community sample of 416 individuals to complete measures of reflective cognition, exploration, and symptoms of anxiety and depression. Consistent with our hypotheses, we found significant associations between each of these measures in expected directions (i.e., positive relationships between reflective cognition and strategic information-seeking behavior or "directed exploration", and negative relationships between these measures and anxiety/depression symptoms). Further analyses suggested that the relationship between directed exploration and depression/anxiety was due in part to an ambiguity aversion promoting exploration in conditions where information-seeking was not beneficial (as opposed to only being due to under-exploration when more information would aid future choices). In contrast, reflectiveness was associated with greater exploration in appropriate settings and separately accounted for differences in reaction times, decision noise, and choice accuracy in expected directions. These results shed light on the mechanisms underlying information-seeking behavior and how they may contribute to symptoms of emotional disorders. They also highlight the potential clinical relevance of individual differences in reflectiveness and exploration and should motivate future research on their possible contributions to vulnerability and/or maintenance of affective disorders.
Collapse
Affiliation(s)
- Ryan Smith
- Laureate Institute for Brain Research, Tulsa, OK, United States
| | - Samuel Taylor
- Laureate Institute for Brain Research, Tulsa, OK, United States
| | - Robert C. Wilson
- Department of Psychology, University of Arizona, Tucson, AZ, United States
| | - Anne E. Chuning
- Laureate Institute for Brain Research, Tulsa, OK, United States
| | | | - Siyu Wang
- Department of Psychology, University of Arizona, Tucson, AZ, United States
| | - William D. S. Killgore
- Department of Psychology, University of Arizona, Tucson, AZ, United States
- Department of Psychiatry, University of Arizona, Tucson, AZ, United States
| |
Collapse
|
40
|
Katthagen T, Fromm S, Wieland L, Schlagenhauf F. Models of Dynamic Belief Updating in Psychosis-A Review Across Different Computational Approaches. Front Psychiatry 2022; 13:814111. [PMID: 35492702 PMCID: PMC9039658 DOI: 10.3389/fpsyt.2022.814111] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Accepted: 02/18/2022] [Indexed: 11/20/2022] Open
Abstract
To understand the dysfunctional mechanisms underlying maladaptive reasoning of psychosis, computational models of decision making have widely been applied over the past decade. Thereby, a particular focus has been on the degree to which beliefs are updated based on new evidence, expressed by the learning rate in computational models. Higher order beliefs about the stability of the environment can determine the attribution of meaningfulness to events that deviate from existing beliefs by interpreting these either as noise or as true systematic changes (volatility). Both, the inappropriate downplaying of important changes as noise (belief update too low) as well as the overly flexible adaptation to random events (belief update too high) were theoretically and empirically linked to symptoms of psychosis. Whereas models with fixed learning rates fail to adjust learning in reaction to dynamic changes, increasingly complex learning models have been adopted in samples with clinical and subclinical psychosis lately. These ranged from advanced reinforcement learning models, over fully Bayesian belief updating models to approximations of fully Bayesian models with hierarchical learning or change point detection algorithms. It remains difficult to draw comparisons across findings of learning alterations in psychosis modeled by different approaches e.g., the Hierarchical Gaussian Filter and change point detection. Therefore, this review aims to summarize and compare computational definitions and findings of dynamic belief updating without perceptual ambiguity in (sub)clinical psychosis across these different mathematical approaches. There was strong heterogeneity in tasks and samples. Overall, individuals with schizophrenia and delusion-proneness showed lower behavioral performance linked to failed differentiation between uninformative noise and environmental change. This was indicated by increased belief updating and an overestimation of volatility, which was associated with cognitive deficits. Correlational evidence for computational mechanisms and positive symptoms is still sparse and might diverge from the group finding of instable beliefs. Based on the reviewed studies, we highlight some aspects to be considered to advance the field with regard to task design, modeling approach, and inclusion of participants across the psychosis spectrum. Taken together, our review shows that computational psychiatry offers powerful tools to advance our mechanistic insights into the cognitive anatomy of psychotic experiences.
Collapse
Affiliation(s)
- Teresa Katthagen
- Department of Psychiatry and Neurosciences, CCM, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany
| | - Sophie Fromm
- Department of Psychiatry and Neurosciences, CCM, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany.,Einstein Center for Neurosciences, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany.,Bernstein Center for Computational Neuroscience, Berlin, Germany
| | - Lara Wieland
- Department of Psychiatry and Neurosciences, CCM, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany.,Einstein Center for Neurosciences, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany.,Bernstein Center for Computational Neuroscience, Berlin, Germany
| | - Florian Schlagenhauf
- Department of Psychiatry and Neurosciences, CCM, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany.,Einstein Center for Neurosciences, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany.,Bernstein Center for Computational Neuroscience, Berlin, Germany.,NeuroCure Clinical Research Center, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany
| |
Collapse
|
41
|
Pulcu E, Guinea C, Cowen PJ, Murphy SE, Harmer CJ. A translational perspective on the anti-anhedonic effect of ketamine and its neural underpinnings. Mol Psychiatry 2022; 27:81-87. [PMID: 34158619 PMCID: PMC8960410 DOI: 10.1038/s41380-021-01183-1] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Revised: 05/15/2021] [Accepted: 05/28/2021] [Indexed: 02/06/2023]
Abstract
Anhedonia, a pronounced reduction in interest or pleasure in any of life's daily activities, is a cardinal symptom of major depression. In this Perspective article, we synthesise the recent evidence from rodent, monkey and human neuroimaging literature to highlight how the habenula, a small evolutionarily conserved subcortical structure located in the midbrain, may orchestrate the behavioural expression of anhedonia across fronto-mesolimbic networks. We then review how this circuitry can be modulated by ketamine, an NMDA receptor antagonist with rapid antidepressant properties. We propose that experimental paradigms founded in reinforcement learning and value-based decision-making can usefully probe this network and thereby help elucidate the mechanisms underlying ketamine's rapid antidepressant action.
Collapse
Affiliation(s)
- Erdem Pulcu
- Department of Psychiatry, University of Oxford, Oxford, UK.
- Oxford Health NHS Foundation Trust, Warneford Hospital, Oxford, OX3 7JX, UK.
| | - Calum Guinea
- Department of Psychiatry, University of Oxford, Oxford, UK
- Oxford Health NHS Foundation Trust, Warneford Hospital, Oxford, OX3 7JX, UK
| | - Philip J Cowen
- Department of Psychiatry, University of Oxford, Oxford, UK
- Oxford Health NHS Foundation Trust, Warneford Hospital, Oxford, OX3 7JX, UK
| | - Susannah E Murphy
- Department of Psychiatry, University of Oxford, Oxford, UK
- Oxford Health NHS Foundation Trust, Warneford Hospital, Oxford, OX3 7JX, UK
| | - Catherine J Harmer
- Department of Psychiatry, University of Oxford, Oxford, UK.
- Oxford Health NHS Foundation Trust, Warneford Hospital, Oxford, OX3 7JX, UK.
| |
Collapse
|
42
|
Foucault C, Meyniel F. Gated recurrence enables simple and accurate sequence prediction in stochastic, changing, and structured environments. eLife 2021; 10:71801. [PMID: 34854377 PMCID: PMC8735865 DOI: 10.7554/elife.71801] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2021] [Accepted: 12/01/2021] [Indexed: 11/13/2022] Open
Abstract
From decision making to perception to language, predicting what is coming next is crucial. It is also challenging in stochastic, changing, and structured environments; yet the brain makes accurate predictions in many situations. What computational architecture could enable this feat? Bayesian inference makes optimal predictions but is prohibitively difficult to compute. Here, we show that a specific recurrent neural network architecture enables simple and accurate solutions in several environments. This architecture relies on three mechanisms: gating, lateral connections, and recurrent weight training. Like the optimal solution and the human brain, such networks develop internal representations of their changing environment (including estimates of the environment’s latent variables and the precision of these estimates), leverage multiple levels of latent structure, and adapt their effective learning rate to changes without changing their connection weights. Being ubiquitous in the brain, gated recurrence could therefore serve as a generic building block to predict in real-life environments.
Collapse
Affiliation(s)
- Cédric Foucault
- INSERM, CEA, Université Paris-Saclay, Gif sur Yvette, France
| | | |
Collapse
|
43
|
|
44
|
Hertz U, Bell V, Raihani N. Trusting and learning from others: immediate and long-term effects of learning from observation and advice. Proc Biol Sci 2021; 288:20211414. [PMID: 34666522 PMCID: PMC8527195 DOI: 10.1098/rspb.2021.1414] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2021] [Accepted: 09/29/2021] [Indexed: 11/16/2022] Open
Abstract
Social learning underpins our species's extraordinary success. Learning through observation has been investigated in several species, but learning from advice-where information is intentionally broadcast-is less understood. We used a pre-registered, online experiment (n = 1492) combined with computational modelling to examine learning through observation and advice. Participants were more likely to immediately follow advice than to copy an observed choice, but this was dependent upon trust in the adviser: highly paranoid participants were less likely to follow advice in the short term. Reinforcement learning modelling revealed two distinct patterns regarding the long-term effects of social information: some individuals relied fully on social information, whereas others reverted to trial-and-error learning. This variation may affect the prevalence and fidelity of socially transmitted information. Our results highlight the privileged status of advice relative to observation and how the assimilation of intentionally broadcast information is affected by trust in others.
Collapse
Affiliation(s)
- Uri Hertz
- Department of Cognitive Sciences, University of Haifa, Haifa, Israel
| | - Vaughan Bell
- Department of Clinical, Education and Health Psychology, University College London, London, UK
| | - Nichola Raihani
- Department of Experimental Psychology, University College London, WC1H 0AP, London, UK
| |
Collapse
|
45
|
Nassar MR, Waltz JA, Albrecht MA, Gold JM, Frank MJ. All or nothing belief updating in patients with schizophrenia reduces precision and flexibility of beliefs. Brain 2021; 144:1013-1029. [PMID: 33434284 DOI: 10.1093/brain/awaa453] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2020] [Revised: 10/09/2020] [Accepted: 10/14/2020] [Indexed: 01/11/2023] Open
Abstract
Schizophrenia is characterized by abnormal perceptions and beliefs, but the computational mechanisms through which these abnormalities emerge remain unclear. One prominent hypothesis asserts that such abnormalities result from overly precise representations of prior knowledge, which in turn lead beliefs to become insensitive to feedback. In contrast, another prominent hypothesis asserts that such abnormalities result from a tendency to interpret prediction errors as indicating meaningful change, leading to the assignment of aberrant salience to noisy or misleading information. Here we examine behaviour of patients and control subjects in a behavioural paradigm capable of adjudicating between these competing hypotheses and characterizing belief updates directly on individual trials. We show that patients are more prone to completely ignoring new information and perseverating on previous responses, but when they do update, tend to do so completely. This updating strategy limits the integration of information over time, reducing both the flexibility and precision of beliefs and provides a potential explanation for how patients could simultaneously show over-sensitivity and under-sensitivity to feedback in different paradigms.
Collapse
Affiliation(s)
- Matthew R Nassar
- Robert J. and Nancy D. Carney Institute for Brain Science, Brown University, Providence RI 02912-1821, USA.,Department of Neuroscience, Brown University, Providence RI 02912-1821, USA
| | - James A Waltz
- Maryland Psychiatric Research Center, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Matthew A Albrecht
- School of Public Health, Curtin University, Perth, Western Australia, Australia
| | - James M Gold
- Maryland Psychiatric Research Center, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Michael J Frank
- Robert J. and Nancy D. Carney Institute for Brain Science, Brown University, Providence RI 02912-1821, USA.,Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence RI 02912-1821, USA
| |
Collapse
|
46
|
Pfeffer T, Ponce-Alvarez A, Tsetsos K, Meindertsma T, Gahnström CJ, van den Brink RL, Nolte G, Engel AK, Deco G, Donner TH. Circuit mechanisms for the chemical modulation of cortex-wide network interactions and behavioral variability. SCIENCE ADVANCES 2021; 7:eabf5620. [PMID: 34272245 PMCID: PMC8284895 DOI: 10.1126/sciadv.abf5620] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/04/2020] [Accepted: 06/03/2021] [Indexed: 05/07/2023]
Abstract
Influential theories postulate distinct roles of catecholamines and acetylcholine in cognition and behavior. However, previous physiological work reported similar effects of these neuromodulators on the response properties (specifically, the gain) of individual cortical neurons. Here, we show a double dissociation between the effects of catecholamines and acetylcholine at the level of large-scale interactions between cortical areas in humans. A pharmacological boost of catecholamine levels increased cortex-wide interactions during a visual task, but not rest. An acetylcholine boost decreased interactions during rest, but not task. Cortical circuit modeling explained this dissociation by differential changes in two circuit properties: the local excitation-inhibition balance (more strongly increased by catecholamines) and intracortical transmission (more strongly reduced by acetylcholine). The inferred catecholaminergic mechanism also predicted noisier decision-making, which we confirmed for both perceptual and value-based choice behavior. Our work highlights specific circuit mechanisms for shaping cortical network interactions and behavioral variability by key neuromodulatory systems.
Collapse
Affiliation(s)
- Thomas Pfeffer
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
- Center for Brain and Cognition, Computational Neuroscience Group, Universitat Pompeu Fabra, Barcelona, Spain
| | - Adrian Ponce-Alvarez
- Center for Brain and Cognition, Computational Neuroscience Group, Universitat Pompeu Fabra, Barcelona, Spain
| | - Konstantinos Tsetsos
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Thomas Meindertsma
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| | - Christoffer Julius Gahnström
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Ruud Lucas van den Brink
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Guido Nolte
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Andreas Karl Engel
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Gustavo Deco
- Center for Brain and Cognition, Computational Neuroscience Group, Universitat Pompeu Fabra, Barcelona, Spain
- Institució Catalana de la Recerca i Estudis Avançats (ICREA), Barcelona, Spain
- Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
- School of Psychological Sciences, Monash University, Melbourne, Australia
| | - Tobias Hinrich Donner
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
- Amsterdam Brain and Cognition, University of Amsterdam, Amsterdam, Netherlands
- Bernstein Center for Computational Neuroscience Berlin, Berlin, Germany
| |
Collapse
|
47
|
de Gee JW, Correa CMC, Weaver M, Donner TH, van Gaal S. Pupil Dilation and the Slow Wave ERP Reflect Surprise about Choice Outcome Resulting from Intrinsic Variability in Decision Confidence. Cereb Cortex 2021; 31:3565-3578. [PMID: 33822917 PMCID: PMC8196307 DOI: 10.1093/cercor/bhab032] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2020] [Revised: 01/27/2021] [Accepted: 01/27/2021] [Indexed: 12/01/2022] Open
Abstract
Central to human and animal cognition is the ability to learn from feedback in order to optimize future rewards. Such a learning signal might be encoded and broadcasted by the brain's arousal systems, including the noradrenergic locus coeruleus. Pupil responses and the positive slow wave component of event-related potentials reflect rapid changes in the arousal level of the brain. Here, we ask whether and how these variables may reflect surprise: the mismatch between one's expectation about being correct and the outcome of a decision, when expectations fluctuate due to internal factors (e.g., engagement). We show that during an elementary decision task in the face of uncertainty both physiological markers of phasic arousal reflect surprise. We further show that pupil responses and slow wave event-related potential are unrelated to each other and that prediction error computations depend on feedback awareness. These results further advance our understanding of the role of central arousal systems in decision-making under uncertainty.
Collapse
Affiliation(s)
- Jan Willem de Gee
- Department of Psychology, Amsterdam Brain & Cognition (ABC), University of Amsterdam, Nieuwe Achtergracht 129-B, 1018WS, Amsterdam, the Netherlands
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Building N43, Martinistraße 52, 20246, Hamburg, Germany
- Department of Neuroscience, Baylor College of Medicine, 1 Baylor Plaza, Houston, TX 77030, USA
- Jan and Dan Duncan Neurological Research Institute, Texas Children’s Hospital, 1250 Moursund St, Houston, TX 77030, USA
| | - Camile M C Correa
- Department of Psychology, Amsterdam Brain & Cognition (ABC), University of Amsterdam, Nieuwe Achtergracht 129-B, 1018WS, Amsterdam, the Netherlands
- Centre of Functionally Integrative Neuroscience, Aarhus University, 44 Nørrebrogade Building 1A, 8000 Aarhus, Denmark
| | - Matthew Weaver
- Department of Psychology, Amsterdam Brain & Cognition (ABC), University of Amsterdam, Nieuwe Achtergracht 129-B, 1018WS, Amsterdam, the Netherlands
| | - Tobias H Donner
- Department of Psychology, Amsterdam Brain & Cognition (ABC), University of Amsterdam, Nieuwe Achtergracht 129-B, 1018WS, Amsterdam, the Netherlands
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Building N43, Martinistraße 52, 20246, Hamburg, Germany
| | - Simon van Gaal
- Department of Psychology, Amsterdam Brain & Cognition (ABC), University of Amsterdam, Nieuwe Achtergracht 129-B, 1018WS, Amsterdam, the Netherlands
| |
Collapse
|
48
|
Interacting with volatile environments stabilizes hidden-state inference and its brain signatures. Nat Commun 2021; 12:2228. [PMID: 33850124 PMCID: PMC8044147 DOI: 10.1038/s41467-021-22396-6] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2019] [Accepted: 03/05/2021] [Indexed: 11/26/2022] Open
Abstract
Making accurate decisions in uncertain environments requires identifying the generative cause of sensory cues, but also the expected outcomes of possible actions. Although both cognitive processes can be formalized as Bayesian inference, they are commonly studied using different experimental frameworks, making their formal comparison difficult. Here, by framing a reversal learning task either as cue-based or outcome-based inference, we found that humans perceive the same volatile environment as more stable when inferring its hidden state by interaction with uncertain outcomes than by observation of equally uncertain cues. Multivariate patterns of magnetoencephalographic (MEG) activity reflected this behavioral difference in the neural interaction between inferred beliefs and incoming evidence, an effect originating from associative regions in the temporal lobe. Together, these findings indicate that the degree of control over the sampling of volatile environments shapes human learning and decision-making under uncertainty. Here, the authors show that humans perceive uncertain environments as more stable when actively interacting with them than when observing them. Magnetoencephalographic signals in the temporal lobe were associated with the increased stability of beliefs during active sampling.
Collapse
|
49
|
Wilson RC, Bonawitz E, Costa VD, Ebitz RB. Balancing exploration and exploitation with information and randomization. Curr Opin Behav Sci 2021; 38:49-56. [PMID: 33184605 PMCID: PMC7654823 DOI: 10.1016/j.cobeha.2020.10.001] [Citation(s) in RCA: 78] [Impact Index Per Article: 26.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022]
Abstract
Explore-exploit decisions require us to trade off the benefits of exploring unknown options to learn more about them, with exploiting known options, for immediate reward. Such decisions are ubiquitous in nature, but from a computational perspective, they are notoriously hard. There is therefore much interest in how humans and animals make these decisions and recently there has been an explosion of research in this area. Here we provide a biased and incomplete snapshot of this field focusing on the major finding that many organisms use two distinct strategies to solve the explore-exploit dilemma: a bias for information ('directed exploration') and the randomization of choice ('random exploration'). We review evidence for the existence of these strategies, their computational properties, their neural implementations, as well as how directed and random exploration vary over the lifespan. We conclude by highlighting open questions in this field that are ripe to both explore and exploit.
Collapse
Affiliation(s)
- Robert C. Wilson
- Department of Psychology, University of Arizona, Tucson AZ USA
- Cognitive Science Program, University of Arizona, Tucson AZ USA
- Evelyn F. McKnight Brain Institute, University of Arizona, Tucson AZ USA
| | | | - Vincent D. Costa
- Department of Behavioral Neuroscience, Oregon Health and Science University, Portland OR USA
| | - R. Becket Ebitz
- Department of Neuroscience, University of Montréal, Montréal, Québec, Canada
| |
Collapse
|
50
|
Findling C, Wyart V. Computation noise in human learning and decision-making: origin, impact, function. Curr Opin Behav Sci 2021. [DOI: 10.1016/j.cobeha.2021.02.018] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022]
|