1
|
Schwöbel S, Marković D, Smolka MN, Kiebel S. Joint modeling of choices and reaction times based on Bayesian contextual behavioral control. PLoS Comput Biol 2024; 20:e1012228. [PMID: 38968304 DOI: 10.1371/journal.pcbi.1012228] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2023] [Accepted: 06/04/2024] [Indexed: 07/07/2024] Open
Abstract
In cognitive neuroscience and psychology, reaction times are an important behavioral measure. However, in instrumental learning and goal-directed decision making experiments, findings often rely only on choice probabilities from a value-based model, instead of reaction times. Recent advancements have shown that it is possible to connect value-based decision models with reaction time models. However, typically these models do not provide an integrated account of both value-based choices and reaction times, but simply link two types of models. Here, we propose a novel integrative joint model of both choices and reaction times by combining a computational account of Bayesian sequential decision making with a sampling procedure. This allows us to describe how internal uncertainty in the planning process shapes reaction time distributions. Specifically, we use a recent context-specific Bayesian forward planning model which we extend by a Markov chain Monte Carlo (MCMC) sampler to obtain both choices and reaction times. As we will show this makes the sampler an integral part of the decision making process and enables us to reproduce, using simulations, well-known experimental findings in value based-decision making as well as classical inhibition and switching tasks. Specifically, we use the proposed model to explain both choice behavior and reaction times in instrumental learning and automatized behavior, in the Eriksen flanker task and in task switching. These findings show that the proposed joint behavioral model may describe common underlying processes in these different decision making paradigms.
Collapse
Affiliation(s)
- Sarah Schwöbel
- Department of Psychology, Technische Universität Dresden, Dresden, Germany
| | - Dimitrije Marković
- Department of Psychology, Technische Universität Dresden, Dresden, Germany
| | - Michael N Smolka
- Department of Psychiatry and Psychotherapy, Technische Universität Dresden, Dresden, Germany
| | - Stefan Kiebel
- Department of Psychology, Technische Universität Dresden, Dresden, Germany
- Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, Dresden, Germany
| |
Collapse
|
2
|
Parr T, Limanowski J. Synchronising our internal clocks: Comment on: "An active inference model of hierarchical action understanding, learning and imitation" by Proietti et al. Phys Life Rev 2023; 46:258-260. [PMID: 37544051 DOI: 10.1016/j.plrev.2023.07.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Accepted: 07/24/2023] [Indexed: 08/08/2023]
Affiliation(s)
- Thomas Parr
- Wellcome Centre for Human Neuroimaging, Queen Square Institute of Neurology, University College London, United Kingdom.
| | | |
Collapse
|
3
|
Stolz C, Pickering AD, Mueller EM. Dissociable feedback valence effects on frontal midline theta during reward gain versus threat avoidance learning. Psychophysiology 2022; 60:e14235. [PMID: 36529988 DOI: 10.1111/psyp.14235] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Revised: 10/17/2022] [Accepted: 11/17/2022] [Indexed: 12/23/2022]
Abstract
While frontal midline theta (FMθ) has been associated with threat processing, with cognitive control in the context of anxiety, and with reinforcement learning, most reinforcement learning studies on FMθ have used reward rather than threat-related stimuli as reinforcer. Accordingly, the role of FMθ in threat-related reinforcement learning is largely unknown. Here, n = 23 human participants underwent one reward-, and one punishment-, based reversal learning task, which differed only with regard to the kind of reinforcers that feedback was tied to (i.e., monetary gain vs. loud noise burst, respectively). In addition to single-trial EEG, we assessed single-trial feedback expectations based on both a reinforcement learning computational model and trial-by-trial subjective feedback expectation ratings. While participants' performance and feedback expectations were comparable between the reward and punishment tasks, FMθ was more reliably amplified to negative vs. positive feedback in the reward vs. punishment task. Regressions with feedback valence, computationally derived, and self-reported expectations as predictors and FMθ as criterion further revealed that trial-by-trial variations in FMθ specifically relate to reward-related feedback-valence and not to threat-related feedback or to violated expectations/prediction errors. These findings suggest that FMθ as measured in reinforcement learning tasks may be less sensitive to the processing of events with direct relevance for fear and anxiety.
Collapse
Affiliation(s)
- Christopher Stolz
- Department of Psychology University of Marburg Marburg Germany
- Leibniz Institute for Neurobiology (LIN) Magdeburg Germany
- Department of Psychology Goldsmiths, University of London London UK
| | | | - Erik M. Mueller
- Department of Psychology University of Marburg Marburg Germany
| |
Collapse
|
4
|
Marković D, Reiter AMF, Kiebel SJ. Revealing human sensitivity to a latent temporal structure of changes. Front Behav Neurosci 2022; 16:962494. [PMID: 36325156 PMCID: PMC9621332 DOI: 10.3389/fnbeh.2022.962494] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2022] [Accepted: 09/26/2022] [Indexed: 11/29/2022] Open
Abstract
Precisely timed behavior and accurate time perception plays a critical role in our everyday lives, as our wellbeing and even survival can depend on well-timed decisions. Although the temporal structure of the world around us is essential for human decision making, we know surprisingly little about how representation of temporal structure of our everyday environment impacts decision making. How does the representation of temporal structure affect our ability to generate well-timed decisions? Here we address this question by using a well-established dynamic probabilistic learning task. Using computational modeling, we found that human subjects' beliefs about temporal structure are reflected in their choices to either exploit their current knowledge or to explore novel options. The model-based analysis illustrates a large within-group and within-subject heterogeneity. To explain these results, we propose a normative model for how temporal structure is used in decision making, based on the semi-Markov formalism in the active inference framework. We discuss potential key applications of the presented approach to the fields of cognitive phenotyping and computational psychiatry.
Collapse
Affiliation(s)
- Dimitrije Marković
- Department of Psychology, Technische Universität Dresden, Dresden, Germany
- *Correspondence: Dimitrije Marković
| | - Andrea M. F. Reiter
- Department of Psychology, Technische Universität Dresden, Dresden, Germany
- Department of Child and Adolescence Psychiatry, Psychosomatics and Psychotherapy, Centre of Mental Health, University Hospital Würzburg, Würzburg, Germany
- German Center of Prevention Research on Mental Health, Julius-Maximilians Universität Würzburg, Würzburg, Germany
| | - Stefan J. Kiebel
- Department of Psychology, Technische Universität Dresden, Dresden, Germany
- Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, Dresden, Germany
| |
Collapse
|
5
|
Marković D, Stojić H, Schwöbel S, Kiebel SJ. An empirical evaluation of active inference in multi-armed bandits. Neural Netw 2021; 144:229-246. [PMID: 34507043 DOI: 10.1016/j.neunet.2021.08.018] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Revised: 07/07/2021] [Accepted: 08/11/2021] [Indexed: 10/20/2022]
Abstract
A key feature of sequential decision making under uncertainty is a need to balance between exploiting-choosing the best action according to the current knowledge, and exploring-obtaining information about values of other actions. The multi-armed bandit problem, a classical task that captures this trade-off, served as a vehicle in machine learning for developing bandit algorithms that proved to be useful in numerous industrial applications. The active inference framework, an approach to sequential decision making recently developed in neuroscience for understanding human and animal behaviour, is distinguished by its sophisticated strategy for resolving the exploration-exploitation trade-off. This makes active inference an exciting alternative to already established bandit algorithms. Here we derive an efficient and scalable approximate active inference algorithm and compare it to two state-of-the-art bandit algorithms: Bayesian upper confidence bound and optimistic Thompson sampling. This comparison is done on two types of bandit problems: a stationary and a dynamic switching bandit. Our empirical evaluation shows that the active inference algorithm does not produce efficient long-term behaviour in stationary bandits. However, in the more challenging switching bandit problem active inference performs substantially better than the two state-of-the-art bandit algorithms. The results open exciting venues for further research in theoretical and applied machine learning, as well as lend additional credibility to active inference as a general framework for studying human and animal behaviour.
Collapse
Affiliation(s)
- Dimitrije Marković
- Faculty of Psychology, Technische Universität Dresden, 01062 Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, 01062 Dresden, Germany.
| | - Hrvoje Stojić
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, 10-12 Russell Square, London, WC1B 5EH, United Kingdom; Secondmind, 72 Hills Rd, Cambridge, CB2 1LA, United Kingdom
| | - Sarah Schwöbel
- Faculty of Psychology, Technische Universität Dresden, 01062 Dresden, Germany
| | - Stefan J Kiebel
- Faculty of Psychology, Technische Universität Dresden, 01062 Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, 01062 Dresden, Germany
| |
Collapse
|
6
|
Marković D, Goschke T, Kiebel SJ. Meta-control of the exploration-exploitation dilemma emerges from probabilistic inference over a hierarchy of time scales. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2021; 21:509-533. [PMID: 33372237 PMCID: PMC8208938 DOI: 10.3758/s13415-020-00837-x] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Accepted: 09/17/2020] [Indexed: 12/12/2022]
Abstract
Cognitive control is typically understood as a set of mechanisms that enable humans to reach goals that require integrating the consequences of actions over longer time scales. Importantly, using routine behaviour or making choices beneficial only at short time scales would prevent one from attaining these goals. During the past two decades, researchers have proposed various computational cognitive models that successfully account for behaviour related to cognitive control in a wide range of laboratory tasks. As humans operate in a dynamic and uncertain environment, making elaborate plans and integrating experience over multiple time scales is computationally expensive. Importantly, it remains poorly understood how uncertain consequences at different time scales are integrated into adaptive decisions. Here, we pursue the idea that cognitive control can be cast as active inference over a hierarchy of time scales, where inference, i.e., planning, at higher levels of the hierarchy controls inference at lower levels. We introduce the novel concept of meta-control states, which link higher-level beliefs with lower-level policy inference. Specifically, we conceptualize cognitive control as inference over these meta-control states, where solutions to cognitive control dilemmas emerge through surprisal minimisation at different hierarchy levels. We illustrate this concept using the exploration-exploitation dilemma based on a variant of a restless multi-armed bandit task. We demonstrate that beliefs about contexts and meta-control states at a higher level dynamically modulate the balance of exploration and exploitation at the lower level of a single action. Finally, we discuss the generalisation of this meta-control concept to other control dilemmas.
Collapse
Affiliation(s)
- Dimitrije Marković
- Chair of Neuroimaging, Faculty of Psychology, Technische Universität Dresden, 01062, Dresden, Germany
| | - Thomas Goschke
- Chair of General Psychology, Faculty of Psychology, Technische Universität Dresden, 01062, Dresden, Germany
- Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, 01062, Dresden, Germany
| | - Stefan J Kiebel
- Chair of Neuroimaging, Faculty of Psychology, Technische Universität Dresden, 01062, Dresden, Germany.
- Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, 01062, Dresden, Germany.
| |
Collapse
|
7
|
Da Costa L, Parr T, Sengupta B, Friston K. Neural Dynamics under Active Inference: Plausibility and Efficiency of Information Processing. ENTROPY (BASEL, SWITZERLAND) 2021; 23:454. [PMID: 33921298 PMCID: PMC8069154 DOI: 10.3390/e23040454] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/16/2021] [Accepted: 04/06/2021] [Indexed: 02/07/2023]
Abstract
Active inference is a normative framework for explaining behaviour under the free energy principle-a theory of self-organisation originating in neuroscience. It specifies neuronal dynamics for state-estimation in terms of a descent on (variational) free energy-a measure of the fit between an internal (generative) model and sensory observations. The free energy gradient is a prediction error-plausibly encoded in the average membrane potentials of neuronal populations. Conversely, the expected probability of a state can be expressed in terms of neuronal firing rates. We show that this is consistent with current models of neuronal dynamics and establish face validity by synthesising plausible electrophysiological responses. We then show that these neuronal dynamics approximate natural gradient descent, a well-known optimisation algorithm from information geometry that follows the steepest descent of the objective in information space. We compare the information length of belief updating in both schemes, a measure of the distance travelled in information space that has a direct interpretation in terms of metabolic cost. We show that neural dynamics under active inference are metabolically efficient and suggest that neural representations in biological agents may evolve by approximating steepest descent in information space towards the point of optimal inference.
Collapse
Affiliation(s)
- Lancelot Da Costa
- Department of Mathematics, Imperial College London, London SW7 2AZ, UK
- Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, London WC1N 3BG, UK; (T.P.); (B.S.); (K.F.)
| | - Thomas Parr
- Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, London WC1N 3BG, UK; (T.P.); (B.S.); (K.F.)
| | - Biswa Sengupta
- Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, London WC1N 3BG, UK; (T.P.); (B.S.); (K.F.)
- Core Machine Learning Group, Zebra AI, London WC2H 8TJ, UK
- Department of Bioengineering, Imperial College London, London SW7 2AZ, UK
| | - Karl Friston
- Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, London WC1N 3BG, UK; (T.P.); (B.S.); (K.F.)
| |
Collapse
|
8
|
Abstract
The word diaphragm comes from the Greek (διάϕραγμα), which meant something that divides, but also expressed a concept related to emotions and intellect. Breath is part of a concept of symmorphosis, that is the maximum ability to adapt to multiple functional questions in a defined biological context. The act of breathing determines and defines our holobiont: how we react and who we are. The article reviews the fascial structure that involves and forms the diaphragm muscle with the aim of changing the vision of this complex muscle: from an anatomical and mechanistic form to a fractal and asynchronous form. Another step forward for understanding the diaphragm muscle is that it is not only covered, penetrated and made up of connective tissue, but the contractile tissue itself is a fascial tissue with the same embryological derivation. All the diaphragm muscle is fascia.
Collapse
Affiliation(s)
- Bruno Bordoni
- Cardiology, Foundation Don Carlo Gnocchi, Milan, ITA
| | | | - Bruno Morabito
- Osteopathy, School of Osteopathic Centre for Research and Studies, Milan, ITA
| |
Collapse
|