A step-by-step tutorial on active inference and its application to empirical data

Counteracting uncertainty: exploring the impact of anxiety on updating predictions about environmental states

Anxious emotional states disrupt decision-making and control of dexterous motor actions. Computational work has shown that anxiety-induced uncertainty alters the rate at which we learn about the environment, but the subsequent impact on the predictive beliefs that drive action control remains to be understood. In the present work we tested whether anxiety alters predictive (oculo)motor control mechanisms. Thirty participants completed an experimental task that consisted of manual interception of a projectile performed in virtual reality. Participants were subjected to conditions designed to induce states of high or low anxiety using performance incentives and social-evaluative pressure. We measured subsequent effects on physiological arousal, self-reported state anxiety, and eye movements. Under high pressure conditions we observed visual sampling of the task environment characterised by higher variability and entropy of position prior to release of the projectile, consistent with an active attempt to reduce uncertainty. Computational modelling of predictive beliefs, using gaze data as inputs to a partially observable Markov decision process model, indicated that trial-to-trial updating of predictive beliefs was reduced during anxiety, suggesting that updates to priors were constrained. Additionally, state anxiety was related to a less deterministic mapping of beliefs to actions. These results support the idea that organisms may attempt to counter anxiety-related uncertainty by moving towards more familiar and certain sensorimotor patterns.

An Active Inference Model of the Optimism Bias

The optimism bias is a cognitive bias where individuals overestimate the likelihood of good outcomes and underestimate the likelihood of bad outcomes. Associated with improved quality of life, optimism bias is considered to be adaptive and is a promising avenue of research for mental health interventions in conditions where individuals lack optimism such as major depressive disorder. Here we lay the groundwork for future research on optimism as an intervention by introducing a domain general formal model of optimism bias, which can be applied in different task settings. Employing the active inference framework, we propose a model of the optimism bias as high precision likelihood biased towards positive outcomes. First, we simulate how optimism may be lost during development by exposure to negative events. We then ground our model in the empirical literature by showing how the developmentally acquired differences in optimism are expressed in a belief updating task typically used to assess optimism bias. Finally, we show how optimism affects action in a modified two-armed bandit task. Our model and the simulations it affords provide a computational basis for understanding how optimism bias may emerge, how it may be expressed in standard tasks used to assess optimism, and how it affects agents' decision-making and actions; in combination, this provides a basis for future research on optimism as a mental health intervention.

Modelling sensory attenuation as Bayesian causal inference across two datasets

INTRODUCTION

To interact with the environment, it is crucial to distinguish between sensory information that is externally generated and inputs that are self-generated. The sensory consequences of one's own movements tend to induce attenuated behavioral- and neural responses compared to externally generated inputs. We propose a computational model of sensory attenuation (SA) based on Bayesian Causal Inference, where SA occurs when an internal cause for sensory information is inferred.

METHODS

Experiment 1investigates sensory attenuation during a stroking movement. Tactile stimuli on the stroking finger were suppressed, especially when they were predictable. Experiment 2 showed impaired delay detection between an arm movement and a video of the movement when participants were moving vs. when their arm was moved passively. We reconsider these results from the perspective of Bayesian Causal Inference (BCI). Using a hierarchical Markov Model (HMM) and variational message passing, we first qualitatively capture patterns of task behavior and sensory attenuation in simulations. Next, we identify participant-specific model parameters for both experiments using optimization.

RESULTS

A sequential BCI model is well equipped to capture empirical patterns of SA across both datasets. Using participant-specific optimized model parameters, we find a good agreement between data and model predictions, with the model capturing both tactile detections in Experiment 1 and delay detections in Experiment 2.

DISCUSSION

BCI is an appropriate framework to model sensory attenuation in humans. Computational models of sensory attenuation may help to bridge the gap across different sensory modalities and experimental paradigms and may contribute towards an improved description and understanding of deficits in specific patient groups (e.g. schizophrenia).

Premovement activity in the corticospinal tract is amplified by the placebo effect: an active inference account

The aim of this study is to investigate whether expectancy, induced through a placebo procedure, favors the activation of the corticospinal tract before movement initiation. By adopting the premovement facilitation paradigm, we applied transcranial magnetic stimulation over the left or right primary motor cortex at rest and 100 ms or 50 ms before movement onset while healthy volunteers performed a reaction time (RT) motor task consisting of abductions of the right or left thumb after a go signal. Participants in the placebo group received an inert electrical device applied on the right forearm along with information on its speed-enhancing properties. A control group received the same device with overt information about its inert nature, while another control group underwent no intervention. Along with RT, we measured the amplitude of the motor evoked potential (MEP) before and after the procedure. Compared to the control groups, the placebo group had faster RT and greater MEP amplitude before movement initiation. This study demonstrates that the placebo effect can boost the activity of the corticospinal tract before movement onset, and this modulation positively impacts motor performance. These results give experimental support to the active inference account.

Introducing ActiveInference.jl: A Julia Library for Simulation and Parameter Estimation with Active Inference Models

We introduce a new software package for the Julia programming language, the library ActiveInference.jl. To make active inference agents with Partially Observable Markov Decision Process (POMDP) generative models available to the growing research community using Julia, we re-implemented the pymdp library for Python. ActiveInference.jl is compatible with cutting-edge Julia libraries designed for cognitive and behavioural modelling, as it is used in computational psychiatry, cognitive science and neuroscience. This means that POMDP active inference models can now be easily fit to empirically observed behaviour using sampling, as well as variational methods. In this article, we show how ActiveInference.jl makes building POMDP active inference models straightforward, and how it enables researchers to use them for simulation, as well as fitting them to data or performing a model comparison.

Dynamic planning in hierarchical active inference

By dynamic planning, we refer to the ability of the human brain to infer and impose motor trajectories related to cognitive decisions. A recent paradigm, active inference, brings fundamental insights into the adaptation of biological organisms, constantly striving to minimize prediction errors to restrict themselves to life-compatible states. Over the past years, many studies have shown how human and animal behaviors could be explained in terms of active inference - either as discrete decision-making or continuous motor control - inspiring innovative solutions in robotics and artificial intelligence. Still, the literature lacks a comprehensive outlook on effectively planning realistic actions in changing environments. Setting ourselves the goal of modeling complex tasks such as tool use, we delve into the topic of dynamic planning in active inference, keeping in mind two crucial aspects of biological behavior: the capacity to understand and exploit affordances for object manipulation, and to learn the hierarchical interactions between the self and the environment, including other agents. We start from a simple unit and gradually describe more advanced structures, comparing recently proposed design choices and providing basic examples. This study distances itself from traditional views centered on neural networks and reinforcement learning, and points toward a yet unexplored direction in active inference: hybrid representations in hierarchical models.

Flow and intuition: a systems neuroscience comparison

This paper explores the relationship between intuition and flow from a neurodynamics perspective. Flow and intuition represent two cognitive phenomena rooted in nonconscious information processing; however, there are clear differences in both their phenomenal characteristics and, more broadly, their contribution to action and cognition. We propose, extrapolating from dual processing theory, that intuition serves as a rapid, nonconscious decision-making process, while flow facilitates this process in action, achieving optimal cognitive control and performance without [conscious] deliberation. By exploring these points of convergence between flow and intuition, we also attempt to reconcile the apparent paradox of the presence of enhanced intuition in flow, which is also a state of heightened cognitive control. To do so, we utilize a revised dual-processing framework, which allows us to productively align and differentiate flow and intuition (including intuition in flow). Furthermore, we draw on recent work examining flow from an active inference perspective. Our account not only heightens understanding of human cognition and consciousness, but also raises new questions for future research, aiming to deepen our comprehension of how flow and intuition can be harnessed to elevate human performance and wellbeing.

Body as First Teacher: The Role of Rhythmic Visceral Dynamics in Early Cognitive Development

Embodied cognition-the idea that mental states and processes should be understood in relation to one's bodily constitution and interactions with the world-remains a controversial topic within cognitive science. Recently, however, increasing interest in predictive processing theories among proponents and critics of embodiment alike has raised hopes of a reconciliation. This article sets out to appraise the unificatory potential of predictive processing, focusing in particular on embodied formulations of active inference. Our analysis suggests that most active-inference accounts invoke weak, potentially trivial conceptions of embodiment; those making stronger claims do so independently of the theoretical commitments of the active-inference framework. We argue that a more compelling version of embodied active inference can be motivated by adopting a diachronic perspective on the way rhythmic physiological activity shapes neural development in utero. According to this visceral afferent training hypothesis, early-emerging physiological processes are essential not only for supporting the biophysical development of neural structures but also for configuring the cognitive architecture those structures entail. Focusing in particular on the cardiovascular system, we propose three candidate mechanisms through which visceral afferent training might operate: (a) activity-dependent neuronal development, (b) periodic signal modeling, and (c) oscillatory network coordination.

Increasing the Construct Validity of Computational Phenotypes of Mental Illness Through Active Inference and Brain Imaging

After more than 30 years since its inception, the utility of brain imaging for understanding and diagnosing mental illnesses is in doubt, receiving well-grounded criticisms from clinical practitioners. Symptom-based correlational approaches have struggled to provide psychiatry with reliable brain-imaging metrics. However, the emergence of computational psychiatry has paved a new path not only for understanding the psychopathology of mental illness but also to provide practical tools for clinical practice in terms of computational metrics, specifically computational phenotypes. However, these phenotypes still lack sufficient test-retest reliability. In this review, we describe recent works revealing that mind and brain-related computational phenotypes show structural (not random) variation over time, longitudinal changes. Furthermore, we show that these findings suggest that understanding the causes of these changes will improve the construct validity of the phenotypes with an ensuing increase in test-retest reliability. We propose that the active inference framework offers a general-purpose approach for causally understanding these longitudinal changes by incorporating brain imaging as observations within partially observable Markov decision processes.

Learning dynamic cognitive map with autonomous navigation

Inspired by animal navigation strategies, we introduce a novel computational model to navigate and map a space rooted in biologically inspired principles. Animals exhibit extraordinary navigation prowess, harnessing memory, imagination, and strategic decision-making to traverse complex and aliased environments adeptly. Our model aims to replicate these capabilities by incorporating a dynamically expanding cognitive map over predicted poses within an active inference framework, enhancing our agent's generative model plasticity to novelty and environmental changes. Through structure learning and active inference navigation, our model demonstrates efficient exploration and exploitation, dynamically expanding its model capacity in response to anticipated novel un-visited locations and updating the map given new evidence contradicting previous beliefs. Comparative analyses in mini-grid environments with the clone-structured cognitive graph model (CSCG), which shares similar objectives, highlight our model's ability to rapidly learn environmental structures within a single episode, with minimal navigation overlap. Our model achieves this without prior knowledge of observation and world dimensions, underscoring its robustness and efficacy in navigating intricate environments.

Mechanisms of brain self-regulation: psychological factors, mechanistic models and neural substrates

While neurofeedback represents a promising tool for neuroscience and a brain self-regulation approach to psychological rehabilitation, the field faces several problems and challenges. Current research has shown great variability and even failure among human participants in learning to self-regulate target features of brain activity with neurofeedback. A better understanding of cognitive mechanisms, psychological factors and neural substrates underlying self-regulation might help improve neurofeedback's scientific and clinical practices. This article reviews the current understanding of the neural mechanisms of brain self-regulation by drawing on findings from human and animal studies in neurofeedback, brain-computer/machine interfaces and neuroprosthetics. In this article, we look closer at the following topics: cognitive processes and psychophysiological factors affecting self-regulation, theoretical models and neural substrates underlying self-regulation, and finally, we provide an outlook on the outstanding gaps in knowledge and technical challenges. This article is part of the theme issue 'Neurofeedback: new territories and neurocognitive mechanisms of endogenous neuromodulation'.

Associative Learning and Active Inference

Associative learning is a behavioral phenomenon in which individuals develop connections between stimuli or events based on their co-occurrence. Initially studied by Pavlov in his conditioning experiments, the fundamental principles of learning have been expanded on through the discovery of a wide range of learning phenomena. Computational models have been developed based on the concept of minimizing reward prediction errors. The Rescorla-Wagner model, in particular, is a well-known model that has greatly influenced the field of reinforcement learning. However, the simplicity of these models restricts their ability to fully explain the diverse range of behavioral phenomena associated with learning. In this study, we adopt the free energy principle, which suggests that living systems strive to minimize surprise or uncertainty under their internal models of the world. We consider the learning process as the minimization of free energy and investigate its relationship with the Rescorla-Wagner model, focusing on the informational aspects of learning, different types of surprise, and prediction errors based on beliefs and values. Furthermore, we explore how well-known behavioral phenomena such as blocking, overshadowing, and latent inhibition can be modeled within the active inference framework. We accomplish this by using the informational and novelty aspects of attention, which share similar ideas proposed by seemingly contradictory models such as Mackintosh and Pearce-Hall models. Thus, we demonstrate that the free energy principle, as a theoretical framework derived from first principles, can integrate the ideas and models of associative learning proposed based on empirical experiments and serve as a framework for a better understanding of the computational processes behind associative learning in the brain.

Bayesian Mechanics of Synaptic Learning Under the Free-Energy Principle

The brain is a biological system comprising nerve cells and orchestrates its embodied agent's perception, behavior, and learning in dynamic environments. The free-energy principle (FEP) advocated by Karl Friston explicates the local, recurrent, and self-supervised cognitive dynamics of the brain's higher-order functions. In this study, we continue to refine the FEP through a physics-guided formulation; specifically, we apply our theory to synaptic learning by considering it an inference problem under the FEP and derive the governing equations, called Bayesian mechanics. Our study uncovers how the brain infers weight changes and postsynaptic activity, conditioned on the presynaptic input, by deploying generative models of the likelihood and prior belief. Consequently, we exemplify the synaptic efficacy in the brain with a simple model; in particular, we illustrate that the brain organizes an optimal trajectory in neural phase space during synaptic learning in continuous time, which variationally minimizes synaptic surprisal.

A hierarchical active inference model of spatial alternation tasks and the hippocampal-prefrontal circuit

Cognitive problem-solving benefits from cognitive maps aiding navigation and planning. Physical space navigation involves hippocampal (HC) allocentric codes, while abstract task space engages medial prefrontal cortex (mPFC) task-specific codes. Previous studies show that challenging tasks, like spatial alternation, require integrating these two types of maps. The disruption of the HC-mPFC circuit impairs performance. We propose a hierarchical active inference model clarifying how this circuit solves spatial interaction tasks by bridging physical and task-space maps. Simulations demonstrate that the model's dual layers develop effective cognitive maps for physical and task space. The model solves spatial alternation tasks through reciprocal interactions between the two layers. Disrupting its communication impairs decision-making, which is consistent with empirical evidence. Additionally, the model adapts to switching between multiple alternation rules, providing a mechanistic explanation of how the HC-mPFC circuit supports spatial alternation tasks and the effects of disruption.

Neurophysiological principles underlying predictive coding during dynamic perception-action integration

A major concept in cognitive neuroscience is that brains are "prediction machines". Yet, conceptual frameworks on how perception and action become integrated still lack the concept of predictability and it is unclear how neural processes may implement predictive coding during dynamic perception-action integration. We show that distinct neurophysiological mechanisms of nonlinearly directed connectivities in the theta and alpha band between cortical structures underlie these processes. During the integration of perception and motor codes, especially theta band activity in the insular cortex and temporo-hippocampal structures is modulated by the predictability of upcoming information. Here, the insular cortex seems to guide processes. Conversely, the retrieval of such integrated perception-action codes during actions heavily relies on alpha band activity. Here, directed top-down influence of alpha band activity from inferior frontal structures on insular and temporo-hippocampal structures is key. This suggests that these top-down effects reflect attentional shielding of retrieval processes operating in the same neuroanatomical structures previously involved in the integration of perceptual and motor codes. Through neurophysiology, the present study connects predictive coding mechanisms with frameworks specifying the dynamic integration of perception and action.

Slow but flexible or fast but rigid? Discrete and continuous processes compared

A tradeoff exists when dealing with complex tasks composed of multiple steps. High-level cognitive processes can find the best sequence of actions to achieve a goal in uncertain environments, but they are slow and require significant computational demand. In contrast, lower-level processing allows reacting to environmental stimuli rapidly, but with limited capacity to determine optimal actions or to replan when expectations are not met. Through reiteration of the same task, biological organisms find the optimal tradeoff: from action primitives, composite trajectories gradually emerge by creating task-specific neural structures. The two frameworks of active inference - a recent brain paradigm that views action and perception as subject to the same free energy minimization imperative - well capture high-level and low-level processes of human behavior, but how task specialization occurs in these terms is still unclear. In this study, we compare two strategies on a dynamic pick-and-place task: a hybrid (discrete-continuous) model with planning capabilities and a continuous-only model with fixed transitions. Both models rely on a hierarchical (intrinsic and extrinsic) structure, well suited for defining reaching and grasping movements, respectively. Our results show that continuous-only models perform better and with minimal resource expenditure but at the cost of less flexibility. Finally, we propose how discrete actions might lead to continuous attractors and compare the two frameworks with different motor learning phases, laying the foundations for further studies on bio-inspired task adaptation.

Impressions about harm are formed rapidly and then refined, modulated by serotonin

Attributing motives to others is a crucial aspect of mentalizing, can be biased by prejudice, and is affected by common psychiatric disorders. It is therefore important to understand in depth the mechanisms underpinning it. Toward improving models of mentalizing motives, we hypothesized that people quickly infer whether other's motives are likely beneficial or detrimental, then refine their judgment (classify-refine). To test this, we used a modified Dictator game, a game theoretic task, where participants judged the likelihood of intent to harm vs. self-interest in economic decisions. Toward testing the role of serotonin in judgments of intent to harm, we delivered the task in a week-long, placebo vs. citalopram study. Computational model comparison provided clear evidence for the superiority of classify-refine models over traditional ones, strongly supporting the central hypothesis. Further, while citalopram helped refine attributions about motives through learning, it did not induce more positive initial inferences about others' motives. Finally, model comparison indicated a minimal role for racial bias within economic decisions for the large majority of our sample. Overall, these results support a proposal that classify-refine social cognition is adaptive, although relevant mechanisms of serotonergic antidepressant action will need to be studied over longer time spans.

Active Inference in Psychology and Psychiatry: Progress to Date?

The free energy principle is a formal theory of adaptive self-organising systems that emerged from statistical thermodynamics, machine learning and theoretical neuroscience and has since been translated into biologically plausible 'process theories' of cognition and behaviour, which fall under the banner of 'active inference'. Despite the promise this theory holds for theorising, research and practical applications in psychology and psychiatry, its impact on these disciplines has only now begun to bear fruit. The aim of this treatment is to consider the extent to which active inference has informed theoretical progress in psychology, before exploring its contributions to our understanding and treatment of psychopathology. Despite facing persistent translational obstacles, progress suggests that active inference has the potential to become a new paradigm that promises to unite psychology's subdisciplines, while readily incorporating the traditionally competing paradigms of evolutionary and developmental psychology. To date, however, progress towards this end has been slow. Meanwhile, the main outstanding question is whether this theory will make a positive difference through applications in clinical psychology, and its sister discipline of psychiatry.

Psilocybin increases optimistic engagement over time: computational modelling of behaviour in rats

Psilocybin has shown promise as a novel pharmacological intervention for treatment of depression, where post-acute effects of psilocybin treatment have been associated with increased positive mood and decreased pessimism. Although psilocybin is proving to be effective in clinical trials for treatment of psychiatric disorders, the information processing mechanisms affected by psilocybin are not well understood. Here, we fit active inference and reinforcement learning computational models to a novel two-armed bandit reversal learning task capable of capturing engagement behaviour in rats. The model revealed that after receiving psilocybin, rats achieve more rewards through increased task engagement, mediated by modification of forgetting rates and reduced loss aversion. These findings suggest that psilocybin may afford an optimism bias that arises through altered belief updating, with translational potential for clinical populations characterised by lack of optimism.

Making the Thermodynamic Cost of Active Inference Explicit

When describing Active Inference Agents (AIAs), the term "energy" can have two distinct meanings. One is the energy that is utilized by the AIA (e.g., electrical energy or chemical energy). The second meaning is so-called Variational Free Energy (VFE), a statistical quantity which provides an upper bound on surprisal. In this paper, we develop an account of the former quantity-the Thermodynamic Free Energy (TFE)-and its relationship with the latter. We highlight the necessary tradeoffs between these two in a generic, quantum information-theoretic formulation, and the macroscopic consequences of those tradeoffs for the ways that organisms approach their environments. By making this tradeoff explicit, we provide a theoretical basis for the different metabolic strategies that organisms from plants to predators use to survive.

Active Inference for Learning and Development in Embodied Neuromorphic Agents

Taking inspiration from humans can help catalyse embodied AI solutions for important real-world applications. Current human-inspired tools include neuromorphic systems and the developmental approach to learning. However, this developmental neurorobotics approach is currently lacking important frameworks for human-like computation and learning. We propose that human-like computation is inherently embodied, with its interface to the world being neuromorphic, and its learning processes operating across different timescales. These constraints necessitate a unified framework: active inference, underpinned by the free energy principle (FEP). Herein, we describe theoretical and empirical support for leveraging this framework in embodied neuromorphic agents with autonomous mental development. We additionally outline current implementation approaches (including toolboxes) and challenges, and we provide suggestions for next steps to catalyse this important field.

Joint modeling of choices and reaction times based on Bayesian contextual behavioral control

In cognitive neuroscience and psychology, reaction times are an important behavioral measure. However, in instrumental learning and goal-directed decision making experiments, findings often rely only on choice probabilities from a value-based model, instead of reaction times. Recent advancements have shown that it is possible to connect value-based decision models with reaction time models. However, typically these models do not provide an integrated account of both value-based choices and reaction times, but simply link two types of models. Here, we propose a novel integrative joint model of both choices and reaction times by combining a computational account of Bayesian sequential decision making with a sampling procedure. This allows us to describe how internal uncertainty in the planning process shapes reaction time distributions. Specifically, we use a recent context-specific Bayesian forward planning model which we extend by a Markov chain Monte Carlo (MCMC) sampler to obtain both choices and reaction times. As we will show this makes the sampler an integral part of the decision making process and enables us to reproduce, using simulations, well-known experimental findings in value based-decision making as well as classical inhibition and switching tasks. Specifically, we use the proposed model to explain both choice behavior and reaction times in instrumental learning and automatized behavior, in the Eriksen flanker task and in task switching. These findings show that the proposed joint behavioral model may describe common underlying processes in these different decision making paradigms.

The memory systems of the human brain and generative artificial intelligence

Generative Artificial Intelligence foundation models (for example Generative Pre-trained Transformer - GPT - models) can generate the next token given a sequence of tokens. How can this 'generative AI' be compared with the 'real' intelligence of the human brain, when for example a human generates a whole memory in response to an incomplete retrieval cue, and then generates further prospective thoughts? Here these two types of generative intelligence, artificial in machines and real in the human brain are compared, and it is shown how when whole memories are generated by hippocampal recall in response to an incomplete retrieval cue, what the human brain computes, and how it computes it, are very different from generative AI. Key differences are the use of local associative learning rules in the hippocampal memory system, and of non-local backpropagation of error learning in AI. Indeed, it is argued that the whole operation of the human brain is performed computationally very differently to what is implemented in generative AI. Moreover, it is emphasized that the primate including human hippocampal system includes computations about spatial view and where objects and people are in scenes, whereas in rodents the emphasis is on place cells and path integration by movements between places. This comparison with generative memory and processing in the human brain has interesting implications for the further development of generative AI and for neuroscience research.

Minimum Information Variability in Linear Langevin Systems via Model Predictive Control

Controlling the time evolution of a probability distribution that describes the dynamics of a given complex system is a challenging problem. Achieving success in this endeavour will benefit multiple practical scenarios, e.g., controlling mesoscopic systems. Here, we propose a control approach blending the model predictive control technique with insights from information geometry theory. Focusing on linear Langevin systems, we use model predictive control online optimisation capabilities to determine the system inputs that minimise deviations from the geodesic of the information length over time, ensuring dynamics with minimum "geometric information variability". We validate our methodology through numerical experimentation on the Ornstein-Uhlenbeck process and Kramers equation, demonstrating its feasibility. Furthermore, in the context of the Ornstein-Uhlenbeck process, we analyse the impact on the entropy production and entropy rate, providing a physical understanding of the effects of minimum information variability control.

A social inference model of idealization and devaluation

People often form polarized beliefs, imbuing objects (e.g., themselves or others) with unambiguously positive or negative qualities. In clinical settings, this is referred to as dichotomous thinking or "splitting" and is a feature of several psychiatric disorders. Here, we introduce a Bayesian model of splitting that parameterizes a tendency to rigidly categorize objects as either entirely "Bad" or "Good," rather than to flexibly learn dispositions along a continuous scale. Distinct from the previous descriptive theories, the model makes quantitative predictions about how dichotomous beliefs emerge and are updated in light of new information. Specifically, the model addresses how splitting is context-dependent, yet exhibits stability across time. A key model feature is that phases of devaluation and/or idealization are consolidated by rationally attributing counter-evidence to external factors. For example, when another person is idealized, their less-than-perfect behavior is attributed to unfavorable external circumstances. However, sufficient counter-evidence can trigger switches of polarity, producing bistable dynamics. We show that the model can be fitted to empirical data, to measure individual susceptibility to relational instability. For example, we find that a latent categorical belief that others are "Good" accounts for less changeable, and more certain, character impressions of benevolent as opposed to malevolent others among healthy participants. By comparison, character impressions made by participants with borderline personality disorder reveal significantly higher and more symmetric splitting. The generative framework proposed invites applications for modeling oscillatory relational and affective dynamics in psychotherapeutic contexts. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Clere Smithe T, Friedman DA, Friston K, Ramstead MJD. Shared Protentions in Multi-Agent Active Inference

In this paper, we unite concepts from Husserlian phenomenology, the active inference framework in theoretical biology, and category theory in mathematics to develop a comprehensive framework for understanding social action premised on shared goals. We begin with an overview of Husserlian phenomenology, focusing on aspects of inner time-consciousness, namely, retention, primal impression, and protention. We then review active inference as a formal approach to modeling agent behavior based on variational (approximate Bayesian) inference. Expanding upon Husserl's model of time consciousness, we consider collective goal-directed behavior, emphasizing shared protentions among agents and their connection to the shared generative models of active inference. This integrated framework aims to formalize shared goals in terms of shared protentions, and thereby shed light on the emergence of group intentionality. Building on this foundation, we incorporate mathematical tools from category theory, in particular, sheaf and topos theory, to furnish a mathematical image of individual and group interactions within a stochastic environment. Specifically, we employ morphisms between polynomial representations of individual agent models, allowing predictions not only of their own behaviors but also those of other agents and environmental responses. Sheaf and topos theory facilitates the construction of coherent agent worldviews and provides a way of representing consensus or shared understanding. We explore the emergence of shared protentions, bridging the phenomenology of temporal structure, multi-agent active inference systems, and category theory. Shared protentions are highlighted as pivotal for coordination and achieving common objectives. We conclude by acknowledging the intricacies stemming from stochastic systems and uncertainties in realizing shared goals.

Active Inference and Social Actors: Towards a Neuro-Bio-Social Theory of Brains and Bodies in Their Worlds

Although research including biological concepts and variables has gained more prominence in sociology, progress assimilating the organ of experience, the brain, has been theoretically and technically challenging. Formal uptake and assimilation have thus been slow. Within psychology and neuroscience, the traditional brain, which has made brief appearances in sociological research, is a "bottom-up" processor in which sensory signals are passed up the neural hierarchy where they are eventually cognitively and emotionally processed, after which actions and responses are generated. In this paper, we introduce the Active Inference Framework (AIF), which casts the brain as a Bayesian "inference engine" that tests its "top-down" predictive models against "bottom-up" sensory error streams in its attempts to resolve uncertainty and make the world more predictable. After assembling and presenting key concepts in the AIF, we describe an integrated neuro-bio-social model that prioritizes the microsociological assertion that the scene of action is the situation, wherein brains enculturate. Through such social dynamics, enculturated brains share models of the world with one another, enabling collective realities that disclose the actions afforded in those times and places. We conclude by discussing this neuro-bio-social model within the context of exemplar sociological research areas, including the sociology of stress and health, the sociology of emotions, and cognitive cultural sociology, all areas where the brain has received some degree of recognition and incorporation. In each case, sociological insights that do not fit naturally with the traditional brain model emerge intuitively from the predictive AIF model, further underscoring the interconnections and interdependencies between these areas, while also providing a foundation for a probabilistic sociology.

The empirical status of predictive coding and active inference

Research on predictive processing models has focused largely on two specific algorithmic theories: Predictive Coding for perception and Active Inference for decision-making. While these interconnected theories possess broad explanatory potential, they have only recently begun to receive direct empirical evaluation. Here, we review recent studies of Predictive Coding and Active Inference with a focus on evaluating the degree to which they are empirically supported. For Predictive Coding, we find that existing empirical evidence offers modest support. However, some positive results can also be explained by alternative feedforward (e.g., feature detection-based) models. For Active Inference, most empirical studies have focused on fitting these models to behavior as a means of identifying and explaining individual or group differences. While Active Inference models tend to explain behavioral data reasonably well, there has not been a focus on testing empirical validity of active inference theory per se, which would require formal comparison to other models (e.g., non-Bayesian or model-free reinforcement learning models). This review suggests that, while promising, a number of specific research directions are still necessary to evaluate the empirical adequacy and explanatory power of these algorithms.

Cultivating creativity: predictive brains and the enlightened room problem

How can one conciliate the claim that humans are uncertainty minimizing systems that seek to navigate predictable and familiar environments with the claim that humans can be creative? We call this the Enlightened Room Problem (ERP). The solution, we suggest, lies not (or not only) in the error-minimizing brain but in the environment itself. Creativity emerges from various degrees of interplay between predictive brains and changing environments: ones that repeatedly move the goalposts for our own error-minimizing machinery. By (co)constructing these challenging worlds, we effectively alter and expand the space within which our own prediction engines operate, and that function as 'exploration bubbles' that enable information seeking, uncertainty minimizing minds to penetrate deeper and deeper into artistic, scientific and engineering space. In what follows, we offer a proof of principle for this kind of environmentally led cognitive expansion. This article is part of the theme issue 'Art, aesthetics and predictive processing: theoretical and empirical perspectives'.

A model of time-varying music engagement

The current paper offers a model of time-varying music engagement, defined as changes in curiosity, attention and positive valence, as music unfolds over time. First, we present research (including new data) showing that listeners tend to allocate attention to music in a manner that is guided by both features of the music and listeners' individual differences. Next, we review relevant predictive processing literature before using this body of work to inform our model. In brief, we propose that music engagement, over the course of an extended listening episode, may constitute several cycles of curiosity, attention and positive valence that are interspersed with moments of mind-wandering. Further, we suggest that refocusing on music after an episode of mind-wandering can be due to triggers in the music or, conversely, mental action that occurs when the listener realizes they are mind-wandering. Finally, we argue that factors that modulate both overall levels of music engagement and how it changes over time include music complexity, listener background and the listening context. Our paper highlights how music can be used to provide insights into the temporal dynamics of attention and into how curiosity might emerge in everyday contexts. This article is part of the theme issue 'Art, aesthetics and predictive processing: theoretical and empirical perspectives'.

From reinforcement learning to agency: Frameworks for understanding basal cognition

Organisms play, explore, and mimic those around them. Is there a purpose to this behavior? Are organisms just behaving, or are they trying to achieve goals? We believe this is a false dichotomy. To that end, to understand organisms, we attempt to unify two approaches for understanding complex agents, whether evolved or engineered. We argue that formalisms describing multiscale competencies and goal-directedness in biology (e.g., TAME), and reinforcement learning (RL), can be combined in a symbiotic framework. While RL has been largely focused on higher-level organisms and robots of high complexity, TAME is naturally capable of describing lower-level organisms and minimal agents as well. We propose several novel questions that come from using RL/TAME to understand biology as well as ones that come from using biology to formulate new theory in AI. We hope that the research programs proposed in this piece shape future efforts to understand biological organisms and also future efforts to build artificial agents.

Active learning impairments in substance use disorders when resolving the explore-exploit dilemma: A replication and extension of previous computational modeling results

BACKGROUND

Substance use disorders (SUDs) represent a major public health risk. Yet, our understanding of the mechanisms that maintain these disorders remains incomplete. In a recent computational modeling study, we found initial evidence that SUDs are associated with slower learning rates from negative outcomes and less value-sensitive choice (low "action precision"), which could help explain continued substance use despite harmful consequences.

METHODS

Here we aimed to replicate and extend these results in a pre-registered study with a new sample of 168 individuals with SUDs and 99 healthy comparisons (HCs). We performed the same computational modeling and group comparisons as in our prior report (doi: 10.1016/j.drugalcdep.2020.108208) to confirm previously observed effects. After completing all pre-registered replication analyses, we then combined the previous and current datasets (N = 468) to assess whether differences were transdiagnostic or driven by specific disorders.

RESULTS

Replicating prior results, SUDs showed slower learning rates for negative outcomes in both Bayesian and frequentist analyses (partial η2=.02). Previously observed differences in action precision were not confirmed. Learning rates for positive outcomes were also similar between groups. Logistic regressions including all computational parameters as predictors in the combined datasets could differentiate several specific disorders from HCs, but could not differentiate most disorders from each other.

CONCLUSIONS

These results provide robust evidence that individuals with SUDs adjust behavior more slowly in the face of negative outcomes than HCs. They also suggest this effect is common across several different SUDs. Future research should examine its neural basis and whether learning rates could represent a new treatment target or moderator of treatment outcome.

An Introduction to Predictive Processing Models of Perception and Decision-Making

The predictive processing framework includes a broad set of ideas, which might be articulated and developed in a variety of ways, concerning how the brain may leverage predictive models when implementing perception, cognition, decision-making, and motor control. This article provides an up-to-date introduction to the two most influential theories within this framework: predictive coding and active inference. The first half of the paper (Sections 2-5) reviews the evolution of predictive coding, from early ideas about efficient coding in the visual system to a more general model encompassing perception, cognition, and motor control. The theory is characterized in terms of the claims it makes at Marr's computational, algorithmic, and implementation levels of description, and the conceptual and mathematical connections between predictive coding, Bayesian inference, and variational free energy (a quantity jointly evaluating model accuracy and complexity) are explored. The second half of the paper (Sections 6-8) turns to recent theories of active inference. Like predictive coding, active inference models assume that perceptual and learning processes minimize variational free energy as a means of approximating Bayesian inference in a biologically plausible manner. However, these models focus primarily on planning and decision-making processes that predictive coding models were not developed to address. Under active inference, an agent evaluates potential plans (action sequences) based on their expected free energy (a quantity that combines anticipated reward and information gain). The agent is assumed to represent the world as a partially observable Markov decision process with discrete time and discrete states. Current research applications of active inference models are described, including a range of simulation work, as well as studies fitting models to empirical data. The paper concludes by considering future research directions that will be important for further development of both models.

A primer on Variational Laplace (VL)

This article details a scheme for approximate Bayesian inference, which has underpinned thousands of neuroimaging studies since its introduction 15 years ago. Variational Laplace (VL) provides a generic approach to fitting linear or non-linear models, which may be static or dynamic, returning a posterior probability density over the model parameters and an approximation of log model evidence, which enables Bayesian model comparison. VL applies variational Bayesian inference in conjunction with quadratic or Laplace approximations of the evidence lower bound (free energy). Importantly, update equations do not need to be derived for each model under consideration, providing a general method for fitting a broad class of models. This primer is intended for experimenters and modellers who may wish to fit models to data using variational Bayesian methods, without assuming previous experience of variational Bayes or machine learning. Accompanying code demonstrates how to fit different kinds of model using the reference implementation of the VL scheme in the open-source Statistical Parametric Mapping (SPM) software package. In addition, we provide a standalone software function that does not require SPM, in order to ease translation to other fields, together with detailed pseudocode. Finally, the supplementary materials provide worked derivations of the key equations.

Phi fluctuates with surprisal: An empirical pre-study for the synthesis of the free energy principle and integrated information theory

The Free Energy Principle (FEP) and Integrated Information Theory (IIT) are two ambitious theoretical approaches. The first aims to make a formal framework for describing self-organizing and life-like systems in general, and the second attempts a mathematical theory of conscious experience based on the intrinsic properties of a system. They are each concerned with complementary aspects of the properties of systems, one with life and behavior, the other with meaning and experience, so combining them has potential for scientific value. In this paper, we take a first step towards such a synthesis by expanding on the results of an earlier published evolutionary simulation study, which show a relationship between IIT-measures and fitness in differing complexities of tasks. We relate a basic information theoretic measure from the FEP, surprisal, to this result, finding that the surprisal of simulated agents' observations is inversely related to the general increase in fitness and integration over evolutionary time. Moreover, surprisal fluctuates together with IIT-based consciousness measures in within-trial time. This suggests that the consciousness measures used in IIT indirectly depend on the relation between the agent and the external world, and that it should therefore be possible to relate them to the theoretical concepts used in the FEP. Lastly, we suggest a future approach for investigating this relationship empirically.

An active inference model of hierarchical action understanding, learning and imitation

We advance a novel active inference model of the cognitive processing that underlies the acquisition of a hierarchical action repertoire and its use for observation, understanding and imitation. We illustrate the model in four simulations of a tennis learner who observes a teacher performing tennis shots, forms hierarchical representations of the observed actions, and imitates them. Our simulations show that the agent's oculomotor activity implements an active information sampling strategy that permits inferring the kinematic aspects of the observed movement, which lie at the lowest level of the action hierarchy. In turn, this low-level kinematic inference supports higher-level inferences about deeper aspects of the observed actions: proximal goals and intentions. Finally, the inferred action representations can steer imitative responses, but interfere with the execution of different actions. Our simulations show that hierarchical active inference provides a unified account of action observation, understanding, learning and imitation and helps explain the neurobiological underpinnings of visuomotor cognition, including the multiple routes for action understanding in the dorsal and ventral streams and mirror mechanisms.

Planning to Learn: A Novel Algorithm for Active Learning during Model-Based Planning

Active Inference is a recently developed framework for modeling decision processes under uncertainty. Over the last several years, empirical and theoretical work has begun to evaluate the strengths and weaknesses of this approach and how it might be extended and improved. One recent extension is the "sophisticated inference" (SI) algorithm, which improves performance on multi-step planning problems through a recursive decision tree search. However, little work to date has been done to compare SI to other established planning algorithms in reinforcement learning (RL). In addition, SI was developed with a focus on inference as opposed to learning. The present paper therefore has two aims. First, we compare performance of SI to Bayesian RL schemes designed to solve similar problems. Second, we present and compare an extension of SI - sophisticated learning (SL) - that more fully incorporates active learning during planning. SL maintains beliefs about how model parameters would change under the future observations expected under each policy. This allows a form of counterfactual retrospective inference in which the agent considers what could be learned from current or past observations given different future observations. To accomplish these aims, we make use of a novel, biologically inspired environment that requires an optimal balance between goal-seeking and active learning, and which was designed to highlight the problem structure for which SL offers a unique solution. This setup requires an agent to continually search an open environment for available (but changing) resources in the presence of competing affordances for information gain. Our simulations demonstrate that SL outperforms all other algorithms in this context - most notably, Bayes-adaptive RL and upper confidence bound (UCB) algorithms, which aim to solve multi-step planning problems using similar principles (i.e., directed exploration and counterfactual reasoning about belief updates given different possible actions/observations). These results provide added support for the utility of Active Inference in solving this class of biologically-relevant problems and offer added tools for testing hypotheses about human cognition.

The role of pharmacogenomics in precision psychiatry

The field of psychiatry is facing an important paradigm shift in the provision of clinical care and mental health service organization toward personalization and integration of multimodal data science. This approach, termed precision psychiatry, aims at identifying subgroups of patients more prone to the development of a certain phenotype, such as symptoms or severe mental disorders (risk detection), and/or to guide treatment selection. Pharmacogenomics and computational psychiatry are two fundamental tools of precision psychiatry, which have seen increasing levels of integration in clinical settings. Here we present a brief overview of these two applications of precision psychiatry in clinical settings.

Models of the Translation Process and the Free Energy Principle

Translation process research (TPR) has generated a large number of models that aim at explaining human translation processes. In this paper, I suggest an extension of the monitor model to incorporate aspects of relevance theory (RT) and to adopt the free energy principle (FEP) as a generative model to elucidate translational behaviour. The FEP-and its corollary, active inference-provide a general, mathematical framework to explain how organisms resist entropic erosion so as to remain within their phenotypic bounds. It posits that organisms reduce the gap between their expectations and observations by minimising a quantity called free energy. I map these concepts on the translation process and exemplify them with behavioural data. The analysis is based on the notion of translation units (TUs) which exhibit observable traces of the translator's epistemic and pragmatic engagement with their translation environment, (i.e., the text) that can be measured in terms of translation effort and effects. Sequences of TUs cluster into translation states (steady state, orientation, and hesitation). Drawing on active inference, sequences of translation states combine into translation policies that reduce expected free energy. I show how the notion of free energy is compatible with the concept of relevance, as developed in RT, and how essential concepts of the monitor model and RT can be formalised as deep temporal generative models that can be interpreted under a representationalist view, but also support a non-representationalist account.

Free energy and inference in living systems

Organisms are non-equilibrium, stationary systems self-organized via spontaneous symmetry breaking and undergoing metabolic cycles with broken detailed balance in the environment. The thermodynamic free-energy (FE) principle describes an organism's homeostasis as the regulation of biochemical work constrained by the physical FE cost. By contrast, recent research in neuroscience and theoretical biology explains a higher organism's homeostasis and allostasis as Bayesian inference facilitated by the informational FE. As an integrated approach to living systems, this study presents an FE minimization theory overarching the essential features of both the thermodynamic and neuroscientific FE principles. Our results reveal that the perception and action of animals result from active inference entailed by FE minimization in the brain, and the brain operates as a Schrödinger's machine conducting the neural mechanics of minimizing sensory uncertainty. A parsimonious model suggests that the Bayesian brain develops the optimal trajectories in neural manifolds and induces a dynamic bifurcation between neural attractors in the process of active inference.

Post-injury pain and behaviour: a control theory perspective

Injuries of various types occur commonly in the lives of humans and other animals and lead to a pattern of persistent pain and recuperative behaviour that allows safe and effective recovery. In this Perspective, we propose a control-theoretic framework to explain the adaptive processes in the brain that drive physiological post-injury behaviour. We set out an evolutionary and ethological view on how animals respond to injury, illustrating how the behavioural state associated with persistent pain and recuperation may be just as important as phasic pain in ensuring survival. Adopting a normative approach, we suggest that the brain implements a continuous optimal inference of the current state of injury from diverse sensory and physiological signals. This drives the various effector control mechanisms of behavioural homeostasis, which span the modulation of ongoing motivation and perception to drive rest and hyper-protective behaviours. However, an inherent problem with this is that these protective behaviours may partially obscure information about whether injury has resolved. Such information restriction may seed a tendency to aberrantly or persistently infer injury, and may thus promote the transition to pathological chronic pain states.

Elevated decision uncertainty and reduced avoidance drives in depression, anxiety and substance use disorders during approach-avoidance conflict: a replication study

BACKGROUND

Decision-making under approach-avoidance conflict (AAC; e.g., sacrificing quality of life to avoid feared outcomes) may be affected in multiple psychiatric disorders. Recently, we used a computational (active inference) model to characterize information processing differences during AAC in individuals with depression, anxiety and/or substance use disorders. Individuals with psychiatric disorders exhibited increased decision uncertainty (DU) and reduced sensitivity to unpleasant stimuli. This preregistered study aimed to determine the replicability of this processing dysfunction.

METHODS

A new sample of participants completed the AAC task. Individual-level computational parameter estimates, reflecting decision uncertainty and sensitivity to unpleasant stimuli ("emotion conflict"; EC), were obtained and compared between groups. Subsequent analyses combining the prior and current samples allowed assessment of narrower disorder categories.

RESULTS

The sample in the present study included 480 participants: 97 healthy controls, 175 individuals with substance use disorders and 208 individuals with depression and/or anxiety disorders. Individuals with substance use disorders showed higher DU and lower EC values than healthy controls. The EC values were lower in females, but not males, with depression and/or anxiety disorders than in healthy controls. However, the previously observed difference in DU between participants with depression and/or anxiety disorders and healthy controls did not replicate. Analyses of specific disorders in the combined samples indicated that effects were common across different substance use disorders and affective disorders.

LIMITATIONS

There were differences, although with small effect size, in age and baseline intellectual functioning between the previous and current sample, which may have affected replication of DU differences in participants with depression and/or anxiety disorders.

CONCLUSION

The now robust evidence base for these clinical group differences motivates specific questions that should be addressed in future research: can DU and EC become behavioural treatment targets, and can we identify neural substrates of DU and EC that could be used to measure severity of dysfunction or as neuromodulatory treatment targets?

Quantifying aberrant approach-avoidance conflict in psychopathology: A review of computational approaches

Making effective decisions during approach-avoidance conflict is critical in daily life. Aberrant decision-making during approach-avoidance conflict is evident in a range of psychological disorders, including anxiety, depression, trauma-related disorders, substance use disorders, and alcohol use disorders. To help clarify etiological pathways and reveal novel intervention targets, clinical research into decision-making is increasingly adopting a computational psychopathology approach. This approach uses mathematical models that can identify specific decision-making related processes that are altered in mental health disorders. In our review, we highlight foundational approach-avoidance conflict research, followed by more in-depth discussion of computational approaches that have been used to model behavior in these tasks. Specifically, we describe the computational models that have been applied to approach-avoidance conflict (e.g., drift-diffusion, active inference, and reinforcement learning models), and provide resources to guide clinical researchers who may be interested in applying computational modeling. Finally, we identify notable gaps in the current literature and potential future directions for computational approaches aimed at identifying mechanisms of approach-avoidance conflict in psychopathology.

Cross-Modality Evidence for Reduced Choice History Biases in Psychosis-Prone Individuals

OBJECTIVES

Predictive processing posits that perception emerges from inferential processes within a hierarchical cortical system. Alterations of these processes may result in psychotic experiences, such as hallucinations and delusions. Central to the predictive processing account of psychosis is the notion of aberrant weights attributed to prior information and sensory input. Based on the notion that previous perceptual choices represent a relevant source of prior information, we here asked whether the propensity towards psychotic experiences may be related to altered choice history biases in perceptual decision-making.

METHODS

We investigated the relationship between choice history biases in perceptual decision-making and psychosis proneness in the general population. Choice history biases and their adaptation to experimentally induced changes in stimulus serial dependencies were investigated in decision-making tasks with auditory (experiment 1) and visual (experiment 2) stimuli. We further explored a potential compensatory mechanism for reduced choice history biases by reliance on predictive cross-modal cues.

RESULTS

In line with our preregistered hypothesis, psychosis proneness was associated with decreased choice history biases in both experiments. This association is generalized across conditions with and without stimulus serial dependencies. We did not find consistent evidence for a compensatory reliance on cue information in psychosis-prone individuals across experiments.

CONCLUSIONS

Our results show reduced choice history biases in psychosis proneness. A compensatory mechanism between implicit choice history effects and explicit cue information is not supported unequivocally by our data.

Hierarchical predictive coding in distributed pain circuits

Predictive coding is a computational theory on describing how the brain perceives and acts, which has been widely adopted in sensory processing and motor control. Nociceptive and pain processing involves a large and distributed network of circuits. However, it is still unknown whether this distributed network is completely decentralized or requires networkwide coordination. Multiple lines of evidence from human and animal studies have suggested that the cingulate cortex and insula cortex (cingulate-insula network) are two major hubs in mediating information from sensory afferents and spinothalamic inputs, whereas subregions of cingulate and insula cortices have distinct projections and functional roles. In this mini-review, we propose an updated hierarchical predictive coding framework for pain perception and discuss its related computational, algorithmic, and implementation issues. We suggest active inference as a generalized predictive coding algorithm, and hierarchically organized traveling waves of independent neural oscillations as a plausible brain mechanism to integrate bottom-up and top-down information across distributed pain circuits.

Interpersonal attunement in social interactions: from collective psychophysiology to inter-personalized psychiatry and beyond

In this article, we analyse social interactions, drawing on diverse points of views, ranging from dialectics, second-person neuroscience and enactivism to dynamical systems, active inference and machine learning. To this end, we define interpersonal attunement as a set of multi-scale processes of building up and materializing social expectations-put simply, anticipating and interacting with others and ourselves. While cultivating and negotiating common ground, via communication and culture-building activities, are indispensable for the survival of the individual, the relevant multi-scale mechanisms have been largely considered in isolation. Here, collective psychophysiology, we argue, can lend itself to the fine-tuned analysis of social interactions, without neglecting the individual. On the other hand, an interpersonal mismatch of expectations can lead to a breakdown of communication and social isolation known to negatively affect mental health. In this regard, we review psychopathology in terms of interpersonal misattunement, conceptualizing psychiatric disorders as disorders of social interaction, to describe how individual mental health is inextricably linked to social interaction. By doing so, we foresee avenues for an inter-personalized psychiatry, which moves from a static spectrum of disorders to a dynamic relational space, focusing on how the multi-faceted processes of social interaction can help to promote mental health. This article is part of the theme issue 'Concepts in interaction: social engagement and inner experiences'.

Maximum Entropy Exploration in Contextual Bandits with Neural Networks and Energy Based Models

Contextual bandits can solve a huge range of real-world problems. However, current popular algorithms to solve them either rely on linear models or unreliable uncertainty estimation in non-linear models, which are required to deal with the exploration-exploitation trade-off. Inspired by theories of human cognition, we introduce novel techniques that use maximum entropy exploration, relying on neural networks to find optimal policies in settings with both continuous and discrete action spaces. We present two classes of models, one with neural networks as reward estimators, and the other with energy based models, which model the probability of obtaining an optimal reward given an action. We evaluate the performance of these models in static and dynamic contextual bandit simulation environments. We show that both techniques outperform standard baseline algorithms, such as NN HMC, NN Discrete, Upper Confidence Bound, and Thompson Sampling, where energy based models have the best overall performance. This provides practitioners with new techniques that perform well in static and dynamic settings, and are particularly well suited to non-linear scenarios with continuous action spaces.