Planning and navigation as active inference

Bayesian reinforcement learning for navigation planning in unknown environments

This study focuses on a rescue mission problem, particularly enabling agents/robots to navigate efficiently in unknown environments. Technological advances, including manufacturing, sensing, and communication systems, have raised interest in using robots or drones for rescue operations. Effective rescue operations require quick identification of changes in the environment and/or locating the victims/injuries as soon as possible. Several techniques have been developed in recent years for autonomy in rescue missions, including motion planning, adaptive control, and more recently, reinforcement learning techniques. These techniques rely on full knowledge of the environment or the availability of simulators that can represent real environments during rescue operations. However, in practice, agents might have little or no information about the environment or the number or locations of injuries, preventing/limiting the application of most existing techniques. This study provides a probabilistic/Bayesian representation of the unknown environment, which jointly models the stochasticity in the agent's navigation and the environment uncertainty into a vector called the belief state. This belief state allows offline learning of the optimal Bayesian policy in an unknown environment without the need for any real data/interactions, which guarantees taking actions that are optimal given all available information. To address the large size of belief space, deep reinforcement learning is developed for computing an approximate Bayesian planning policy. The numerical experiments using different maze problems demonstrate the high performance of the proposed policy.

Exploration-exploitation model of moth-inspired olfactory navigation

Navigation of male moths towards females during the mating search offers a unique perspective on the exploration-exploitation (EE) model in decision-making. This study uses the EE model to explain male moth pheromone-driven flight paths. Wind tunnel measurements and three-dimensional tracking using infrared cameras have been leveraged to gain insights into male moth behaviour. During the experiments in the wind tunnel, disturbance to the airflow has been added and the effect of increased fluctuations on moth flights has been analysed, in the context of the proposed EE model. The exploration and exploitation phases are separated using a genetic algorithm to the experimentally obtained dataset of moth three-dimensional trajectories. First, the exploration-to-exploitation rate (EER) increases with distance from the source of the female pheromone is demonstrated, which can be explained in the context of the EE model. Furthermore, our findings reveal a compelling relationship between EER and increased flow fluctuations near the pheromone source. Using an olfactory navigation simulation and our moth-inspired navigation model, the phenomenon where male moths exhibit an enhanced EER as turbulence levels increase is explained. This research extends our understanding of optimal navigation strategies based on general biological EE models and supports the development of bioinspired navigation algorithms.

Forgetting ourselves in flow: an active inference account of flow states and how we experience ourselves within them

Flow has been described as a state of optimal performance, experienced universally across a broad range of domains: from art to athletics, gaming to writing. However, its phenomenal characteristics can, at first glance, be puzzling. Firstly, individuals in flow supposedly report a loss of self-awareness, even though they perform in a manner which seems to evince their agency and skill. Secondly, flow states are felt to be effortless, despite the prerequisite complexity of the tasks that engender them. In this paper, we unpick these features of flow, as well as others, through the active inference framework, which posits that action and perception are forms of active Bayesian inference directed at sustained self-organisation; i.e., the minimisation of variational free energy. We propose that the phenomenology of flow is rooted in the deployment of high precision weight over (i) the expected sensory consequences of action and (ii) beliefs about how action will sequentially unfold. This computational mechanism thus draws the embodied cognitive system to minimise the ensuing (i.e., expected) free energy through the exploitation of the pragmatic affordances at hand. Furthermore, given the challenging dynamics the flow-inducing situation presents, attention must be wholly focussed on the unfolding task whilst counterfactual planning is restricted, leading to the attested loss of the sense of self-as-object. This involves the inhibition of both the sense of self as a temporally extended object and higher-order, meta-cognitive forms of self-conceptualisation. Nevertheless, we stress that self-awareness is not entirely lost in flow. Rather, it is pre-reflective and bodily. Our approach to bodily-action-centred phenomenology can be applied to similar facets of seemingly agentive experience beyond canonical flow states, providing insights into the mechanisms of so-called selfless experiences, embodied expertise and wellbeing.

On Predictive Planning and Counterfactual Learning in Active Inference

Given the rapid advancement of artificial intelligence, understanding the foundations of intelligent behaviour is increasingly important. Active inference, regarded as a general theory of behaviour, offers a principled approach to probing the basis of sophistication in planning and decision-making. This paper examines two decision-making schemes in active inference based on "planning" and "learning from experience". Furthermore, we also introduce a mixed model that navigates the data complexity trade-off between these strategies, leveraging the strengths of both to facilitate balanced decision-making. We evaluate our proposed model in a challenging grid-world scenario that requires adaptability from the agent. Additionally, our model provides the opportunity to analyse the evolution of various parameters, offering valuable insights and contributing to an explainable framework for intelligent decision-making.

Active Inference With Empathy Mechanism for Socially Behaved Artificial Agents in Diverse Situations

This article proposes a method for an artificial agent to behave in a social manner. Although defining proper social behavior is difficult because it differs from situation to situation, the agent following the proposed method adaptively behaves appropriately in each situation by empathizing with the surrounding others. The proposed method is achieved by incorporating empathy into active inference. We evaluated the proposed method regarding control of autonomous mobile robots in diverse situations. From the evaluation results, an agent controlled by the proposed method could behave more adaptively socially than an agent controlled by the standard active inference in the diverse situations. In the case of two agents, the agent controlled with the proposed method behaved in a social way that reduced the other agent's travel distance by 13.7% and increased the margin between the agents by 25.8%, even though it increased the agent's travel distance by 8.2%. Also, the agent controlled with the proposed method behaved more socially when it was surrounded by altruistic others but less socially when it was surrounded by selfish others.

Naturalistic reinforcement learning

Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans' ability to navigate complex, multidimensional real-world environments so successfully.

I like therefore I can, and I can therefore I like: the role of self-efficacy and affect in active inference of allostasis

Active inference (AIF) is a theory of the behavior of information-processing open dynamic systems. It describes them as generative models (GM) generating inferences on the causes of sensory input they receive from their environment. Based on these inferences, GMs generate predictions about sensory input. The discrepancy between a prediction and the actual input results in prediction error. GMs then execute action policies predicted to minimize the prediction error. The free-energy principle provides a rationale for AIF by stipulating that information-processing open systems must constantly minimize their free energy (through suppressing the cumulative prediction error) to avoid decay. The theory of homeostasis and allostasis has a similar logic. Homeostatic set points are expectations of living organisms. Discrepancies between set points and actual states generate stress. For optimal functioning, organisms avoid stress by preserving homeostasis. Theories of AIF and homeostasis have recently converged, with AIF providing a formal account for homeo- and allostasis. In this paper, we present bacterial chemotaxis as molecular AIF, where mutual constraints by extero- and interoception play an essential role in controlling bacterial behavior supporting homeostasis. Extending this insight to the brain, we propose a conceptual model of the brain homeostatic GM, in which we suggest partition of the brain GM into cognitive and physiological homeostatic GMs. We outline their mutual regulation as well as their integration based on the free-energy principle. From this analysis, affect and self-efficacy emerge as the main regulators of the cognitive homeostatic GM. We suggest fatigue and depression as target neurocognitive phenomena for studying the neural mechanisms of such regulation.

Spatial and Temporal Hierarchy for Autonomous Navigation Using Active Inference in Minigrid Environment

Robust evidence suggests that humans explore their environment using a combination of topological landmarks and coarse-grained path integration. This approach relies on identifiable environmental features (topological landmarks) in tandem with estimations of distance and direction (coarse-grained path integration) to construct cognitive maps of the surroundings. This cognitive map is believed to exhibit a hierarchical structure, allowing efficient planning when solving complex navigation tasks. Inspired by human behaviour, this paper presents a scalable hierarchical active inference model for autonomous navigation, exploration, and goal-oriented behaviour. The model uses visual observation and motion perception to combine curiosity-driven exploration with goal-oriented behaviour. Motion is planned using different levels of reasoning, i.e., from context to place to motion. This allows for efficient navigation in new spaces and rapid progress toward a target. By incorporating these human navigational strategies and their hierarchical representation of the environment, this model proposes a new solution for autonomous navigation and exploration. The approach is validated through simulations in a mini-grid environment.

From active affordance to active inference: vertical integration of cognition in the cerebral cortex through dual subcortical control systems

In previous papers, we proposed that the dorsal attention system's top-down control is regulated by the dorsal division of the limbic system, providing a feedforward or impulsive form of control generating expectancies during active inference. In contrast, we proposed that the ventral attention system is regulated by the ventral limbic division, regulating feedback constraints and error-correction for active inference within the neocortical hierarchy. Here, we propose that these forms of cognitive control reflect vertical integration of subcortical arousal control systems that evolved for specific forms of behavior control. The feedforward impetus to action is regulated by phasic arousal, mediated by lemnothalamic projections from the reticular activating system of the lower brainstem, and then elaborated by the hippocampus and dorsal limbic division. In contrast, feedback constraint-based on environmental requirements-is regulated by the tonic activation furnished by collothalamic projections from the midbrain arousal control centers, and then sustained and elaborated by the amygdala, basal ganglia, and ventral limbic division. In an evolutionary-developmental analysis, understanding these differing forms of active affordance-for arousal and motor control within the subcortical vertebrate neuraxis-may help explain the evolution of active inference regulating the cognition of expectancy and error-correction within the mammalian 6-layered neocortex.

Bistable perception, precision and neuromodulation

Bistable perception follows from observing a static, ambiguous, (visual) stimulus with two possible interpretations. Here, we present an active (Bayesian) inference account of bistable perception and posit that perceptual transitions between different interpretations (i.e. inferences) of the same stimulus ensue from specific eye movements that shift the focus to a different visual feature. Formally, these inferences are a consequence of precision control that determines how confident beliefs are and change the frequency with which one can perceive-and alternate between-two distinct percepts. We hypothesized that there are multiple, but distinct, ways in which precision modulation can interact to give rise to a similar frequency of bistable perception. We validated this using numerical simulations of the Necker cube paradigm and demonstrate the multiple routes that underwrite the frequency of perceptual alternation. Our results provide an (enactive) computational account of the intricate precision balance underwriting bistable perception. Importantly, these precision parameters can be considered the computational homologs of particular neurotransmitters-i.e. acetylcholine, noradrenaline, dopamine-that have been previously implicated in controlling bistable perception, providing a computational link between the neurochemistry and perception.

Self-motion perception and sequential decision-making: where are we heading?

To navigate and guide adaptive behaviour in a dynamic environment, animals must accurately estimate their own motion relative to the external world. This is a fundamentally multisensory process involving integration of visual, vestibular and kinesthetic inputs. Ideal observer models, paired with careful neurophysiological investigation, helped to reveal how visual and vestibular signals are combined to support perception of linear self-motion direction, or heading. Recent work has extended these findings by emphasizing the dimension of time, both with regard to stimulus dynamics and the trade-off between speed and accuracy. Both time and certainty-i.e. the degree of confidence in a multisensory decision-are essential to the ecological goals of the system: terminating a decision process is necessary for timely action, and predicting one's accuracy is critical for making multiple decisions in a sequence, as in navigation. Here, we summarize a leading model for multisensory decision-making, then show how the model can be extended to study confidence in heading discrimination. Lastly, we preview ongoing efforts to bridge self-motion perception and navigation per se, including closed-loop virtual reality and active self-motion. The design of unconstrained, ethologically inspired tasks, accompanied by large-scale neural recordings, raise promise for a deeper understanding of spatial perception and decision-making in the behaving animal. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

A Variational Synthesis of Evolutionary and Developmental Dynamics

This paper introduces a variational formulation of natural selection, paying special attention to the nature of 'things' and the way that different 'kinds' of 'things' are individuated from-and influence-each other. We use the Bayesian mechanics of particular partitions to understand how slow phylogenetic processes constrain-and are constrained by-fast, phenotypic processes. The main result is a formulation of adaptive fitness as a path integral of phenotypic fitness. Paths of least action, at the phenotypic and phylogenetic scales, can then be read as inference and learning processes, respectively. In this view, a phenotype actively infers the state of its econiche under a generative model, whose parameters are learned via natural (Bayesian model) selection. The ensuing variational synthesis features some unexpected aspects. Perhaps the most notable is that it is not possible to describe or model a population of conspecifics per se. Rather, it is necessary to consider populations of distinct natural kinds that influence each other. This paper is limited to a description of the mathematical apparatus and accompanying ideas. Subsequent work will use these methods for simulations and numerical analyses-and identify points of contact with related mathematical formulations of evolution.

Cognitive effort and active inference

This paper aims to integrate some key constructs in the cognitive neuroscience of cognitive control and executive function by formalising the notion of cognitive (or mental) effort in terms of active inference. To do so, we call upon a task used in neuropsychology to assess impulse inhibition-a Stroop task. In this task, participants must suppress the impulse to read a colour word and instead report the colour of the text of the word. The Stroop task is characteristically effortful, and we unpack a theory of mental effort in which, to perform this task accurately, participants must overcome prior beliefs about how they would normally act. However, our interest here is not in overt action, but in covert (mental) action. Mental actions change our beliefs but have no (direct) effect on the outside world-much like deploying covert attention. This account of effort as mental action lets us generate multimodal (choice, reaction time, and electrophysiological) data of the sort we might expect from a human participant engaging in this task. We analyse how parameters determining cognitive effort influence simulated responses and demonstrate that-when provided only with performance data-these parameters can be recovered, provided they are within a certain range.

Decoding reward-curiosity conflict in decision-making from irrational behaviors

Humans and animals are not always rational. They not only rationally exploit rewards but also explore an environment owing to their curiosity. However, the mechanism of such curiosity-driven irrational behavior is largely unknown. Here, we developed a decision-making model for a two-choice task based on the free energy principle, which is a theory integrating recognition and action selection. The model describes irrational behaviors depending on the curiosity level. We also proposed a machine learning method to decode temporal curiosity from behavioral data. By applying it to rat behavioral data, we found that the rat had negative curiosity, reflecting conservative selection sticking to more certain options and that the level of curiosity was upregulated by the expected future information obtained from an uncertain environment. Our decoding approach can be a fundamental tool for identifying the neural basis for reward-curiosity conflicts. Furthermore, it could be effective in diagnosing mental disorders.

Computational cross-species views of the hippocampal formation

The discovery of place cells and head direction cells in the hippocampal formation of freely foraging rodents has led to an emphasis of its role in encoding allocentric spatial relationships. In contrast, studies in head-fixed primates have additionally found representations of spatial views. We review recent experiments in freely moving monkeys that expand upon these findings and show that postural variables such as eye/head movements strongly influence neural activity in the hippocampal formation, suggesting that the function of the hippocampus depends on where the animal looks. We interpret these results in the light of recent studies in humans performing challenging navigation tasks which suggest that depending on the context, eye/head movements serve one of two roles-gathering information about the structure of the environment (active sensing) or externalizing the contents of internal beliefs/deliberation (embodied cognition). These findings prompt future experimental investigations into the information carried by signals flowing between the hippocampal formation and the brain regions controlling postural variables, and constitute a basis for updating computational theories of the hippocampal system to accommodate the influence of eye/head movements.

Modelling mood updating: a proof of principle study

BACKGROUND

Recent developments in computational psychiatry have led to the hypothesis that mood represents an expectation (prior belief) on the likely interoceptive consequences of action (i.e. emotion). This stems from ideas about how the brain navigates its external world by minimising an upper bound on surprisal (free energy) of sensory information and echoes developments in other perceptual domains.

AIMS

In this paper we aim to present a simple partial observable Markov decision process that models mood updating in response to stressful or non-stressful environmental fluctuations while seeking to minimise surprisal in relation to prior beliefs about the likely interoceptive signals experienced with specific actions (attenuating or amplifying stress and pleasure signals).

METHOD

We examine how, by altering these prior beliefs we can model mood updating in depression, mania and anxiety.

RESULTS

We discuss how these models provide a computational account of mood and its related psychopathology and relate it to previous research in reward processing.

CONCLUSIONS

Models such as this can provide hypotheses for experimental work and also open up the potential modelling of predicted disease trajectories in individual patients.

Small steps for mankind: Modeling the emergence of cumulative culture from joint active inference communication

Although the increase in the use of dynamical modeling in the literature on cultural evolution makes current models more mathematically sophisticated, these models have yet to be tested or validated. This paper provides a testable deep active inference formulation of social behavior and accompanying simulations of cumulative culture in two steps: First, we cast cultural transmission as a bi-directional process of communication that induces a generalized synchrony (operationalized as a particular convergence) between the belief states of interlocutors. Second, we cast social or cultural exchange as a process of active inference by equipping agents with the choice of who to engage in communication with. This induces trade-offs between confirmation of current beliefs and exploration of the social environment. We find that cumulative culture emerges from belief updating (i.e., active inference and learning) in the form of a joint minimization of uncertainty. The emergent cultural equilibria are characterized by a segregation into groups, whose belief systems are actively sustained by selective, uncertainty minimizing, dyadic exchanges. The nature of these equilibria depends sensitively on the precision afforded by various probabilistic mappings in each individual's generative model of their encultured niche.

A neural active inference model of perceptual-motor learning

The active inference framework (AIF) is a promising new computational framework grounded in contemporary neuroscience that can produce human-like behavior through reward-based learning. In this study, we test the ability for the AIF to capture the role of anticipation in the visual guidance of action in humans through the systematic investigation of a visual-motor task that has been well-explored-that of intercepting a target moving over a ground plane. Previous research demonstrated that humans performing this task resorted to anticipatory changes in speed intended to compensate for semi-predictable changes in target speed later in the approach. To capture this behavior, our proposed "neural" AIF agent uses artificial neural networks to select actions on the basis of a very short term prediction of the information about the task environment that these actions would reveal along with a long-term estimate of the resulting cumulative expected free energy. Systematic variation revealed that anticipatory behavior emerged only when required by limitations on the agent's movement capabilities, and only when the agent was able to estimate accumulated free energy over sufficiently long durations into the future. In addition, we present a novel formulation of the prior mapping function that maps a multi-dimensional world-state to a uni-dimensional distribution of free-energy/reward. Together, these results demonstrate the use of AIF as a plausible model of anticipatory visually guided behavior in humans.

Flexible intentions: An Active Inference theory

We present a normative computational theory of how the brain may support visually-guided goal-directed actions in dynamically changing environments. It extends the Active Inference theory of cortical processing according to which the brain maintains beliefs over the environmental state, and motor control signals try to fulfill the corresponding sensory predictions. We propose that the neural circuitry in the Posterior Parietal Cortex (PPC) compute flexible intentions-or motor plans from a belief over targets-to dynamically generate goal-directed actions, and we develop a computational formalization of this process. A proof-of-concept agent embodying visual and proprioceptive sensors and an actuated upper limb was tested on target-reaching tasks. The agent behaved correctly under various conditions, including static and dynamic targets, different sensory feedbacks, sensory precisions, intention gains, and movement policies; limit conditions were individuated, too. Active Inference driven by dynamic and flexible intentions can thus support goal-directed behavior in constantly changing environments, and the PPC might putatively host its core intention mechanism. More broadly, the study provides a normative computational basis for research on goal-directed behavior in end-to-end settings and further advances mechanistic theories of active biological systems.

Integrated world modeling theory expanded: Implications for the future of consciousness

Integrated world modeling theory (IWMT) is a synthetic theory of consciousness that uses the free energy principle and active inference (FEP-AI) framework to combine insights from integrated information theory (IIT) and global neuronal workspace theory (GNWT). Here, I first review philosophical principles and neural systems contributing to IWMT's integrative perspective. I then go on to describe predictive processing models of brains and their connections to machine learning architectures, with particular emphasis on autoencoders (perceptual and active inference), turbo-codes (establishment of shared latent spaces for multi-modal integration and inferential synergy), and graph neural networks (spatial and somatic modeling and control). Future directions for IIT and GNWT are considered by exploring ways in which modules and workspaces may be evaluated as both complexes of integrated information and arenas for iterated Bayesian model selection. Based on these considerations, I suggest novel ways in which integrated information might be estimated using concepts from probabilistic graphical models, flow networks, and game theory. Mechanistic and computational principles are also considered with respect to the ongoing debate between IIT and GNWT regarding the physical substrates of different kinds of conscious and unconscious phenomena. I further explore how these ideas might relate to the "Bayesian blur problem," or how it is that a seemingly discrete experience can be generated from probabilistic modeling, with some consideration of analogies from quantum mechanics as potentially revealing different varieties of inferential dynamics. I go on to describe potential means of addressing critiques of causal structure theories based on network unfolding, and the seeming absurdity of conscious expander graphs (without cybernetic symbol grounding). Finally, I discuss future directions for work centered on attentional selection and the evolutionary origins of consciousness as facilitated "unlimited associative learning." While not quite solving the Hard problem, this article expands on IWMT as a unifying model of consciousness and the potential future evolution of minds.

Structure learning enhances concept formation in synthetic Active Inference agents

Humans display astonishing skill in learning about the environment in which they operate. They assimilate a rich set of affordances and interrelations among different elements in particular contexts, and form flexible abstractions (i.e., concepts) that can be generalised and leveraged with ease. To capture these abilities, we present a deep hierarchical Active Inference model of goal-directed behaviour, and the accompanying belief update schemes implied by maximising model evidence. Using simulations, we elucidate the potential mechanisms that underlie and influence concept learning in a spatial foraging task. We show that the representations formed–as a result of foraging–reflect environmental structure in a way that is enhanced and nuanced by Bayesian model reduction, a special case of structure learning that typifies learning in the absence of new evidence. Synthetic agents learn associations and form concepts about environmental context and configuration as a result of inferential, parametric learning, and structure learning processes–three processes that can produce a diversity of beliefs and belief structures. Furthermore, the ensuing representations reflect symmetries for environments with identical configurations.

A fuzzy-oscillatory model of medial prefrontal cortex control function in spatial memory retrieval in human navigation function

Navigation can be broadly defined as the process of moving from an origin to a destination through path-planning. Previous research has shown that navigation is mainly related to the function of the medial temporal lobe (MTL), including the hippocampus (HPC), and medial prefrontal cortex (mPFC), which controls retrieval of the spatial memories from this region. In this study, we suggested a cognitive and computational model of human navigation with a focus on mutual interactions between the hippocampus (HPC) and the mPFC using the concept of synchrony. The Van-der-pol oscillator was used to model the synchronous process of receiving and processing “what stream” information. A fuzzy lookup table system was applied for modeling the controlling function of the mPFC in retrieving spatial information from the HPC. The effect of attention level was also included and simulated. The performance of the model was evaluated using information reported in previous experimental research. Due to the inherent stability of the proposed fuzzy-oscillatory model, it is less sensitive to the exact values of the initial conditions, and therefore, it is shown that it is consistent with the actual human performance in real environments. Analyzing the proposed cognitive and fuzzy-oscillatory computational model demonstrates that the model is able to reproduce certain cognitive and functional disturbances in navigation in related diseases such as Alzheimer’s disease (AD). We have shown that an increase in the bifurcation parameter of the Van-der-pol equation represents an increase in the low-frequency spectral power density and a decrease in the high-frequency spectral power as occurs in AD due to an increase in the amyloid plaques in the brain. These changes in the frequency characteristics of neuronal activity, in turn, lead to impaired recall and retrieval of landmarks information and learned routes upon encountering them. As a result, and because of the wrong frequency code being transmitted, the relevant set of rules in the mPFC is not activated, or another unrelated set will be activated, which leads to forgetfulness and erroneous decisions in routing and eventually losing the route in Alzheimer’s patients.

Revealing human sensitivity to a latent temporal structure of changes

Precisely timed behavior and accurate time perception plays a critical role in our everyday lives, as our wellbeing and even survival can depend on well-timed decisions. Although the temporal structure of the world around us is essential for human decision making, we know surprisingly little about how representation of temporal structure of our everyday environment impacts decision making. How does the representation of temporal structure affect our ability to generate well-timed decisions? Here we address this question by using a well-established dynamic probabilistic learning task. Using computational modeling, we found that human subjects' beliefs about temporal structure are reflected in their choices to either exploit their current knowledge or to explore novel options. The model-based analysis illustrates a large within-group and within-subject heterogeneity. To explain these results, we propose a normative model for how temporal structure is used in decision making, based on the semi-Markov formalism in the active inference framework. We discuss potential key applications of the presented approach to the fields of cognitive phenotyping and computational psychiatry.

Generalized Simultaneous Localization and Mapping (G-SLAM) as unification framework for natural and artificial intelligences: towards reverse engineering the hippocampal/entorhinal system and principles of high-level cognition

Simultaneous localization and mapping (SLAM) represents a fundamental problem for autonomous embodied systems, for which the hippocampal/entorhinal system (H/E-S) has been optimized over the course of evolution. We have developed a biologically-inspired SLAM architecture based on latent variable generative modeling within the Free Energy Principle and Active Inference (FEP-AI) framework, which affords flexible navigation and planning in mobile robots. We have primarily focused on attempting to reverse engineer H/E-S "design" properties, but here we consider ways in which SLAM principles from robotics may help us better understand nervous systems and emergent minds. After reviewing LatentSLAM and notable features of this control architecture, we consider how the H/E-S may realize these functional properties not only for physical navigation, but also with respect to high-level cognition understood as generalized simultaneous localization and mapping (G-SLAM). We focus on loop-closure, graph-relaxation, and node duplication as particularly impactful architectural features, suggesting these computational phenomena may contribute to understanding cognitive insight (as proto-causal-inference), accommodation (as integration into existing schemas), and assimilation (as category formation). All these operations can similarly be describable in terms of structure/category learning on multiple levels of abstraction. However, here we adopt an ecological rationality perspective, framing H/E-S functions as orchestrating SLAM processes within both concrete and abstract hypothesis spaces. In this navigation/search process, adaptive cognitive equilibration between assimilation and accommodation involves balancing tradeoffs between exploration and exploitation; this dynamic equilibrium may be near optimally realized in FEP-AI, wherein control systems governed by expected free energy objective functions naturally balance model simplicity and accuracy. With respect to structure learning, such a balance would involve constructing models and categories that are neither too inclusive nor exclusive. We propose these (generalized) SLAM phenomena may represent some of the most impactful sources of variation in cognition both within and between individuals, suggesting that modulators of H/E-S functioning may potentially illuminate their adaptive significances as fundamental cybernetic control parameters. Finally, we discuss how understanding H/E-S contributions to G-SLAM may provide a unifying framework for high-level cognition and its potential realization in artificial intelligences.

In the Body’s Eye: The computational anatomy of interoceptive inference

A growing body of evidence highlights the intricate linkage of exteroceptive perception to the rhythmic activity of the visceral body. In parallel, interoceptive inference theories of affective perception and self-consciousness are on the rise in cognitive science. However, thus far no formal theory has emerged to integrate these twin domains; instead, most extant work is conceptual in nature. Here, we introduce a formal model of cardiac active inference, which explains how ascending cardiac signals entrain exteroceptive sensory perception and uncertainty. Through simulated psychophysics, we reproduce the defensive startle reflex and commonly reported effects linking the cardiac cycle to affective behaviour. We further show that simulated ‘interoceptive lesions’ blunt affective expectations, induce psychosomatic hallucinations, and exacerbate biases in perceptual uncertainty. Through synthetic heart-rate variability analyses, we illustrate how the balance of arousal-priors and visceral prediction errors produces idiosyncratic patterns of physiological reactivity. Our model thus offers a roadmap for computationally phenotyping disordered brain-body interaction.

Understanding interactions between the brain and the body has become a topic of increased interest in computational neuroscience and psychiatry. A particular question here concerns how visceral, homeostatic rhythms such as the heart beat influence sensory, affective, and cognitive processing. To better understand these and other oscillatory brain-body interactions, we here introduce a novel computational model of interoceptive inference in which a synthetic agent’s perceptual beliefs are coupled to the rhythm of the heart. Our model both helps to explain emerging empirical data indicating that perceptual inference depends upon beat-to-beat heart rhythms, and can be used to better quantify intra- and inter-individual differences in heart-brain coupling. Using proof-of-principle simulations, we demonstrate how future empirical works could utilize our model to better understand and stratify disorders of interoception and brain-body interaction.

The role of intolerance of uncertainty when solving the exploration-exploitation dilemma

When making behavioral decisions, individuals need to balance between exploiting known options or exploring new ones. How individuals solve this exploration-exploitation dilemma (EED) is a key research question across psychology, leading to attempting to disentangle the cognitive mechanisms behind it. A potential predictive factor of performance in an EED is intolerance of uncertainty (IU), an individual difference factor referring to the extent to which uncertain situations are reported to be aversive. Here, we present the results of a series of exploratory analyses in which we tested the relationship between IU and performance in an EED task. For this, we compiled data from 3 experiments, in which participants received the opportunity to exploit different movements in order to avoid a painful stimulus and approach rewards. For decomposing performance in this task, we used different computational models previously employed in studies on the EED. Then, the parameters of the winning model were correlated with the scores of participants in the IU scale. Correlational and cluster analyses, within both frequentists and Bayesian frameworks, did not provide strong evidence for a relation between EED and IU, apart from the decay rate and the subscale "tendency to become paralyzed in the face of uncertainty". Given the theoretical relation between EED and IU, we propose research with different experimental paradigms.

Machines That Feel and Think: The Role of Affective Feelings and Mental Action in (Artificial) General Intelligence

What role do affective feelings (feelings/emotions/moods) play in adaptive behaviour? What are the implications of this for understanding and developing artificial general intelligence? Leading theoretical models of brain function are beginning to shed light on these questions. While artificial agents have excelled within narrowly circumscribed and specialised domains, domain-general intelligence has remained an elusive goal in artificial intelligence research. By contrast, humans and nonhuman animals are characterised by a capacity for flexible behaviour and general intelligence. In this article I argue that computational models of mental phenomena in predictive processing theories of the brain are starting to reveal the mechanisms underpinning domain-general intelligence in biological agents, and can inform the understanding and development of artificial general intelligence. I focus particularly on approaches to computational phenomenology in the active inference framework. Specifically, I argue that computational mechanisms of affective feelings in active inference-affective self-modelling-are revealing of how biological agents are able to achieve flexible behavioural repertoires and general intelligence. I argue that (i) affective self-modelling functions to "tune" organisms to the most tractable goals in the environmental context; and (ii) affective and agentic self-modelling is central to the capacity to perform mental actions in goal-directed imagination and creative cognition. I use this account as a basis to argue that general intelligence of the level and kind found in biological agents will likely require machines to be implemented with analogues of affective self-modelling.

Walking on a minefield: planning, remembering, and avoiding obstacles: preliminary findings

Travel planning (TP) is a kind of planning devoted to spatial orientation that is distinguishable from general planning (GP). It is crucial to reach a destination, since it allows to select the best route according to the environmental features (e.g., the one with little traffic or the safest). TP is also needed to avoid obstacles along the way and to put in place effective strategies to support navigation. TP involves several cognitive processes, such as visuo-spatial and topographic memory as well as other executive functions (i.e., general planning, cognitive flexibility, problem solving, and divergent thinking) and it is affected by internal factors (such as gender, cognitive strategies, age). Here, we focused on the effects of visuo-spatial (VSWM) and topographic (TWM) working memory on TP, using the Minefield Task (MFT), a new tool aimed at testing TP. We tested VSWM, TWM, GP, and TP in 44 college students. First, we checked for gender differences in all the tasks proposed and then assessed the relation among VSWM, TWM, GP, and TP. Results showed that even though gender difference could be found on TWM, GP, and TP, significative correlations emerged among TP, VSWM, and GP as well as a tendency to significance for VSWM and GP in the regression analyses. Though more evidence is needed, these results suggest that when a brand-new route is computed, GP and VSWM can be the most relevant processes, whereas topographic memory was less involved, probably because the MFT does not require to recall a route from memory. The implications of these results in clinical settings are discussed.

Extended active inference: Constructing predictive cognition beyond skulls

Cognitive niche construction is the process whereby organisms create and maintain cause-effect models of their niche as guides for fitness influencing behavior. Extended mind theory claims that cognitive processes extend beyond the brain to include predictable states of the world. Active inference and predictive processing in cognitive science assume that organisms embody predictive (i.e., generative) models of the world optimized by standard cognitive functions (e.g., perception, action, learning). This paper presents an active inference formulation that views cognitive niche construction as a cognitive function aimed at optimizing organisms' generative models. We call that process of optimization extended active inference.

From representations in predictive processing to degrees of representational features

Whilst the topic of representations is one of the key topics in philosophy of mind, it has only occasionally been noted that representations and representational features may be gradual. Apart from vague allusions, little has been said on what representational gradation amounts to and why it could be explanatorily useful. The aim of this paper is to provide a novel take on gradation of representational features within the neuroscientific framework of predictive processing. More specifically, we provide a gradual account of two features of structural representations: structural similarity and decoupling. We argue that structural similarity can be analysed in terms of two dimensions: number of preserved relations and state space granularity. Both dimensions can take on different values and hence render structural similarity gradual. We further argue that decoupling is gradual in two ways. First, we show that different brain areas are involved in decoupled cognitive processes to a greater or lesser degree depending on the cause (internal or external) of their activity. Second, and more importantly, we show that the degree of decoupling can be further regulated in some brain areas through precision weighting of prediction error. We lastly argue that gradation of decoupling (via precision weighting) and gradation of structural similarity (via state space granularity) are conducive to behavioural success.

An Active Inference Account of Skilled Anticipation in Sport: Using Computational Models to Formalise Theory and Generate New Hypotheses

Optimal performance in time-constrained and dynamically changing environments depends on making reliable predictions about future outcomes. In sporting tasks, performers have been found to employ multiple information sources to maximise the accuracy of their predictions, but questions remain about how different information sources are weighted and integrated to guide anticipation. In this paper, we outline how predictive processing approaches, and active inference in particular, provide a unifying account of perception and action that explains many of the prominent findings in the sports anticipation literature. Active inference proposes that perception and action are underpinned by the organism’s need to remain within certain stable states. To this end, decision making approximates Bayesian inference and actions are used to minimise future prediction errors during brain–body–environment interactions. Using a series of Bayesian neurocomputational models based on a partially observable Markov process, we demonstrate that key findings from the literature can be recreated from the first principles of active inference. In doing so, we formulate a number of novel and empirically falsifiable hypotheses about human anticipation capabilities that could guide future investigations in the field.

Eye movements reveal spatiotemporal dynamics of visually-informed planning in navigation

Goal-oriented navigation is widely understood to depend upon internal maps. Although this may be the case in many settings, humans tend to rely on vision in complex, unfamiliar environments. To study the nature of gaze during visually-guided navigation, we tasked humans to navigate to transiently visible goals in virtual mazes of varying levels of difficulty, observing that they took near-optimal trajectories in all arenas. By analyzing participants’ eye movements, we gained insights into how they performed visually-informed planning. The spatial distribution of gaze revealed that environmental complexity mediated a striking trade-off in the extent to which attention was directed towards two complimentary aspects of the world model: the reward location and task-relevant transitions. The temporal evolution of gaze revealed rapid, sequential prospection of the future path, evocative of neural replay. These findings suggest that the spatiotemporal characteristics of gaze during navigation are significantly shaped by the unique cognitive computations underlying real-world, sequential decision making.

Piloting the Update: The Use of Therapeutic Relationship for Change - A Free Energy Account

We apply the Free Energy Principle (FEP) to cognitive behavioral therapy (CBT). FEP describes the basic functioning of the brain as a predictive organ and states that any self-organizing system that is in equilibrium with its environment must minimize its free energy. Based on an internal model of the world and the self, predictions-so-called priors-are created, which are matched with the information input. The sum of prediction errors corresponds to the Free Energy, which must be minimized. Internal models can be identified with the cognitive-affective schemas of the individual that has become dysfunctional in patients. The role of CBT in this picture is to help the patient update her/his priors. They have evolved in learning history and no longer provide adaptive predictions. We discuss the process of updating in terms of the exploration-exploitation dilemma. This consists of the extent to which one relies on what one already has, i.e., whether one continues to maintain and "exploit" one's previous priors ("better safe than sorry") or whether one does explore new data that lead to an update of priors. Questioning previous priors triggers stress, which is associated with increases in Free Energy in short term. The role of therapeutic relationship is to buffer this increase in Free Energy, thereby increasing the level of perceived safety. The therapeutic relationship is represented in a dual model of affective alliance and goal attainment alliance and is aligned with FEP. Both forms of alliance support exploration and updating of priors. All aspects are illustrated with the help of a clinical case example.

Integrating Evolutionary, Cultural, and Computational Psychiatry: A Multilevel Systemic Approach

This paper proposes an integrative perspective on evolutionary, cultural and computational approaches to psychiatry. These three approaches attempt to frame mental disorders as multiscale entities and offer modes of explanations and modeling strategies that can inform clinical practice. Although each of these perspectives involves systemic thinking, each is limited in its ability to address the complex developmental trajectories and larger social systemic interactions that lead to mental disorders. Inspired by computational modeling in theoretical biology, this paper aims to integrate the modes of explanation offered by evolutionary, cultural and computational psychiatry in a multilevel systemic perspective. We apply the resulting Evolutionary, Cultural and Computational (ECC) model to Major Depressive Disorder (MDD) to illustrate how this integrative approach can guide research and practice in psychiatry.

A step-by-step tutorial on active inference and its application to empirical data

The active inference framework, and in particular its recent formulation as a partially observable Markov decision process (POMDP), has gained increasing popularity in recent years as a useful approach for modeling neurocognitive processes. This framework is highly general and flexible in its ability to be customized to model any cognitive process, as well as simulate predicted neuronal responses based on its accompanying neural process theory. It also affords both simulation experiments for proof of principle and behavioral modeling for empirical studies. However, there are limited resources that explain how to build and run these models in practice, which limits their widespread use. Most introductions assume a technical background in programming, mathematics, and machine learning. In this paper we offer a step-by-step tutorial on how to build POMDPs, run simulations using standard MATLAB routines, and fit these models to empirical data. We assume a minimal background in programming and mathematics, thoroughly explain all equations, and provide exemplar scripts that can be customized for both theoretical and empirical studies. Our goal is to provide the reader with the requisite background knowledge and practical tools to apply active inference to their own research. We also provide optional technical sections and multiple appendices, which offer the interested reader additional technical details. This tutorial should provide the reader with all the tools necessary to use these models and to follow emerging advances in active inference research.

Effects of Pulsed-Wave Chromotherapy and Guided Relaxation on the Theta-Alpha Oscillation During Arrest Reaction

The search for the best wellness practice has promoted the development of devices integrating different technologies and guided meditation. However, the final effects on the electrical activity of the brain remain relatively sparse. Here, we have analyzed of the alpha and theta electroencephalographic oscillations during the realization of the arrest reaction (AR; eyes close/eyes open transition) when a chromotherapy session performed in a dedicated room [Rebalance (RB) device], with an ergonomic bed integrating pulsed-wave light (PWL) stimulation, guided breathing, and body scan exercises. We demonstrated that the PWL induced an evoked-related potential characterized by the N2-P3 components maximally recorded on the fronto-central areas and accompanied by an event-related synchronization (ERS) of the delta–theta–alpha oscillations. The power of the alpha and theta oscillations was analyzed during repeated ARs testing realized along with the whole RB session. We showed that the power of the alpha and theta oscillations was significantly increased during the session in comparison to their values recorded before. Of the 14 participants, 11 and 6 showed a significant power increase of the alpha and theta oscillations, respectively. These increased powers were not observed in two different control groups (n = 28) who stayed passively outside or inside the RB room but without any type of stimulation. These preliminary results suggest that PWL chromotherapy and guided relaxation induce measurable electrical brain changes that could be beneficial under neuropsychiatric perspectives.

Synthetic Spatial Foraging With Active Inference in a Geocaching Task

Humans are highly proficient in learning about the environments in which they operate. They form flexible spatial representations of their surroundings that can be leveraged with ease during spatial foraging and navigation. To capture these abilities, we present a deep Active Inference model of goal-directed behavior, and the accompanying belief updating. Active Inference rests upon optimizing Bayesian beliefs to maximize model evidence or marginal likelihood. Bayesian beliefs are probability distributions over the causes of observable outcomes. These causes include an agent's actions, which enables one to treat planning as inference. We use simulations of a geocaching task to elucidate the belief updating-that underwrites spatial foraging-and the associated behavioral and neurophysiological responses. In a geocaching task, the aim is to find hidden objects in the environment using spatial coordinates. Here, synthetic agents learn about the environment via inference and learning (e.g., learning about the likelihoods of outcomes given latent states) to reach a target location, and then forage locally to discover the hidden object that offers clues for the next location.

Canonical neural networks perform active inference

This work considers a class of canonical neural networks comprising rate coding models, wherein neural activity and plasticity minimise a common cost function-and plasticity is modulated with a certain delay. We show that such neural networks implicitly perform active inference and learning to minimise the risk associated with future outcomes. Mathematical analyses demonstrate that this biological optimisation can be cast as maximisation of model evidence, or equivalently minimisation of variational free energy, under the well-known form of a partially observed Markov decision process model. This equivalence indicates that the delayed modulation of Hebbian plasticity-accompanied with adaptation of firing thresholds-is a sufficient neuronal substrate to attain Bayes optimal inference and control. We corroborated this proposition using numerical analyses of maze tasks. This theory offers a universal characterisation of canonical neural networks in terms of Bayesian belief updating and provides insight into the neuronal mechanisms underlying planning and adaptive behavioural control.

Early Electrophysiological Markers of Navigational Affordances in Scenes

Scene perception and spatial navigation are interdependent cognitive functions, and there is increasing evidence that cortical areas that process perceptual scene properties also carry information about the potential for navigation in the environment (navigational affordances). However, the temporal stages by which visual information is transformed into navigationally relevant information are not yet known. We hypothesized that navigational affordances are encoded during perceptual processing and therefore should modulate early visually evoked ERPs, especially the scene-selective P2 component. To test this idea, we recorded ERPs from participants while they passively viewed computer-generated room scenes matched in visual complexity. By simply changing the number of doors (no doors, 1 door, 2 doors, 3 doors), we were able to systematically vary the number of pathways that afford movement in the local environment, while keeping the overall size and shape of the environment constant. We found that rooms with no doors evoked a higher P2 response than rooms with three doors, consistent with prior research reporting higher P2 amplitude to closed relative to open scenes. Moreover, we found P2 amplitude scaled linearly with the number of doors in the scenes. Navigability effects on the ERP waveform were also observed in a multivariate analysis, which showed significant decoding of the number of doors and their location at earlier time windows. Together, our results suggest that navigational affordances are represented in the early stages of scene perception. This complements research showing that the occipital place area automatically encodes the structure of navigable space and strengthens the link between scene perception and navigation.

Active inference leads to Bayesian neurophysiology

The neuronal substrates that implement the free-energy principle and ensuing active inference at the neuron and synapse level have not been fully elucidated. This Review considers possible neuronal substrates underlying the principle. First, the foundations of the free-energy principle are introduced, and then its ability to empirically explain various brain functions and psychological and biological phenomena in terms of Bayesian inference is described. Mathematically, the dynamics of neural activity and plasticity that minimise a cost function can be cast as performing Bayesian inference that minimises variational free energy. This equivalence licenses the adoption of the free-energy principle as a universal characterisation of neural networks. Further, the neural network structure itself represents a generative model under which an agent operates. A virtue of this perspective is that it enables the formal association of neural network properties with prior beliefs that regulate inference and learning. The possible neuronal substrates that implement prior beliefs and how to empirically examine the theory are discussed. This perspective renders brain activity explainable, leading to a deeper understanding of the neuronal mechanisms underlying basic psychology and psychiatric disorders in terms of an implicit generative model.

World model learning and inference

Understanding information processing in the brain-and creating general-purpose artificial intelligence-are long-standing aspirations of scientists and engineers worldwide. The distinctive features of human intelligence are high-level cognition and control in various interactions with the world including the self, which are not defined in advance and are vary over time. The challenge of building human-like intelligent machines, as well as progress in brain science and behavioural analyses, robotics, and their associated theoretical formalisations, speaks to the importance of the world-model learning and inference. In this article, after briefly surveying the history and challenges of internal model learning and probabilistic learning, we introduce the free energy principle, which provides a useful framework within which to consider neuronal computation and probabilistic world models. Next, we showcase examples of human behaviour and cognition explained under that principle. We then describe symbol emergence in the context of probabilistic modelling, as a topic at the frontiers of cognitive robotics. Lastly, we review recent progress in creating human-like intelligence by using novel probabilistic programming languages. The striking consensus that emerges from these studies is that probabilistic descriptions of learning and inference are powerful and effective ways to create human-like artificial intelligent machines and to understand intelligence in the context of how humans interact with their world.

The Acquisition of Culturally Patterned Attention Styles Under Active Inference

This paper presents an active inference based simulation study of visual foraging. The goal of the simulation is to show the effect of the acquisition of culturally patterned attention styles on cognitive task performance, under active inference. We show how cultural artefacts like antique vase decorations drive cognitive functions such as perception, action and learning, as well as task performance in a simple visual discrimination task. We thus describe a new active inference based research pipeline that future work may employ to inquire on deep guiding principles determining the manner in which material culture drives human thought, by building and rebuilding our patterns of attention.

A New Paradigm for the Study of Cognitive Flexibility in Children and Adolescents: The "Virtual House Locomotor Maze" (VHLM)

Classical neuropsychological assessments are designed to explore cognitive brain functions using paper-and-pencil or digital tests. The purpose of this study was to design and to test a new protocol named the "Virtual House Locomotor Maze" (VHLM) for studying inhibitory control as well as mental flexibility using a visuo-spatial locomotor memory test. The VHLM is a simple maze including six houses using the technology of the Virtual Carpet Paradigm™. Ten typical development children (TD) were enrolled in this study. The participants were instructed to reach a target house as quickly as possible and to bear in mind the experimental instructions. We examined their planning and replanning abilities to take the shortest path to reach a target house. In order to study the cognitive processes during navigation, we implemented a spatio-temporal index based on the measure of kinematics behaviors (i.e., trajectories, tangential velocity and head direction). Replanning was tested by first repeating a path chosen by the subject to reach a given house. After learning this path, it was blocked imposing that the subject inhibited the learned trajectory and designed a new trajectory to reach the same house. We measured the latency of the departure after the presentation of each house and the initial direction of the trajectory. The results suggest that several strategies are used by the subjects for replanning and our measures could be used as an index of impulsivity.

An empirical evaluation of active inference in multi-armed bandits

A key feature of sequential decision making under uncertainty is a need to balance between exploiting-choosing the best action according to the current knowledge, and exploring-obtaining information about values of other actions. The multi-armed bandit problem, a classical task that captures this trade-off, served as a vehicle in machine learning for developing bandit algorithms that proved to be useful in numerous industrial applications. The active inference framework, an approach to sequential decision making recently developed in neuroscience for understanding human and animal behaviour, is distinguished by its sophisticated strategy for resolving the exploration-exploitation trade-off. This makes active inference an exciting alternative to already established bandit algorithms. Here we derive an efficient and scalable approximate active inference algorithm and compare it to two state-of-the-art bandit algorithms: Bayesian upper confidence bound and optimistic Thompson sampling. This comparison is done on two types of bandit problems: a stationary and a dynamic switching bandit. Our empirical evaluation shows that the active inference algorithm does not produce efficient long-term behaviour in stationary bandits. However, in the more challenging switching bandit problem active inference performs substantially better than the two state-of-the-art bandit algorithms. The results open exciting venues for further research in theoretical and applied machine learning, as well as lend additional credibility to active inference as a general framework for studying human and animal behaviour.

Consciousness in active inference: Deep self-models, other minds, and the challenge of psychedelic-induced ego-dissolution

Predictive processing approaches to brain function are increasingly delivering promise for illuminating the computational underpinnings of a wide range of phenomenological states. It remains unclear, however, whether predictive processing is equipped to accommodate a theory of consciousness itself. Furthermore, objectors have argued that without specification of the core computational mechanisms of consciousness, predictive processing is unable to inform the attribution of consciousness to other non-human (biological and artificial) systems. In this paper, I argue that an account of consciousness in the predictive brain is within reach via recent accounts of phenomenal self-modelling in the active inference framework. The central claim here is that phenomenal consciousness is underpinned by 'subjective valuation'-a deep inference about the precision or 'predictability' of the self-evidencing ('fitness-promoting') outcomes of action. Based on this account, I argue that this approach can critically inform the distribution of experience in other systems, paying particular attention to the complex sensory attenuation mechanisms associated with deep self-models. I then consider an objection to the account: several recent papers argue that theories of consciousness that invoke self-consciousness as constitutive or necessary for consciousness are undermined by states (or traits) of 'selflessness'; in particular the 'totally selfless' states of ego-dissolution occasioned by psychedelic drugs. Drawing on existing work that accounts for psychedelic-induced ego-dissolution in the active inference framework, I argue that these states do not threaten to undermine an active inference theory of consciousness. Instead, these accounts corroborate the view that subjective valuation is the constitutive facet of experience, and they highlight the potential of psychedelic research to inform consciousness science, computational psychiatry and computational phenomenology.

Sophisticated Inference

Active inference offers a first principle account of sentient behavior, from which special and important cases-for example, reinforcement learning, active learning, Bayes optimal inference, Bayes optimal design-can be derived. Active inference finesses the exploitation-exploration dilemma in relation to prior preferences by placing information gain on the same footing as reward or value. In brief, active inference replaces value functions with functionals of (Bayesian) beliefs, in the form of an expected (variational) free energy. In this letter, we consider a sophisticated kind of active inference using a recursive form of expected free energy. Sophistication describes the degree to which an agent has beliefs about beliefs. We consider agents with beliefs about the counterfactual consequences of action for states of affairs and beliefs about those latent states. In other words, we move from simply considering beliefs about "what would happen if I did that" to "what I would believe about what would happen if I did that." The recursive form of the free energy functional effectively implements a deep tree search over actions and outcomes in the future. Crucially, this search is over sequences of belief states as opposed to states per se. We illustrate the competence of this scheme using numerical simulations of deep decision problems.