Active inference and epistemic value

Locus Coeruleus tracking of prediction errors optimises cognitive flexibility: An Active Inference model

The locus coeruleus (LC) in the pons is the major source of noradrenaline (NA) in the brain. Two modes of LC firing have been associated with distinct cognitive states: changes in tonic rates of firing are correlated with global levels of arousal and behavioural flexibility, whilst phasic LC responses are evoked by salient stimuli. Here, we unify these two modes of firing by modelling the response of the LC as a correlate of a prediction error when inferring states for action planning under Active Inference (AI). We simulate a classic Go/No-go reward learning task and a three-arm 'explore/exploit' task and show that, if LC activity is considered to reflect the magnitude of high level 'state-action' prediction errors, then both tonic and phasic modes of firing are emergent features of belief updating. We also demonstrate that when contingencies change, AI agents can update their internal models more quickly by feeding back this state-action prediction error-reflected in LC firing and noradrenaline release-to optimise learning rate, enabling large adjustments over short timescales. We propose that such prediction errors are mediated by cortico-LC connections, whilst ascending input from LC to cortex modulates belief updating in anterior cingulate cortex (ACC). In short, we characterise the LC/ NA system within a general theory of brain function. In doing so, we show that contrasting, behaviour-dependent firing patterns are an emergent property of the LC that translates state-action prediction errors into an optimal balance between plasticity and stability.

Systems of Bounded Rational Agents with Information-Theoretic Constraints

Specialization and hierarchical organization are important features of efficient collaboration in economical, artificial, and biological systems. Here, we investigate the hypothesis that both features can be explained by the fact that each entity of such a system is limited in a certain way. We propose an information-theoretic approach based on a free energy principle in order to computationally analyze systems of bounded rational agents that deal with such limitations optimally. We find that specialization allows a focus on fewer tasks, thus leading to a more efficient execution, but in turn, it requires coordination in hierarchical structures of specialized experts and coordinating units. Our results suggest that hierarchical architectures of specialized units at lower levels that are coordinated by units at higher levels are optimal, given that each unit's information-processing capability is limited and conforms to constraints on complexity costs.

The Projective Consciousness Model and Phenomenal Selfhood

We summarize our recently introduced Projective Consciousness Model (PCM) (Rudrauf et al., 2017) and relate it to outstanding conceptual issues in the theory of consciousness. The PCM combines a projective geometrical model of the perspectival phenomenological structure of the field of consciousness with a variational Free Energy minimization model of active inference, yielding an account of the cybernetic function of consciousness, viz., the modulation of the field's cognitive and affective dynamics for the effective control of embodied agents. The geometrical and active inference components are linked via the concept of projective transformation, which is crucial to understanding how conscious organisms integrate perception, emotion, memory, reasoning, and perspectival imagination in order to control behavior, enhance resilience, and optimize preference satisfaction. The PCM makes substantive empirical predictions and fits well into a (neuro)computationalist framework. It also helps us to account for aspects of subjective character that are sometimes ignored or conflated: pre-reflective self-consciousness, the first-person point of view, the sense of minenness or ownership, and social self-consciousness. We argue that the PCM, though still in development, offers us the most complete theory to date of what Thomas Metzinger has called "phenomenal selfhood."

Active Fovea-Based Vision Through Computationally-Effective Model-Based Prediction

What motivates an action in the absence of a definite reward? Taking the case of visuomotor control, we consider a minimal control problem that is how select the next saccade, in a sequence of discrete eye movements, when the final objective is to better interpret the current visual scene. The visual scene is modeled here as a partially-observed environment, with a generative model explaining how the visual data is shaped by action. This allows to interpret different action selection metrics proposed in the literature, including the Salience, the Infomax and the Variational Free Energy, under a single information theoretic construct, namely the view-based Information Gain. Pursuing this analytic track, two original action selection metrics named the Information Gain Lower Bound (IGLB) and the Information Gain Upper Bound (IGUB) are then proposed. Showing either a conservative or an optimistic bias regarding the Information Gain, they strongly simplify its calculation. An original fovea-based visual scene decoding setup is then proposed, with numerical experiments highlighting different facets of artificial fovea-based vision. A first and principal result is that state-of-the-art recognition rates are obtained with fovea-based saccadic exploration, using less than 10% of the original image's data. Those satisfactory results illustrate the advantage of mixing predictive control with accurate state-of-the-art predictors, namely a deep neural network. A second result is the sub-optimality of some classical action-selection metrics widely used in the literature, that is not manifest with finely-tuned inference models, but becomes patent when coarse or faulty models are used. Last, a computationally-effective predictive model is developed using the IGLB objective, with pre-processed visual scan-path read-out from memory, bypassing computationally-demanding predictive calculations. This last simplified setting is shown effective in our case, showing both a competing accuracy and a good robustness to model flaws.

Deep active inference

This work combines the free energy principle and the ensuing active inference dynamics with recent advances in variational inference in deep generative models, and evolution strategies to introduce the "deep active inference" agent. This agent minimises a variational free energy bound on the average surprise of its sensations, which is motivated by a homeostatic argument. It does so by optimising the parameters of a generative latent variable model of its sensory inputs, together with a variational density approximating the posterior distribution over the latent variables, given its observations, and by acting on its environment to actively sample input that is likely under this generative model. The internal dynamics of the agent are implemented using deep and recurrent neural networks, as used in machine learning, making the deep active inference agent a scalable and very flexible class of active inference agent. Using the mountain car problem, we show how goal-directed behaviour can be implemented by defining appropriate priors on the latent states in the agent's model. Furthermore, we show that the deep active inference agent can learn a generative model of the environment, which can be sampled from to understand the agent's beliefs about the environment and its interaction therewith.

The Anatomy of Inference: Generative Models and Brain Structure

To infer the causes of its sensations, the brain must call on a generative (predictive) model. This necessitates passing local messages between populations of neurons to update beliefs about hidden variables in the world beyond its sensory samples. It also entails inferences about how we will act. Active inference is a principled framework that frames perception and action as approximate Bayesian inference. This has been successful in accounting for a wide range of physiological and behavioral phenomena. Recently, a process theory has emerged that attempts to relate inferences to their neurobiological substrates. In this paper, we review and develop the anatomical aspects of this process theory. We argue that the form of the generative models required for inference constrains the way in which brain regions connect to one another. Specifically, neuronal populations representing beliefs about a variable must receive input from populations representing the Markov blanket of that variable. We illustrate this idea in four different domains: perception, planning, attention, and movement. In doing so, we attempt to show how appealing to generative models enables us to account for anatomical brain architectures. Ultimately, committing to an anatomical theory of inference ensures we can form empirical hypotheses that can be tested using neuroimaging, neuropsychological, and electrophysiological experiments.

Impulsivity and Active Inference

This paper characterizes impulsive behavior using a patch-leaving paradigm and active inference-a framework for describing Bayes optimal behavior. This paradigm comprises different environments (patches) with limited resources that decline over time at different rates. The challenge is to decide when to leave the current patch for another to maximize reward. We chose this task because it offers an operational characterization of impulsive behavior, namely, maximizing proximal reward at the expense of future gain. We use a Markov decision process formulation of active inference to simulate behavioral and electrophysiological responses under different models and prior beliefs. Our main finding is that there are at least three distinct causes of impulsive behavior, which we demonstrate by manipulating three different components of the Markov decision process model. These components comprise (i) the depth of planning, (ii) the capacity to maintain and process information, and (iii) the perceived value of immediate (relative to delayed) rewards. We show how these manipulations change beliefs and subsequent choices through variational message passing. Furthermore, we appeal to the process theories associated with this message passing to simulate neuronal correlates. In future work, we will use this scheme to identify the prior beliefs that underlie different sorts of impulsive behavior-and ask whether different causes of impulsivity can be inferred from the electrophysiological correlates of choice behavior.

Context-Dependent Risk Aversion: A Model-Based Approach

Most research on risk aversion in behavioral science with human subjects has focused on a component of risk aversion that does not adapt itself to context. More recently, studies have explored risk aversion adaptation to changing circumstances in sequential decision-making tasks. It is an open question whether one can identify evidence, at the single subject level, for such risk aversion adaptation. We conducted a behavioral experiment on human subjects, using a sequential decision making task. We developed a model-based approach for estimating the adaptation of risk-taking behavior with single-trial resolution by modeling a subject's goals and internal representation of task contingencies. Using this model-based approach, we estimated the subject-specific adaptation of risk aversion depending on the current task context. We found striking inter-subject variations in the adaptation of risk-taking behavior. We show that these differences can be explained by differences in subjects' internal representations of task contingencies and goals. We discuss that the proposed approach can be adapted to a wide range of experimental paradigms and be used to analyze behavioral measures other than risk aversion.

Dopaminergic genes are associated with both directed and random exploration

In order to maximize long-term rewards, agents must balance exploitation (choosing the option with the highest payoff) and exploration (gathering information about options that might have higher payoffs). Although the optimal solution to this trade-off is intractable, humans make use of two effective strategies: selectively exploring options with high uncertainty (directed exploration), and increasing the randomness of their choices when they are more uncertain (random exploration). Using a task that independently manipulates these two forms of exploration, we show that single nucleotide polymorphisms related to dopamine are associated with individual differences in exploration strategies. Variation in a gene linked to prefrontal dopamine (COMT) predicted the degree of directed exploration, as well as the overall randomness of responding. Variation in a gene linked to striatal dopamine (DARPP-32) predicted the degree of both directed and random exploration. These findings suggest that dopamine makes multiple contributions to exploration, depending on its afferent target.

Attention or salience?

While attention is widely recognised as central to perception, the term is often used to mean very different things. Prominent theories of attention - notably the premotor theory - relate it to planned or executed eye movements. This contrasts with the notion of attention as a gain control process that weights the information carried by different sensory channels. We draw upon recent advances in theoretical neurobiology to argue for a distinction between attentional gain mechanisms and salience attribution. The former depends upon estimating the precision of sensory data, while the latter is a consequence of the need to actively engage with the sensorium. Having established this distinction, we consider the intimate relationship between attention and salience.

Free-energy minimization in joint agent-environment systems: A niche construction perspective

The free-energy principle is an attempt to explain the structure of the agent and its brain, starting from the fact that an agent exists (Friston and Stephan, 2007; Friston et al., 2010). More specifically, it can be regarded as a systematic attempt to understand the 'fit' between an embodied agent and its niche, where the quantity of free-energy is a measure for the 'misfit' or disattunement (Bruineberg and Rietveld, 2014) between agent and environment. This paper offers a proof-of-principle simulation of niche construction under the free-energy principle. Agent-centered treatments have so far failed to address situations where environments change alongside agents, often due to the action of agents themselves. The key point of this paper is that the minimum of free-energy is not at a point in which the agent is maximally adapted to the statistics of a static environment, but can better be conceptualized an attracting manifold within the joint agent-environment state-space as a whole, which the system tends toward through mutual interaction. We will provide a general introduction to active inference and the free-energy principle. Using Markov Decision Processes (MDPs), we then describe a canonical generative model and the ensuing update equations that minimize free-energy. We then apply these equations to simulations of foraging in an environment; in which an agent learns the most efficient path to a pre-specified location. In some of those simulations, unbeknownst to the agent, the 'desire paths' emerge as a function of the activity of the agent (i.e. niche construction occurs). We will show how, depending on the relative inertia of the environment and agent, the joint agent-environment system moves to different attracting sets of jointly minimized free-energy.

An information-theoretic perspective on the costs of cognition

In statistics and machine learning, model accuracy is traded off with complexity, which can be viewed as the amount of information extracted from the data. Here, we discuss how cognitive costs can be expressed in terms of similar information costs, i.e. as a function of the amount of information required to update a person's prior knowledge (or internal model) to effectively solve a task. We then examine the theoretical consequences that ensue from this assumption. This framework naturally explains why some tasks - for example, unfamiliar or dual tasks - are costly and permits to quantify these costs using information-theoretic measures. Finally, we discuss brain implementation of this principle and show that subjective cognitive costs can originate either from local or global capacity limitations on information processing or from increased rate of metabolic alterations. These views shed light on the potential adaptive value of cost-avoidance mechanisms.

Precision and False Perceptual Inference

Accurate perceptual inference fundamentally depends upon accurate beliefs about the reliability of sensory data. In this paper, we describe a Bayes optimal and biologically plausible scheme that refines these beliefs through a gradient descent on variational free energy. To illustrate this, we simulate belief updating during visual foraging and show that changes in estimated sensory precision (i.e., confidence in visual data) are highly sensitive to prior beliefs about the contents of a visual scene. In brief, confident prior beliefs induce an increase in estimated precision when consistent with sensory evidence, but a decrease when they conflict. Prior beliefs held with low confidence are rapidly updated to posterior beliefs, determined by sensory data. These induce much smaller changes in beliefs about sensory precision. We argue that pathologies of scene construction may be due to abnormal priors, and show that these can induce a reduction in estimated sensory precision. Having previously associated this precision with cholinergic signaling, we note that several neurodegenerative conditions are associated with visual disturbances and cholinergic deficits; notably, the synucleinopathies. On relating the message passing in our model to the functional anatomy of the ventral visual stream, we find that simulated neuronal loss in temporal lobe regions induces confident, inaccurate, empirical prior beliefs at lower levels in the visual hierarchy. This provides a plausible, if speculative, computational mechanism for the loss of cholinergic signaling and the visual disturbances associated with temporal lobe Lewy body pathology. This may be seen as an illustration of the sorts of hypotheses that may be expressed within this computational framework.

Toward mechanistic models of action-oriented and detached cognition

To be successful, the research agenda for a novel control view of cognition should foresee more detailed, computationally specified process models of cognitive operations including higher cognition. These models should cover all domains of cognition, including those cognitive abilities that can be characterized as online interactive loops and detached forms of cognition that depend on internally generated neuronal processing.

Being a Beast Machine: The Somatic Basis of Selfhood

Modern psychology has long focused on the body as the basis of the self. Recently, predictive processing accounts of interoception (perception of the body 'from within') have become influential in accounting for experiences of body ownership and emotion. Here, we describe embodied selfhood in terms of 'instrumental interoceptive inference' that emphasises allostatic regulation and physiological integrity. We apply this approach to the distinctive phenomenology of embodied selfhood, accounting for its non-object-like character and subjective stability over time. Our perspective has implications for the development of selfhood and illuminates longstanding debates about relations between life and mind, implying, contrary to Descartes, that experiences of embodied selfhood arise because of, and not in spite of, our nature as 'beast machines'.

Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis

While the neurobiology of simple and habitual choices is relatively well known, our current understanding of goal-directed choices and planning in the brain is still limited. Theoretical work suggests that goal-directed computations can be productively associated to model-based (reinforcement learning) computations, yet a detailed mapping between computational processes and neuronal circuits remains to be fully established. Here we report a computational analysis that aligns Bayesian nonparametrics and model-based reinforcement learning (MB-RL) to the functioning of the hippocampus (HC) and the ventral striatum (vStr)-a neuronal circuit that increasingly recognized to be an appropriate model system to understand goal-directed (spatial) decisions and planning mechanisms in the brain. We test the MB-RL agent in a contextual conditioning task that depends on intact hippocampus and ventral striatal (shell) function and show that it solves the task while showing key behavioral and neuronal signatures of the HC-vStr circuit. Our simulations also explore the benefits of biological forms of look-ahead prediction (forward sweeps) during both learning and control. This article thus contributes to fill the gap between our current understanding of computational algorithms and biological realizations of (model-based) reinforcement learning.

The Discrete and Continuous Brain: From Decisions to Movement-And Back Again

To act upon the world, creatures must change continuous variables such as muscle length or chemical concentration. In contrast, decision making is an inherently discrete process, involving the selection among alternative courses of action. In this article, we consider the interface between the discrete and continuous processes that translate our decisions into movement in a Newtonian world-and how movement informs our decisions. We do so by appealing to active inference, with a special focus on the oculomotor system. Within this exemplar system, we argue that the superior colliculus is well placed to act as a discrete-continuous interface. Interestingly, when the neuronal computations within the superior colliculus are formulated in terms of active inference, we find that many aspects of its neuroanatomy emerge from the computations it must perform in this role.

Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop

Active inference is an ambitious theory that treats perception, inference, and action selection of autonomous agents under the heading of a single principle. It suggests biologically plausible explanations for many cognitive phenomena, including consciousness. In active inference, action selection is driven by an objective function that evaluates possible future actions with respect to current, inferred beliefs about the world. Active inference at its core is independent from extrinsic rewards, resulting in a high level of robustness across e.g., different environments or agent morphologies. In the literature, paradigms that share this independence have been summarized under the notion of intrinsic motivations. In general and in contrast to active inference, these models of motivation come without a commitment to particular inference and action selection mechanisms. In this article, we study if the inference and action selection machinery of active inference can also be used by alternatives to the originally included intrinsic motivation. The perception-action loop explicitly relates inference and action selection to the environment and agent memory, and is consequently used as foundation for our analysis. We reconstruct the active inference approach, locate the original formulation within, and show how alternative intrinsic motivations can be used while keeping many of the original features intact. Furthermore, we illustrate the connection to universal reinforcement learning by means of our formalism. Active inference research may profit from comparisons of the dynamics induced by alternative intrinsic motivations. Research on intrinsic motivations may profit from an additional way to implement intrinsically motivated agents that also share the biological plausibility of active inference.

Intelligence and uncertainty: Implications of hierarchical predictive processing for the neuroscience of cognitive ability

Hierarchical predictive processing (PP) has recently emerged as a candidate theoretical paradigm for neurobehavioral research. To date, PP has found support through its success in offering compelling explanations for a number of perceptual, cognitive, and psychiatric phenomena, as well as from accumulating neurophysiological evidence. However, its implications for understanding intelligence and its neural basis have received relatively little attention. The present review outlines the key tenets and evidence for PP, and assesses its implications for intelligence research. It is argued that PP suggests indeterminacy as a unifying principle from which to investigate the cognitive hierarchy and brain-ability correlations. The resulting framework not only accommodates prominent psychometric models of intelligence, but also incorporates key findings from neuroanatomical and functional activation research, and motivates new predictions via the mechanisms of prediction-error minimization. Because PP also suggests unique neural signatures of experience-dependent activity, it may also help clarify environmental contributions to intellectual development. It is concluded that PP represents a plausible, integrative framework that could enhance progress in the neuroscience of intelligence.

Uncertainty, epistemics and active inference

Biological systems—like ourselves—are constantly faced with uncertainty. Despite noisy sensory data, and volatile environments, creatures appear to actively maintain their integrity. To account for this remarkable ability to make optimal decisions in the face of a capricious world, we propose a generative model that represents the beliefs an agent might possess about their own uncertainty. By simulating a noisy and volatile environment, we demonstrate how uncertainty influences optimal epistemic (visual) foraging. In our simulations, saccades were deployed less frequently to regions with a lower sensory precision, while a greater volatility led to a shorter inhibition of return. These simulations illustrate a principled explanation for some cardinal aspects of visual foraging—and allow us to propose a correspondence between the representation of uncertainty and ascending neuromodulatory systems, complementing that suggested by Yu & Dayan (Yu & Dayan 2005 Neuron46, 681–692. (doi:10.1016/j.neuron.2005.04.026)).

Expected Free Energy Formalizes Conflict Underlying Defense in Freudian Psychoanalysis

Freud's core interest in the psyche was the dynamic unconscious: that part of the psyche which is unconscious due to conflict (Freud, 1923/1961). Over the course of his career, Freud variously described conflict as an opposition to the discharge of activation (Freud, 1950), opposition to psychic activity due to the release of unpleasure (Freud, 1990/1991), opposition between the primary principle and the reality principle (Freud, 1911/1963), structural conflict between id, ego, and superego (Freud, 1923/1961), and ambivalence (Freud, 1912/1963). Besides this difficulty of the shifting description of conflict, an underlying question remained the specific shared terrain in which emotions, thoughts, intentions or wishes could come into conflict with one another (the neuronal homolog of conflict), and most especially how they may exist as quantities in opposition within that terrain. Friston's free-energy principle (FEP henceforth) connected to the work of Friston (Friston et al., 2006; Friston, 2010) has provided the potential for a powerful unifying theory in psychology, neuroscience, and related fields that has been shown to have tremendous consilience with psychoanalytic concepts (Hopkins, 2012). Hopkins (2016), drawing on a formulation by Hobson et al. (2014), suggests that conflict may be potentially quantifiable as free energy from a FEP perspective. More recently, work by Friston et al. (2017a) has framed the selection of action as a gradient descent of expected free energy under different policies of action. From this perspective, the article describes how conflict could potentially be formalized as a situation where opposing action policies have similar expected free energy, for example between actions driven by competing basic prototype emotion systems as described by Panksepp (1998). This conflict state may be avoided in the future through updating the relative precision of a particular set of prior beliefs about outcomes: this has the result of tending to favor one of the policies of action over others in future instances, a situation analogous to defense. Through acting as a constraint on the further development of the person, the defensive operation can become entrenched, and resistant to alteration. The implications that this formalization has for psychoanalysis is explored.

Active Inference, Belief Propagation, and the Bethe Approximation

When modeling goal-directed behavior in the presence of various sources of uncertainty, planning can be described as an inference process. A solution to the problem of planning as inference was previously proposed in the active inference framework in the form of an approximate inference scheme based on variational free energy. However, this approximate scheme was based on the mean-field approximation, which assumes statistical independence of hidden variables and is known to show overconfidence and may converge to local minima of the free energy. To better capture the spatiotemporal properties of an environment, we reformulated the approximate inference process using the so-called Bethe approximation. Importantly, the Bethe approximation allows for representation of pairwise statistical dependencies. Under these assumptions, the minimizer of the variational free energy corresponds to the belief propagation algorithm, commonly used in machine learning. To illustrate the differences between the mean-field approximation and the Bethe approximation, we have simulated agent behavior in a simple goal-reaching task with different types of uncertainties. Overall, the Bethe agent achieves higher success rates in reaching goal states. We relate the better performance of the Bethe agent to more accurate predictions about the consequences of its own actions. Consequently, active inference based on the Bethe approximation extends the application range of active inference to more complex behavioral tasks.

The Discrete and Continuous Brain: From Decisions to Movement-And Back Again

To act upon the world, creatures must change continuous variables such as muscle length or chemical concentration. In contrast, decision making is an inherently discrete process, involving the selection among alternative courses of action. In this article, we consider the interface between the discrete and continuous processes that translate our decisions into movement in a Newtonian world—and how movement informs our decisions. We do so by appealing to active inference, with a special focus on the oculomotor system. Within this exemplar system, we argue that the superior colliculus is well placed to act as a discrete-continuous interface. Interestingly, when the neuronal computations within the superior colliculus are formulated in terms of active inference, we find that many aspects of its neuroanatomy emerge from the computations it must perform in this role.

Am I Self-Conscious? (Or Does Self-Organization Entail Self-Consciousness?)

Is self-consciousness necessary for consciousness? The answer is yes. So there you have it-the answer is yes. This was my response to a question I was asked to address in a recent AEON piece (https://aeon.co/essays/consciousness-is-not-a-thing-but-a-process-of-inference). What follows is based upon the notes for that essay, with a special focus on self-organization, self-evidencing and self-modeling. I will try to substantiate my (polemic) answer from the perspective of a physicist. In brief, the argument goes as follows: if we want to talk about creatures, like ourselves, then we have to identify the characteristic behaviors they must exhibit. This is fairly easy to do by noting that living systems return to a set of attracting states time and time again. Mathematically, this implies the existence of a Lyapunov function that turns out to be model evidence (i.e., self-evidence) in Bayesian statistics or surprise (i.e., self-information) in information theory. This means that all biological processes can be construed as performing some form of inference, from evolution through to conscious processing. If this is the case, at what point do we invoke consciousness? The proposal on offer here is that the mind comes into being when self-evidencing has a temporal thickness or counterfactual depth, which grounds inferences about the consequences of my action. On this view, consciousness is nothing more than inference about my future; namely, the self-evidencing consequences of what I could do.

Intrinsic and extrinsic motivators of attachment under active inference

This paper addresses the formation of infant attachment types within the context of active inference: a holistic account of action, perception and learning in the brain. We show how the organised forms of attachment (secure, avoidant and ambivalent) might arise in (Bayesian) infants. Specifically, we show that these distinct forms of attachment emerge from a minimisation of free energy-over interoceptive states relating to internal stress levels-when seeking proximity to caregivers who have a varying impact on these interoceptive states. In line with empirical findings in disrupted patterns of affective communication, we then demonstrate how exteroceptive cues (in the form of caregiver-mediated AMBIANCE affective communication errors, ACE) can result in disorganised forms of attachment in infants of caregivers who consistently increase stress when the infant seeks proximity, but can have an organising (towards ambivalence) effect in infants of inconsistent caregivers. In particular, we differentiate disorganised attachment from avoidance in terms of the high epistemic value of proximity seeking behaviours (resulting from the caregiver's misleading exteroceptive cues) that preclude the emergence of coherent and organised behavioural policies. Our work, the first to formulate infant attachment in terms of active inference, makes a new testable prediction with regards to the types of affective communication errors that engender ambivalent attachment.

A variational approach to niche construction

In evolutionary biology, niche construction is sometimes described as a genuine evolutionary process whereby organisms, through their activities and regulatory mechanisms, modify their environment such as to steer their own evolutionary trajectory, and that of other species. There is ongoing debate, however, on the extent to which niche construction ought to be considered a bona fide evolutionary force, on a par with natural selection. Recent formulations of the variational free-energy principle as applied to the life sciences describe the properties of living systems, and their selection in evolution, in terms of variational inference. We argue that niche construction can be described using a variational approach. We propose new arguments to support the niche construction perspective, and to extend the variational approach to niche construction to current perspectives in various scientific fields.

Hierarchical Active Inference: A Theory of Motivated Control

Motivated control refers to the coordination of behaviour to achieve affectively valenced outcomes or goals. The study of motivated control traditionally assumes a distinction between control and motivational processes, which map to distinct (dorsolateral versus ventromedial) brain systems. However, the respective roles and interactions between these processes remain controversial. We offer a novel perspective that casts control and motivational processes as complementary aspects - goal propagation and prioritization, respectively - of active inference and hierarchical goal processing under deep generative models. We propose that the control hierarchy propagates prior preferences or goals, but their precision is informed by the motivational context, inferred at different levels of the motivational hierarchy. The ensuing integration of control and motivational processes underwrites action and policy selection and, ultimately, motivated behaviour, by enabling deep inference to prioritize goals in a context-sensitive way.

The Active Inference Approach to Ecological Perception: General Information Dynamics for Natural and Artificial Embodied Cognition

The emerging neurocomputational vision of humans as embodied, ecologically embedded, social agents—who shape and are shaped by their environment—offers a golden opportunity to revisit and revise ideas about the physical and information-theoretic underpinnings of life, mind, and consciousness itself. In particular, the active inference framework (AIF) makes it possible to bridge connections from computational neuroscience and robotics/AI to ecological psychology and phenomenology, revealing common underpinnings and overcoming key limitations. AIF opposes the mechanistic to the reductive, while staying fully grounded in a naturalistic and information-theoretic foundation, using the principle of free energy minimization. The latter provides a theoretical basis for a unified treatment of particles, organisms, and interactive machines, spanning from the inorganic to organic, non-life to life, and natural to artificial agents. We provide a brief introduction to AIF, then explore its implications for evolutionary theory, ecological psychology, embodied phenomenology, and robotics/AI research. We conclude the paper by considering implications for machine consciousness.

Unifying Theories of Psychedelic Drug Effects

How do psychedelic drugs produce their characteristic range of acute effects in perception, emotion, cognition, and sense of self? How do these effects relate to the clinical efficacy of psychedelic-assisted therapies? Efforts to understand psychedelic phenomena date back more than a century in Western science. In this article I review theories of psychedelic drug effects and highlight key concepts which have endured over the last 125 years of psychedelic science. First, I describe the subjective phenomenology of acute psychedelic effects using the best available data. Next, I review late 19th-century and early 20th-century theories-model psychoses theory, filtration theory, and psychoanalytic theory-and highlight their shared features. I then briefly review recent findings on the neuropharmacology and neurophysiology of psychedelic drugs in humans. Finally, I describe recent theories of psychedelic drug effects which leverage 21st-century cognitive neuroscience frameworks-entropic brain theory, integrated information theory, and predictive processing-and point out key shared features that link back to earlier theories. I identify an abstract principle which cuts across many theories past and present: psychedelic drugs perturb universal brain processes that normally serve to constrain neural systems central to perception, emotion, cognition, and sense of self. I conclude that making an explicit effort to investigate the principles and mechanisms of psychedelic drug effects is a uniquely powerful way to iteratively develop and test unifying theories of brain function.

Computational Neuropsychology and Bayesian Inference

Computational theories of brain function have become very influential in neuroscience. They have facilitated the growth of formal approaches to disease, particularly in psychiatric research. In this paper, we provide a narrative review of the body of computational research addressing neuropsychological syndromes, and focus on those that employ Bayesian frameworks. Bayesian approaches to understanding brain function formulate perception and action as inferential processes. These inferences combine ‘prior’ beliefs with a generative (predictive) model to explain the causes of sensations. Under this view, neuropsychological deficits can be thought of as false inferences that arise due to aberrant prior beliefs (that are poor fits to the real world). This draws upon the notion of a Bayes optimal pathology – optimal inference with suboptimal priors – and provides a means for computational phenotyping. In principle, any given neuropsychological disorder could be characterized by the set of prior beliefs that would make a patient’s behavior appear Bayes optimal. We start with an overview of some key theoretical constructs and use these to motivate a form of computational neuropsychology that relates anatomical structures in the brain to the computations they perform. Throughout, we draw upon computational accounts of neuropsychological syndromes. These are selected to emphasize the key features of a Bayesian approach, and the possible types of pathological prior that may be present. They range from visual neglect through hallucinations to autism. Through these illustrative examples, we review the use of Bayesian approaches to understand the link between biology and computation that is at the heart of neuropsychology.

The Computational Anatomy of Visual Neglect

Visual neglect is a debilitating neuropsychological phenomenon that has many clinical implications and-in cognitive neuroscience-offers an important lesion deficit model. In this article, we describe a computational model of visual neglect based upon active inference. Our objective is to establish a computational and neurophysiological process theory that can be used to disambiguate among the various causes of this important syndrome; namely, a computational neuropsychology of visual neglect. We introduce a Bayes optimal model based upon Markov decision processes that reproduces the visual searches induced by the line cancellation task (used to characterize visual neglect at the bedside). We then consider 3 distinct ways in which the model could be lesioned to reproduce neuropsychological (visual search) deficits. Crucially, these 3 levels of pathology map nicely onto the neuroanatomy of saccadic eye movements and the systems implicated in visual neglect.

Human visual exploration reduces uncertainty about the sensed world

In previous papers, we introduced a normative scheme for scene construction and epistemic (visual) searches based upon active inference. This scheme provides a principled account of how people decide where to look, when categorising a visual scene based on its contents. In this paper, we use active inference to explain the visual searches of normal human subjects; enabling us to answer some key questions about visual foraging and salience attribution. First, we asked whether there is any evidence for 'epistemic foraging'; i.e. exploration that resolves uncertainty about a scene. In brief, we used Bayesian model comparison to compare Markov decision process (MDP) models of scan-paths that did-and did not-contain the epistemic, uncertainty-resolving imperatives for action selection. In the course of this model comparison, we discovered that it was necessary to include non-epistemic (heuristic) policies to explain observed behaviour (e.g., a reading-like strategy that involved scanning from left to right). Despite this use of heuristic policies, model comparison showed that there is substantial evidence for epistemic foraging in the visual exploration of even simple scenes. Second, we compared MDP models that did-and did not-allow for changes in prior expectations over successive blocks of the visual search paradigm. We found that implicit prior beliefs about the speed and accuracy of visual searches changed systematically with experience. Finally, we characterised intersubject variability in terms of subject-specific prior beliefs. Specifically, we used canonical correlation analysis to see if there were any mixtures of prior expectations that could predict between-subject differences in performance; thereby establishing a quantitative link between different behavioural phenotypes and Bayesian belief updating. We demonstrated that better scene categorisation performance is consistently associated with lower reliance on heuristics; i.e., a greater use of a generative model of the scene to direct its exploration.

Active Inference, Novelty and Neglect

In this chapter, we provide an overview of the principles of active inference. We illustrate how different forms of short-term memory are expressed formally (mathematically) through appealing to beliefs about the causes of our sensations and about the actions we pursue. This is used to motivate an approach to active vision that depends upon inferences about the causes of 'what I have seen' and learning about 'what I would see if I were to look there'. The former could manifest as persistent 'delay-period' activity - of the sort associated with working memory, while the latter is better suited to changes in synaptic efficacy - of the sort that underlies short-term learning and adaptation. We review formulations of these ideas in terms of active inference, their role in directing visual exploration and the consequences - for active vision - of their failures. To illustrate the latter, we draw upon some of our recent work on the computational anatomy of visual neglect.

Balancing New against Old Information: The Role of Puzzlement Surprise in Learning

Surprise describes a range of phenomena from unexpected events to behavioral responses. We propose a novel measure of surprise and use it for surprise-driven learning. Our surprise measure takes into account data likelihood as well as the degree of commitment to a belief via the entropy of the belief distribution. We find that surprise-minimizing learning dynamically adjusts the balance between new and old information without the need of knowledge about the temporal statistics of the environment. We apply our framework to a dynamic decision-making task and a maze exploration task. Our surprise-minimizing framework is suitable for learning in complex environments, even if the environment undergoes gradual or sudden changes, and it could eventually provide a framework to study the behavior of humans and animals as they encounter surprising events.

Top-down models in biology: explanation and control of complex living systems above the molecular level

It is widely assumed in developmental biology and bioengineering that optimal understanding and control of complex living systems follows from models of molecular events. The success of reductionism has overshadowed attempts at top-down models and control policies in biological systems. However, other fields, including physics, engineering and neuroscience, have successfully used the explanations and models at higher levels of organization, including least-action principles in physics and control-theoretic models in computational neuroscience. Exploiting the dynamic regulation of pattern formation in embryogenesis and regeneration requires new approaches to understand how cells cooperate towards large-scale anatomical goal states. Here, we argue that top-down models of pattern homeostasis serve as proof of principle for extending the current paradigm beyond emergence and molecule-level rules. We define top-down control in a biological context, discuss the examples of how cognitive neuroscience and physics exploit these strategies, and illustrate areas in which they may offer significant advantages as complements to the mainstream paradigm. By targeting system controls at multiple levels of organization and demystifying goal-directed (cybernetic) processes, top-down strategies represent a roadmap for using the deep insights of other fields for transformative advances in regenerative medicine and systems bioengineering.

The Role of Working Memory for Cognitive Control in Anorexia Nervosa versus Substance Use Disorder

Prefrontal cortex executive functions, such as working memory (WM) interact with limbic processes to foster impulse control. Such an interaction is referred to in a growing body of publications by terms such as cognitive control, cognitive inhibition, affect regulation, self-regulation, top-down control, and cognitive–emotion interaction. The rising trend of research into cognitive control of impulsivity, using various related terms reflects the importance of research into impulse control, as failure to employ cognitions optimally may eventually result in mental disorder. Against this background, we take a novel approach using an impulse control spectrum model – where anorexia nervosa (AN) and substance use disorder (SUD) are at opposite extremes – to examine the role of WM for cognitive control. With this aim, we first summarize WM processes in the healthy brain in order to frame a systematic review of the neuropsychological, neural and genetic findings of AN and SUD. In our systematic review of WM/cognitive control, we found n = 15 studies of AN with a total of n = 582 AN and n = 365 HC participants; and n = 93 studies of SUD with n = 9106 SUD and n = 3028 HC participants. In particular, we consider how WM load/capacity may support the neural process of excessive epistemic foraging (cognitive sampling of the environment to test predictions about the world) in AN that reduces distraction from salient stimuli. We also consider the link between WM and cognitive control in people with SUD who are prone to ‘jumping to conclusions’ and reduced epistemic foraging. Finally, in light of our review, we consider WM training as a novel research tool and an adjunct to enhance treatment that improves cognitive control of impulsivity.

A Bayesian perspective on delusions: Suggestions for modifying two reasoning tasks

BACKGROUND AND OBJECTIVES

There are a range of mechanistic explanations on the formation and maintenance of delusions. Within the Bayesian brain hypothesis, particularly within the framework of predictive coding models, delusions are seen as an aberrant inference process characterized by either a failure in sensory attenuation or an aberrant weighting of prior experience. Testing of these Bayesian decision theories requires measuring of both the patients' confidence in their beliefs and the confidence they assign new, incoming information. In the Bayesian framework we apply here, the former is referred to as the prior while the latter is usually called the data or likelihood.

METHODS AND RESULTS

This narrative review will commence by giving an introduction to the basic concept underlying the Bayesian decision theory approach to delusion. A consequence of crucial importance of this sketch is that it provides a measure for the persistence of a belief. Experimental tasks measuring these parameters are presented. Further, a modification of two standard reasoning tasks, the beads task and the evidence integration task, is proposed that permits testing the parameters from Bayesian decision theory.

LIMITATIONS

Patients differ from controls by the distress the delusions causes to them. The Bayesian Decision theory framework has no explicit parameter for distress.

CONCLUSIONS

A more detailed reporting of differences between patients with delusions is warranted.

Understanding active sampling strategies: Empirical approaches and implications for attention and decision research

In natural behavior we actively gather information using attention and active sensing behaviors (such as shifts of gaze) to sample relevant cues. However, while attention and decision making are naturally coordinated, in the laboratory they have been dissociated. Attention is studied independently of the actions it serves. Conversely, decision theories make the simplifying assumption that the relevant information is given, and do not attempt to describe how the decision maker may learn and implement active sampling policies. In this paper I review recent studies that address questions of attentional learning, cue validity and information seeking in humans and non-human primates. These studies suggest that learning a sampling policy involves large scale interactions between networks of attention and valuation, which implement these policies based on reward maximization, uncertainty reduction and the intrinsic utility of cognitive states. I discuss the importance of using such paradigms for formalizing the role of attention, as well as devising more realistic theories of decision making that capture a broader range of empirical observations.

Active Inference, Curiosity and Insight

This article offers a formal account of curiosity and insight in terms of active (Bayesian) inference. It deals with the dual problem of inferring states of the world and learning its statistical structure. In contrast to current trends in machine learning (e.g., deep learning), we focus on how people attain insight and understanding using just a handful of observations, which are solicited through curious behavior. We use simulations of abstract rule learning and approximate Bayesian inference to show that minimizing (expected) variational free energy leads to active sampling of novel contingencies. This epistemic behavior closes explanatory gaps in generative models of the world, thereby reducing uncertainty and satisfying curiosity. We then move from epistemic learning to model selection or structure learning to show how abductive processes emerge when agents test plausible hypotheses about symmetries (i.e., invariances or rules) in their generative models. The ensuing Bayesian model reduction evinces mechanisms associated with sleep and has all the hallmarks of "aha" moments. This formulation moves toward a computational account of consciousness in the pre-Cartesian sense of sharable knowledge (i.e., con: "together"; scire: "to know").