Active inference and epistemic value

Generative Models for Active Vision

The active visual system comprises the visual cortices, cerebral attention networks, and oculomotor system. While fascinating in its own right, it is also an important model for sensorimotor networks in general. A prominent approach to studying this system is active inference-which assumes the brain makes use of an internal (generative) model to predict proprioceptive and visual input. This approach treats action as ensuring sensations conform to predictions (i.e., by moving the eyes) and posits that visual percepts are the consequence of updating predictions to conform to sensations. Under active inference, the challenge is to identify the form of the generative model that makes these predictions-and thus directs behavior. In this paper, we provide an overview of the generative models that the brain must employ to engage in active vision. This means specifying the processes that explain retinal cell activity and proprioceptive information from oculomotor muscle fibers. In addition to the mechanics of the eyes and retina, these processes include our choices about where to move our eyes. These decisions rest upon beliefs about salient locations, or the potential for information gain and belief-updating. A key theme of this paper is the relationship between "looking" and "seeing" under the brain's implicit generative model of the visual world.

Representation of Contralateral Visual Space in the Human Hippocampus

The initial encoding of visual information primarily from the contralateral visual field is a fundamental organizing principle of the primate visual system. Recently, the presence of such retinotopic sensitivity has been shown to extend well beyond early visual cortex to regions not historically considered retinotopically sensitive. In particular, human scene-selective regions in parahippocampal and medial parietal cortex exhibit prominent biases for the contralateral visual field. Here, we used fMRI to test the hypothesis that the human hippocampus, which is thought to be anatomically connected with these scene-selective regions, would also exhibit a biased representation of contralateral visual space. First, population receptive field (pRF) mapping with scene stimuli revealed strong biases for the contralateral visual field in bilateral hippocampus. Second, the distribution of retinotopic sensitivity suggested a more prominent representation in anterior medial portions of the hippocampus. Finally, the contralateral bias was confirmed in independent data taken from the Human Connectome Project (HCP) initiative. The presence of contralateral biases in the hippocampus, a structure considered by many as the apex of the visual hierarchy, highlights the truly pervasive influence of retinotopy. Moreover, this finding has important implications for understanding how visual information relates to the allocentric global spatial representations known to be encoded therein.SIGNIFICANCE STATEMENT Retinotopic encoding of visual information is an organizing principle of visual cortex. Recent work demonstrates this sensitivity in structures far beyond early visual cortex, including those anatomically connected to the hippocampus. Here, using population receptive field (pRF) modeling in two independent sets of data we demonstrate a consistent bias for the contralateral visual field in bilateral hippocampus. Such a bias highlights the truly pervasive influence of retinotopy, with important implications for understanding how the presence of retinotopy relates to more allocentric spatial representations.

Information Theory for Agents in Artificial Intelligence, Psychology, and Economics

This review looks at some of the central relationships between artificial intelligence, psychology, and economics through the lens of information theory, specifically focusing on formal models of decision-theory. In doing so we look at a particular approach that each field has adopted and how information theory has informed the development of the ideas of each field. A key theme is expected utility theory, its connection to information theory, the Bayesian approach to decision-making and forms of (bounded) rationality. What emerges from this review is a broadly unified formal perspective derived from three very different starting points that reflect the unique principles of each field. Each of the three approaches reviewed can, in principle at least, be implemented in a computational model in such a way that, with sufficient computational power, they could be compared with human abilities in complex tasks. However, a central critique that can be applied to all three approaches was first put forward by Savage in The Foundations of Statistics and recently brought to the fore by the economist Binmore: Bayesian approaches to decision-making work in what Savage called ‘small worlds’ but cannot work in ‘large worlds’. This point, in various different guises, is central to some of the current debates about the power of artificial intelligence and its relationship to human-like learning and decision-making. Recent work on artificial intelligence has gone some way to bridging this gap but significant questions remain to be answered in all three fields in order to make progress in producing realistic models of human decision-making in the real world in which we live in.

The Thalamus as a Blackboard for Perception and Planning

It has been suggested that the thalamus acts as a blackboard, on which the computations of different cortical modules are composed, coordinated, and integrated. This article asks what blackboard role the thalamus might play, and whether that role is consistent with the neuroanatomy of the thalamus. It does so in a context of Bayesian belief updating, expressed as a Free Energy Principle. We suggest that the thalamus-as-a-blackboard offers important questions for research in spatial cognition. Several prominent features of the thalamus-including its lack of olfactory relay function, its lack of internal excitatory connections, its regular and conserved shape, its inhibitory interneurons, triadic synapses, and diffuse cortical connectivity-are consistent with a blackboard role.Different thalamic nuclei may play different blackboard roles: (1) the Pulvinar, through its reciprocal connections to posterior cortical regions, coordinates perceptual inference about "what is where" from multi-sense-data. (2) The Mediodorsal (MD) nucleus, through its connections to the prefrontal cortex, and the other thalamic nuclei linked to the motor cortex, uses the same generative model for planning and learning novel spatial movements. (3) The paraventricular nucleus may compute risk-reward trade-offs. We also propose that as any new movement is practiced a few times, cortico-thalamocortical (CTC) links entrain the corresponding cortico-cortical links, through a process akin to supervised learning. Subsequently, the movement becomes a fast unconscious habit, not requiring the MD nucleus or other thalamic nuclei, and bypassing the thalamic bottleneck.

A Technical Critique of Some Parts of the Free Energy Principle

We summarize the original formulation of the free energy principle and highlight some technical issues. We discuss how these issues affect related results involving generalised coordinates and, where appropriate, mention consequences for and reveal, up to now unacknowledged, differences from newer formulations of the free energy principle. In particular, we reveal that various definitions of the “Markov blanket” proposed in different works are not equivalent. We show that crucial steps in the free energy argument, which involve rewriting the equations of motion of systems with Markov blankets, are not generally correct without additional (previously unstated) assumptions. We prove by counterexamples that the original free energy lemma, when taken at face value, is wrong. We show further that this free energy lemma, when it does hold, implies the equality of variational density and ergodic conditional density. The interpretation in terms of Bayesian inference hinges on this point, and we hence conclude that it is not sufficiently justified. Additionally, we highlight that the variational densities presented in newer formulations of the free energy principle and lemma are parametrised by different variables than in older works, leading to a substantially different interpretation of the theory. Note that we only highlight some specific problems in the discussed publications. These problems do not rule out conclusively that the general ideas behind the free energy principle are worth pursuing.

Towards Strong AI

Strong AI—artificial intelligence that is in all respects at least as intelligent as humans—is still out of reach. Current AI lacks common sense, that is, it is not able to infer, understand, or explain the hidden processes, forces, and causes behind data. Main stream machine learning research on deep artificial neural networks (ANNs) may even be characterized as being behavioristic. In contrast, various sources of evidence from cognitive science suggest that human brains engage in the active development of compositional generative predictive models (CGPMs) from their self-generated sensorimotor experiences. Guided by evolutionarily-shaped inductive learning and information processing biases, they exhibit the tendency to organize the gathered experiences into event-predictive encodings. Meanwhile, they infer and optimize behavior and attention by means of both epistemic- and homeostasis-oriented drives. I argue that AI research should set a stronger focus on learning CGPMs of the hidden causes that lead to the registered observations. Endowed with suitable information-processing biases, AI may develop that will be able to explain the reality it is confronted with, reason about it, and find adaptive solutions, making it Strong AI. Seeing that such Strong AI can be equipped with a mental capacity and computational resources that exceed those of humans, the resulting system may have the potential to guide our knowledge, technology, and policies into sustainable directions. Clearly, though, Strong AI may also be used to manipulate us even more. Thus, it will be on us to put good, far-reaching and long-term, homeostasis-oriented purpose into these machines.

Sensorimotor Representation Learning for an “Active Self” in Robots: A Model Survey

Safe human-robot interactions require robots to be able to learn how to behave appropriately in spaces populated by people and thus to cope with the challenges posed by our dynamic and unstructured environment, rather than being provided a rigid set of rules for operations. In humans, these capabilities are thought to be related to our ability to perceive our body in space, sensing the location of our limbs during movement, being aware of other objects and agents, and controlling our body parts to interact with them intentionally. Toward the next generation of robots with bio-inspired capacities, in this paper, we first review the developmental processes of underlying mechanisms of these abilities: The sensory representations of body schema, peripersonal space, and the active self in humans. Second, we provide a survey of robotics models of these sensory representations and robotics models of the self; and we compare these models with the human counterparts. Finally, we analyze what is missing from these robotics models and propose a theoretical computational framework, which aims to allow the emergence of the sense of self in artificial agents by developing sensory representations through self-exploration.

Encoding-linked pupil response is modulated by expected and unexpected novelty: Implications for memory formation and neurotransmission

Whether a novel stimulus is expected or unexpected may have implications for the kind of ensuing encoding and the type of subsequent memory. Pupil response was used in the present study to explore the way expected and unexpected stimuli are encoded and whether encoding-linked pupil response is modulated by expectation. Participants first established a contingency relationship between a series of symbols and the type of stimulus (man-made or natural) that followed each one. At encoding, some of the target stimuli violated the previously established relationship (i.e., unexpected), while the majority conformed to this relationship (i.e., expected). Expectation at encoding had opposite effects on familiarity and recollection, the two types of memory that support recognition, and modulated differently the way pupil response predicted subsequent memory. Encoding of unexpected novel stimuli was associated with increased pupil dilation as a predictor of subsequent memory type and strength. In contrast, encoding of expected novel stimuli was associated with decreased pupil response (constriction), which was predictive of subsequent memory type and strength. The findings support the close link between pupil response and memory formation, but critically indicate that this is modulated by the type of novelty as defined by expectation. These novel findings have important implications for the encoding mechanisms involved when different types of novelty are detected and is proposed to indicate the operation of different neurotransmitters during memory formation.

A critical analysis of Markovian monism

Free Energy Principle underlies a unifying framework that integrates theories of origins of life, cognition, and action. Recently, FEP has been developed into a Markovian monist perspective (Friston et al. in BC 102: 227-260, 2020). The paper expresses scepticism about the validity of arguments for Markovian monism. The critique is based on the assumption that Markovian models are scientific models, and while we may defend ontological theories about the nature of scientific models, we could not read off metaphysical theses about the nature of target systems (self-organising conscious systems, in the present context) from our theories of nature of scientific models (Markov blankets). The paper draws attention to different ways of understanding Markovian models, as material entities, fictional entities, and mathematical structures. I argue that none of these interpretations contributes to the defence of a metaphysical stance (either in terms of neutral monism or reductive physicalism). This is because scientific representation is a sophisticated process, and properties of Markovian models-such as the property of being neither physical nor mental-could not be projected onto their targets to determine the ontological properties of targets easily.

Deeply Felt Affect: The Emergence of Valence in Deep Active Inference

The positive-negative axis of emotional valence has long been recognized as fundamental to adaptive behavior, but its origin and underlying function have largely eluded formal theorizing and computational modeling. Using deep active inference, a hierarchical inference scheme that rests on inverting a model of how sensory data are generated, we develop a principled Bayesian model of emotional valence. This formulation asserts that agents infer their valence state based on the expected precision of their action model-an internal estimate of overall model fitness ("subjective fitness"). This index of subjective fitness can be estimated within any environment and exploits the domain generality of second-order beliefs (beliefs about beliefs). We show how maintaining internal valence representations allows the ensuing affective agent to optimize confidence in action selection preemptively. Valence representations can in turn be optimized by leveraging the (Bayes-optimal) updating term for subjective fitness, which we label affective charge (AC). AC tracks changes in fitness estimates and lends a sign to otherwise unsigned divergences between predictions and outcomes. We simulate the resulting affective inference by subjecting an in silico affective agent to a T-maze paradigm requiring context learning, followed by context reversal. This formulation of affective inference offers a principled account of the link between affect, (mental) action, and implicit metacognition. It characterizes how a deep biological system can infer its affective state and reduce uncertainty about such inferences through internal action (i.e., top-down modulation of priors that underwrite confidence). Thus, we demonstrate the potential of active inference to provide a formal and computationally tractable account of affect. Our demonstration of the face validity and potential utility of this formulation represents the first step within a larger research program. Next, this model can be leveraged to test the hypothesized role of valence by fitting the model to behavioral and neuronal responses.

Bayesian mechanics of perceptual inference and motor control in the brain

The free energy principle (FEP) in the neurosciences stipulates that all viable agents induce and minimize informational free energy in the brain to fit their environmental niche. In this study, we continue our effort to make the FEP a more physically principled formalism by implementing free energy minimization based on the principle of least action. We build a Bayesian mechanics (BM) by casting the formulation reported in the earlier publication (Kim in Neural Comput 30:2616-2659, 2018, https://doi.org/10.1162/neco_a_01115 ) to considering active inference beyond passive perception. The BM is a neural implementation of variational Bayes under the FEP in continuous time. The resulting BM is provided as an effective Hamilton's equation of motion and subject to the control signal arising from the brain's prediction errors at the proprioceptive level. To demonstrate the utility of our approach, we adopt a simple agent-based model and present a concrete numerical illustration of the brain performing recognition dynamics by integrating BM in neural phase space. Furthermore, we recapitulate the major theoretical architectures in the FEP by comparing our approach with the common state-space formulations.

Dopamine, Prediction Error and Beyond

A large body of work has linked dopaminergic signaling to learning and reward processing. It stresses the role of dopamine in reward prediction error signaling, a key neural signal that allows us to learn from past experiences, and that facilitates optimal choice behavior. Latterly, it has become clear that dopamine does not merely code prediction error size but also signals the difference between the expected value of rewards, and the value of rewards actually received, which is obtained through the integration of reward attributes such as the type, amount, probability and delay. More recent work has posited a role of dopamine in learning beyond rewards. These theories suggest that dopamine codes absolute or unsigned prediction errors, playing a key role in how the brain models associative regularities within its environment, while incorporating critical information about the reliability of those regularities. Work is emerging supporting this perspective and, it has inspired theoretical models of how certain forms of mental pathology may emerge in relation to dopamine function. Such pathology is frequently related to disturbed inferences leading to altered internal models of the environment. Thus, it is critical to understand the role of dopamine in error-related learning and inference.

Neuronal oscillations and the mouse prefrontal cortex

The mouse prefrontal cortex (PFC) encompasses a collection of agranual brain regions in the rostral neocortex and is considered to be critically involved in the neuronal computations underlying intentional behaviors. Flexible behavioral responses demand coordinated integration of sensory inputs with state, goal and memory information in brain-wide neuronal networks. Neuronal oscillations are proposed to provide a temporal scaffold for coordination of neuronal network activity and routing of information. In the present book chapter, we review findings on the role neuronal oscillations in prefrontal functioning, with a specific focus on research in mice. We discuss discoveries pertaining to local prefrontal processing, as well to interactions with other brain regions. We also discuss how the recent discovery of brain-wide respiration-entrained rhythms (RR) warrant re-evaluation of certain findings on slow oscillations (<10Hz) in prefrontal functioning.

Representation Wars: Enacting an Armistice Through Active Inference

Over the last 30 years, representationalist and dynamicist positions in the philosophy of cognitive science have argued over whether neurocognitive processes should be viewed as representational or not. Major scientific and technological developments over the years have furnished both parties with ever more sophisticated conceptual weaponry. In recent years, an enactive generalization of predictive processing – known as active inference – has been proposed as a unifying theory of brain functions. Since then, active inference has fueled both representationalist and dynamicist campaigns. However, we believe that when diving into the formal details of active inference, one should be able to find a solution to the war; if not a peace treaty, surely an armistice of a sort. Based on an analysis of these formal details, this paper shows how both representationalist and dynamicist sensibilities can peacefully coexist within the new territory of active inference.

Valuing what happens: a biogenic approach to valence and (potentially) affect

Valence is half of the pair of properties that constitute core affect, the foundation of emotion. But what is valence, and where is it found in the natural world? Currently, this question cannot be answered. The idea that emotion is the body's way of driving the organism to secure its survival, thriving and reproduction runs like a leitmotif from the pathfinding work of Antonio Damasio through four book-length neuroscientific accounts of emotion recently published by the field's leading practitioners. Yet while Damasio concluded 20 years ago that the homeostasis-affect linkage is rooted in unicellular life, no agreement exists about whether even non-human animals with brains experience emotions. Simple neural animals-those less brainy than bees, fruit flies and other charismatic invertebrates-are not even on the radar of contemporary affective research, to say nothing of aneural organisms. This near-sightedness has effectively denied the most productive method available for getting a grip on highly complex biological processes to a scientific domain whose importance for understanding biological decision-making cannot be underestimated. Valence arguably is the fulcrum around which the dance of life revolves. Without the ability to discriminate advantage from harm, life very quickly comes to an end. In this paper, we review the concept of valence, where it came from, the work it does in current leading theories of emotion, and some of the odd features revealed via experiment. We present a biologically grounded framework for investigating valence in any organism and sketch a preliminary pathway to a computational model. This article is part of the theme issue 'Basal cognition: conceptual tools and the view from the single cell'.

Whence the Expected Free Energy?

The expected free energy (EFE) is a central quantity in the theory of active inference. It is the quantity that all active inference agents are mandated to minimize through action, and its decomposition into extrinsic and intrinsic value terms is key to the balance of exploration and exploitation that active inference agents evince. Despite its importance, the mathematical origins of this quantity and its relation to the variational free energy (VFE) remain unclear. In this letter, we investigate the origins of the EFE in detail and show that it is not simply "the free energy in the future." We present a functional that we argue is the natural extension of the VFE but actively discourages exploratory behavior, thus demonstrating that exploration does not directly follow from free energy minimization into the future. We then develop a novel objective, the free energy of the expected future (FEEF), which possesses both the epistemic component of the EFE and an intuitive mathematical grounding as the divergence between predicted and desired futures.

Active Inference: Demystified and Compared

Active inference is a first principle account of how autonomous agents operate in dynamic, nonstationary environments. This problem is also considered in reinforcement learning, but limited work exists on comparing the two approaches on the same discrete-state environments. In this letter, we provide (1) an accessible overview of the discrete-state formulation of active inference, highlighting natural behaviors in active inference that are generally engineered in reinforcement learning, and (2) an explicit discrete-state comparison between active inference and reinforcement learning on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of reinforcement learning. We show that by operating in a pure belief-based setting, active inference agents can carry out epistemic exploration-and account for uncertainty about their environment-in a Bayes-optimal fashion. Furthermore, we show that the reliance on an explicit reward signal in reinforcement learning is removed in active inference, where reward can simply be treated as another observation we have a preference over; even in the total absence of rewards, agent behaviors are learned through preference learning. We make these properties explicit by showing two scenarios in which active inference agents can infer behaviors in reward-free environments compared to both Q-learning and Bayesian model-based reinforcement learning agents and by placing zero prior preferences over rewards and learning the prior preferences over the observations corresponding to reward. We conclude by noting that this formalism can be applied to more complex settings (e.g., robotic arm movement, Atari games) if appropriate generative models can be formulated. In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation and demonstrate these behaviors in a OpenAI gym environment, alongside reinforcement learning agents.

Decision Models and Technology Can Help Psychiatry Develop Biomarkers

Why is psychiatry unable to define clinically useful biomarkers? We explore this question from the vantage of data and decision science and consider biomarkers as a form of phenotypic data that resolves a well-defined clinical decision. We introduce a framework that systematizes different forms of phenotypic data and further introduce the concept of decision model to describe the strategies a clinician uses to seek out, combine, and act on clinical data. Though many medical specialties rely on quantitative clinical data and operationalized decision models, we observe that, in psychiatry, clinical data are gathered and used in idiosyncratic decision models that exist solely in the clinician's mind and therefore are outside empirical evaluation. This, we argue, is a fundamental reason why psychiatry is unable to define clinically useful biomarkers: because psychiatry does not currently quantify clinical data, decision models cannot be operationalized and, in the absence of an operationalized decision model, it is impossible to define how a biomarker might be of use. Here, psychiatry might benefit from digital technologies that have recently emerged specifically to quantify clinically relevant facets of human behavior. We propose that digital tools might help psychiatry in two ways: first, by quantifying data already present in the standard clinical interaction and by allowing decision models to be operationalized and evaluated; second, by testing whether new forms of data might have value within an operationalized decision model. We reference successes from other medical specialties to illustrate how quantitative data and operationalized decision models improve patient care.

Tea With Milk? A Hierarchical Generative Framework of Sequential Event Comprehension

To make sense of the world around us, we must be able to segment a continual stream of sensory inputs into discrete events. In this review, I propose that in order to comprehend events, we engage hierarchical generative models that "reverse engineer" the intentions of other agents as they produce sequential action in real time. By generating probabilistic predictions for upcoming events, generative models ensure that we are able to keep up with the rapid pace at which perceptual inputs unfold. By tracking our certainty about other agents' goals and the magnitude of prediction errors at multiple temporal scales, generative models enable us to detect event boundaries by inferring when a goal has changed. Moreover, by adapting flexibly to the broader dynamics of the environment and our own comprehension goals, generative models allow us to optimally allocate limited resources. Finally, I argue that we use generative models not only to comprehend events but also to produce events (carry out goal-relevant sequential action) and to continually learn about new events from our surroundings. Taken together, this hierarchical generative framework provides new insights into how the human brain processes events so effortlessly while highlighting the fundamental links between event comprehension, production, and learning.

The two kinds of free energy and the Bayesian revolution

The concept of free energy has its origins in 19th century thermodynamics, but has recently found its way into the behavioral and neural sciences, where it has been promoted for its wide applicability and has even been suggested as a fundamental principle of understanding intelligent behavior and brain function. We argue that there are essentially two different notions of free energy in current models of intelligent agency, that can both be considered as applications of Bayesian inference to the problem of action selection: one that appears when trading off accuracy and uncertainty based on a general maximum entropy principle, and one that formulates action selection in terms of minimizing an error measure that quantifies deviations of beliefs and policies from given reference models. The first approach provides a normative rule for action selection in the face of model uncertainty or when information processing capabilities are limited. The second approach directly aims to formulate the action selection problem as an inference problem in the context of Bayesian brain theories, also known as Active Inference in the literature. We elucidate the main ideas and discuss critical technical and conceptual issues revolving around these two notions of free energy that both claim to apply at all levels of decision-making, from the high-level deliberation of reasoning down to the low-level information processing of perception.

Active inference on discrete state-spaces: A synthesis

Active inference is a normative principle underwriting perception, action, planning, decision-making and learning in biological or artificial agents. From its inception, its associated process theory has grown to incorporate complex generative models, enabling simulation of a wide range of complex behaviours. Due to successive developments in active inference, it is often difficult to see how its underlying principle relates to process theories and practical implementation. In this paper, we try to bridge this gap by providing a complete mathematical synthesis of active inference on discrete state-space models. This technical summary provides an overview of the theory, derives neuronal dynamics from first principles and relates this dynamics to biological processes. Furthermore, this paper provides a fundamental building block needed to understand active inference for mixed generative models; allowing continuous sensations to inform discrete representations. This paper may be used as follows: to guide research towards outstanding challenges, a practical guide on how to implement active inference to simulate experimental behaviour, or a pointer towards various in-silico neurophysiological responses that may be used to make empirical predictions.

Learning Generative State Space Models for Active Inference

In this paper we investigate the active inference framework as a means to enable autonomous behavior in artificial agents. Active inference is a theoretical framework underpinning the way organisms act and observe in the real world. In active inference, agents act in order to minimize their so called free energy, or prediction error. Besides being biologically plausible, active inference has been shown to solve hard exploration problems in various simulated environments. However, these simulations typically require handcrafting a generative model for the agent. Therefore we propose to use recent advances in deep artificial neural networks to learn generative state space models from scratch, using only observation-action sequences. This way we are able to scale active inference to new and challenging problem domains, whilst still building on the theoretical backing of the free energy principle. We validate our approach on the mountain car problem to illustrate that our learnt models can indeed trade-off instrumental value and ambiguity. Furthermore, we show that generative models can also be learnt using high-dimensional pixel observations, both in the OpenAI Gym car racing environment and a real-world robotic navigation task. Finally we show that active inference based policies are an order of magnitude more sample efficient than Deep Q Networks on RL tasks.

How Do Living Systems Create Meaning?

Meaning has traditionally been regarded as a problem for philosophers and psychologists. Advances in cognitive science since the early 1960s, however, broadened discussions of meaning, or more technically, the semantics of perceptions, representations, and/or actions, into biology and computer science. Here, we review the notion of “meaning” as it applies to living systems, and argue that the question of how living systems create meaning unifies the biological and cognitive sciences across both organizational and temporal scales.

Losing Ourselves: Active Inference, Depersonalization, and Meditation

Disruptions in the ordinary sense of selfhood underpin both pathological and "enlightened" states of consciousness. People suffering from depersonalization can experience the loss of a sense of self as devastating, often accompanied by intense feelings of alienation, fear, and hopelessness. However, for meditative contemplatives from various traditions, "selfless" experiences are highly sought after, being associated with enduring peace and joy. Little is understood about how these contrasting dysphoric and euphoric experiences should be conceptualized. In this paper, we propose a unified account of these selfless experiences within the active inference framework. Building on our recent active inference research, we propose an account of the experiences of selfhood as emerging from a temporally deep generative model. We go on to develop a view of the self as playing a central role in structuring ordinary experience by "tuning" agents to the counterfactually rich possibilities for action. Finally, we explore how depersonalization may result from an inferred loss of allostatic control and contrast this phenomenology with selfless experiences reported by meditation practitioners. We will show how, by beginning with a conception of self-modeling within an active inference framework, we have available to us a new way of conceptualizing the striking experiential similarities and important differences between these selfless experiences within a unifying theoretical framework. We will explore the implications for understanding and treating dissociative disorders, as well as elucidate both the therapeutic potential, and possible dangers, of meditation.

Deep Active Inference and Scene Construction

Adaptive agents must act in intrinsically uncertain environments with complex latent structure. Here, we elaborate a model of visual foraging-in a hierarchical context-wherein agents infer a higher-order visual pattern (a "scene") by sequentially sampling ambiguous cues. Inspired by previous models of scene construction-that cast perception and action as consequences of approximate Bayesian inference-we use active inference to simulate decisions of agents categorizing a scene in a hierarchically-structured setting. Under active inference, agents develop probabilistic beliefs about their environment, while actively sampling it to maximize the evidence for their internal generative model. This approximate evidence maximization (i.e., self-evidencing) comprises drives to both maximize rewards and resolve uncertainty about hidden states. This is realized via minimization of a free energy functional of posterior beliefs about both the world as well as the actions used to sample or perturb it, corresponding to perception and action, respectively. We show that active inference, in the context of hierarchical scene construction, gives rise to many empirical evidence accumulation phenomena, such as noise-sensitive reaction times and epistemic saccades. We explain these behaviors in terms of the principled drives that constitute the expected free energy, the key quantity for evaluating policies under active inference. In addition, we report novel behaviors exhibited by these active inference agents that furnish new predictions for research on evidence accumulation and perceptual decision-making. We discuss the implications of this hierarchical active inference scheme for tasks that require planned sequences of information-gathering actions to infer compositional latent structure (such as visual scene construction and sentence comprehension). This work sets the stage for future experiments to investigate active inference in relation to other formulations of evidence accumulation (e.g., drift-diffusion models) in tasks that require planning in uncertain environments with higher-order structure.

“Ich denke, also sage ich vorher”: Wie “Predictive Processing-Modelle den Einsatz von Verhaltensexperimenten bei Depressionen optimieren können

Verhaltensexperimente stellen eine wichtige Interventionsform bei depressiven Störungen dar, um dysfunktionale Annahmen zu modifizieren. Häufig ist jedoch zu beobachten, dass Patient*innen trotz einer fachgerechten Durchführung von Verhaltensexperimenten weiter an negativen Annahmen festhalten. Vor diesem Hintergrund diskutieren wir in diesem Artikel, wie der Einsatz von Verhaltensexperimenten bei Depressionen optimiert werden kann und beziehen uns dabei auf ein neues Störungsmodell, das interessante Implikationen hierzu liefern könnte. Dieses Störungsmodell aus der neurowissenschaftlichen Forschung zu “Predictive Processing” geht davon aus, dass depressive Störungen durch zwei Kernaspekte gekennzeichnet sind: (1) stark negative Erwartungen, die im Sinne von selbsterfüllenden Pro­phezeiungen zu negativen Erlebnissen führen und zuneh­mend “immun” gegen gegenteilige Erfahrungen machen; (2) das Fehlen von positiven Erwartungen, wodurch die Wahr­scheinlichkeit von positiven Erlebnissen reduziert wird. Darauf aufbauend beschreiben wir zunächst, dass zur Modifikation negativer Erwartungen in einer ausführlichen Vorbesprechung erarbeitet werden sollte, welche möglichen positiven Erfahrungen aus dem Verhaltensexperiment die Patient*innen als glaubwürdig betrachten und zur Veränderung ihrer negativen Erwartungen nutzen würden. Auf diese sollte bei der Durchführung besonders geachtet werden, damit positive Erwartungsverletzungen auch als solche wahrgenommen werden. Zum anderen stellen wir dar, dass es nicht ausreichend ist, nur negative Erwartungen zu redu­zieren, sondern auch explizit positive Erwartungen durch weitere Verhaltensexperimente gefördert werden sollten.

Degeneracy and Redundancy in Active Inference

The notions of degeneracy and redundancy are important constructs in many areas, ranging from genomics through to network science. Degeneracy finds a powerful role in neuroscience, explaining key aspects of distributed processing and structure-function relationships in the brain. For example, degeneracy accounts for the superadditive effect of lesions on functional deficits in terms of a "many-to-one" structure-function mapping. In this paper, we offer a principled account of degeneracy and redundancy, when function is operationalized in terms of active inference, namely, a formulation of perception and action as belief updating under generative models of the world. In brief, "degeneracy" is quantified by the "entropy" of posterior beliefs about the causes of sensations, while "redundancy" is the "complexity" cost incurred by forming those beliefs. From this perspective, degeneracy and redundancy are complementary: Active inference tries to minimize redundancy while maintaining degeneracy. This formulation is substantiated using statistical and mathematical notions of degenerate mappings and statistical efficiency. We then illustrate changes in degeneracy and redundancy during the learning of a word repetition task. Finally, we characterize the effects of lesions-to intrinsic and extrinsic connections-using in silico disconnections. These numerical analyses highlight the fundamental difference between degeneracy and redundancy-and how they score distinct imperatives for perceptual inference and structure learning that are relevant to synthetic and biological intelligence.

"Stopping for knowledge": The sense of beauty in the perception-action cycle

According to a millennial-old philosophical debate, aesthetic emotions have been connected to knowledge acquisition. Recent scientific evidence, collected across different disciplinary domains, confirms this link, but also reveals that motor inhibition plays a crucial role in the process. In this review, we discuss multidisciplinary results and propose an original account of aesthetic appreciation (the stopping for knowledge hypothesis) framed within the predictive coding theory. We discuss evidence showing that aesthetic emotions emerge in correspondence with an inhibition of motor behavior (i.e., minimizing action), promoting a simultaneous perceptual processing enhancement, at the level of sensory cortices (i.e., optimizing learning). Accordingly, we suggest that aesthetic appreciation may represent a hedonic feedback over learning progresses, motivating the individual to inhibit motor routines to seek further knowledge acquisition. Furthermore, the neuroimaging and neuropsychological studies we review reveal the presence of a strong association between aesthetic appreciation and the activation of the dopaminergic reward-related circuits. Finally, we propose a number of possible applications of the stopping for knowledge hypothesis in the clinical and education domains.

Wilding the predictive brain

The Predictive Processing (PP) framework casts the brain as a probabilistic prediction engine that continually generates predictions of the causal structure of the world in order to construct for itself, from the top down, incoming sensory signals. Conceiving of the brain in this way has yielded incredible explanatory power, offering what many believe to be our first glimpse at a unified theory of the mind. In this paper, the picture of the mind brought into view by predictive processing theories is shown to be embodied, deeply affective and nicely poised for cognitive extension. We begin by giving an overview of the main themes of the framework, and situating this approach within embodied cognitive science. We show perception, action, homeostatic regulation and emotion to be underpinned by the very same predictive machinery. We conclude by showing how predictive minds will increasingly be understood as deeply interwoven with, and perhaps extended into, the surrounding social, cultural and technological landscape. This article is categorized under: Philosophy > Foundations of Cognitive Science Psychology > Emotion and Motivation Philosophy > Action.

A Bayesian Account of Generalist and Specialist Formation Under the Active Inference Framework

This paper offers a formal account of policy learning, or habitual behavioral optimization, under the framework of Active Inference. In this setting, habit formation becomes an autodidactic, experience-dependent process, based upon what the agent sees itself doing. We focus on the effect of environmental volatility on habit formation by simulating artificial agents operating in a partially observable Markov decision process. Specifically, we used a "two-step" maze paradigm, in which the agent has to decide whether to go left or right to secure a reward. We observe that in volatile environments with numerous reward locations, the agents learn to adopt a generalist strategy, never forming a strong habitual behavior for any preferred maze direction. Conversely, in conservative or static environments, agents adopt a specialist strategy; forming strong preferences for policies that result in approach to a small number of previously-observed reward locations. The pros and cons of the two strategies are tested and discussed. In general, specialization offers greater benefits, but only when contingencies are conserved over time. We consider the implications of this formal (Active Inference) account of policy learning for understanding the relationship between specialization and habit formation.

Science-Driven Societal Transformation, Part I: Worldview

Humanity faces serious social and environmental problems, including climate change and biodiversity loss. Increasingly, scientists, global policy experts, and the general public conclude that incremental approaches to reduce risk are insufficient and transformative change is needed across all sectors of society. However, the meaning of transformation is still unsettled in the literature, as is the proper role of science in fostering it. This paper is the first in a three-part series that adds to the discussion by proposing a novel science-driven research-and-development program aimed at societal transformation. More than a proposal, it offers a perspective and conceptual framework from which societal transformation might be approached. As part of this, it advances a formal mechanics with which to model and understand self-organizing societies of individuals. While acknowledging the necessity of reform to existing societal systems (e.g., governance, economic, and financial systems), the focus of the series is on transformation understood as systems change or systems migration—the de novo development of and migration to new societal systems. The series provides definitions, aims, reasoning, worldview, and a theory of change, and discusses fitness metrics and design principles for new systems. This first paper proposes a worldview, built using ideas from evolutionary biology, complex systems science, cognitive sciences, and information theory, which is intended to serve as the foundation for the R&D program. Subsequent papers in the series build on the worldview to address fitness metrics, system design, and other topics.

When Beliefs Face Reality: An Integrative Review of Belief Updating in Mental Health and Illness

Belief updating is a relatively nascent field of research that examines how people adjust their beliefs in light of new evidence. So far, belief updating has been investigated in partly unrelated lines of research from different psychological disciplines. In this article, we aim to integrate these disparate lines of research. After presenting some prominent theoretical frameworks and experimental designs that have been used for the study of belief updating, we review how healthy people and people with mental disorders update their beliefs after receiving new information that supports or challenges their views. Available evidence suggests that both healthy people and people with particular mental disorders are prone to certain biases when updating their beliefs, although the nature of the respective biases varies considerably and depends on several factors. Anomalies in belief updating are discussed in terms of both new insights into the psychopathology of various mental disorders and societal implications, such as irreconcilable political and societal controversies due to the failure to take information into account that disconfirms one’s own view. We conclude by proposing a novel integrative model of belief updating and derive directions for future research.

Predictive coding: Neuroscience and art

This paper examines the applicability of predictive coding as an explanatory model for perception. This is carried out from two perspectives. First, the central assumptions of the model are re-examined in light of the neuroscientific evidence for the structure and functioning of key brain areas involved in perception. The inferential processes involved in predictive coding are then investigated in the context of ambiguous stimuli. This showed that while predictive coding may provide an accurate explanation for our perceptual experiences in some cases, there are also several instances where the picture is not as clear cut. Following on from this, particular emphasis is placed on ambiguous art in order to examine the psychological and cognitive implications of predictive coding in affective states. This not only sheds light on the impact of predictive coding for cognition and emotion, but also helps clarify the nature of ambiguous art.

Do Process-1 simulations generate the epistemic feelings that drive Process-2 decision making?

We apply previously developed Chu space and Channel Theory methods, focusing on the construction of Cone-Cocone Diagrams (CCCDs), to study the role of epistemic feelings, particularly feelings of confidence, in dual process models of problem solving. We specifically consider "Bayesian brain" models of probabilistic inference within a global neuronal workspace architecture. We develop a formal representation of Process-1 problem solving in which a solution is reached if and only if a CCCD is completed. We show that in this representation, Process-2 problem solving can be represented as multiply iterated Process-1 problem solving and has the same formal solution conditions. We then model the generation of explicit, reportable subjective probabilities from implicit, experienced confidence as a simulation-based, reverse engineering process and show that this process can also be modeled as a CCCD construction.

Dynamic behavior of the locus coeruleus during arousal-related memory processing in a multi-modal 7T fMRI paradigm

A body of animal and human evidence points to the norepinephrine (NE) locus coeruleus (LC) system in modulating memory for arousing experiences, but whether the LC would recast its role along memory stages remains unknown. Sedation precluded examination of LC dynamics during memory processing in animals. Here, we addressed the contribution of the LC during arousal-associated memory processing through a unique combination of dedicated ultra-high-field LC-imaging methods, a well-established emotional memory task, online physiological and saliva alpha-amylase measurements in young adults. Arousal-related LC activation followed amygdala engagement during encoding. During consolidation and recollection, activation transitioned to hippocampal involvement, reflecting learning and model updating. NE-LC activation is dynamic, plays an arousal-controlling role, and is not sufficient but requires interactions with the amygdala to form adaptive memories of emotional experiences. These findings have implications for understanding contributions of LC dysregulation to disruptions in emotional memory formation, observed in psychiatric and neurocognitive disorders.

Dopamine modulates subcortical responses to surprising sounds

Dopamine guides behavior and learning through pleasure, according to classic understanding. Dopaminergic neurons are traditionally thought to signal positive or negative prediction errors (PEs) when reward expectations are, respectively, exceeded or not matched. These signed PEs are quite different from the unsigned PEs, which report surprise during sensory processing. But mounting theoretical accounts from the predictive processing framework postulate that dopamine, as a neuromodulator, could potentially regulate the postsynaptic gain of sensory neurons, thereby scaling unsigned PEs according to their expected precision or confidence. Despite ample modeling work, the physiological effects of dopamine on the processing of surprising sensory information are yet to be addressed experimentally. In this study, we tested how dopamine modulates midbrain processing of unexpected tones. We recorded extracellular responses from the rat inferior colliculus to oddball and cascade sequences, before, during, and after the microiontophoretic application of dopamine or eticlopride (a D₂-like receptor antagonist). Results demonstrate that dopamine reduces the net neuronal responsiveness exclusively to unexpected sensory input without significantly altering the processing of expected input. We conclude that dopaminergic projections from the thalamic subparafascicular nucleus to the inferior colliculus could encode the expected precision of unsigned PEs, attenuating via D₂-like receptors the postsynaptic gain of sensory inputs forwarded by the auditory midbrain neurons. This direct dopaminergic modulation of sensory PE signaling has profound implications for both the predictive coding framework and the understanding of dopamine function.

Information about unexpected stimuli is encoded in the form of prediction error signals. The earliest prediction error signals identified in the auditory brain emerge subcortically in the inferior colliculus. This study reveals the essential role of dopamine in encoding the precision of prediction errors at the auditory midbrain.

Social epistemic actions

We consider the ways humans engage in social epistemic actions, to guide each other's attention, prediction, and learning processes towards salient information, at the timescale of online social interaction and joint action. This parallels the active guidance of other's attention, prediction, and learning processes at the longer timescale of niche construction and cultural practices, as discussed in the target article.

Inferring What to Do (And What Not to)

In recent years, the "planning as inference" paradigm has become central to the study of behaviour. The advance offered by this is the formalisation of motivation as a prior belief about "how I am going to act". This paper provides an overview of the factors that contribute to this prior. These are rooted in optimal experimental design, information theory, and statistical decision making. We unpack how these factors imply a functional architecture for motivated behaviour. This raises an important question: how can we put this architecture to work in the service of understanding observed neurobiological structure? To answer this question, we draw from established techniques in experimental studies of behaviour. Typically, these examine the influence of perturbations of the nervous system-which include pathological insults or optogenetic manipulations-to see their influence on behaviour. Here, we argue that the message passing that emerges from inferring what to do can be similarly perturbed. If a given perturbation elicits the same behaviours as a focal brain lesion, this provides a functional interpretation of empirical findings and an anatomical grounding for theoretical results. We highlight examples of this approach that influence different sorts of goal-directed behaviour, active learning, and decision making. Finally, we summarise their implications for the neuroanatomy of inferring what to do (and what not to).

Unburdening the Shoulders of Giants: A Quest for Disconnected Academic Psychology

In current academic psychology, scholars typically develop their research and ideas by drawing on the work of other contemporary and preceding psychological scientists and by following certain conventions of the field. I refer to this variant of psychology as connected because the emphasis is on connecting various research findings and ideas generated by different scholars (e.g., by showing how they are related to each other via referencing). In this article, I argue that, although connected psychology advances psychological knowledge, it restricts the total amount of knowledge that could eventually be produced and therefore limits the potential of the discipline to improve the understanding of psychological phenomena. As a solution, I propose that, alongside the currently existing connected psychology, disconnected psychology should be established. In disconnected psychology, researchers develop their ideas by following the main principles of psychological method, but they are disconnected from a “field” consisting of other psychologists and therefore do not follow the discipline’s norms and conventions. By drawing on one of the core constructs from information theory—information entropy—I argue that combining the two streams of psychology would result in the most significant advancement of psychological knowledge.

Making Sense of the World: Infant Learning From a Predictive Processing Perspective

For human infants, the first years after birth are a period of intense exploration-getting to understand their own competencies in interaction with a complex physical and social environment. In contemporary neuroscience, the predictive-processing framework has been proposed as a general working principle of the human brain, the optimization of predictions about the consequences of one's own actions, and sensory inputs from the environment. However, the predictive-processing framework has rarely been applied to infancy research. We argue that a predictive-processing framework may provide a unifying perspective on several phenomena of infant development and learning that may seem unrelated at first sight. These phenomena include statistical learning principles, infants' motor and proprioceptive learning, and infants' basic understanding of their physical and social environment. We discuss how a predictive-processing perspective can advance the understanding of infants' early learning processes in theory, research, and application.

An Investigation of the Free Energy Principle for Emotion Recognition

This paper offers a prospectus of what might be achievable in the development of emotional recognition devices. It provides a conceptual overview of the free energy principle; including Markov blankets, active inference, and-in particular-a discussion of selfhood and theory of mind, followed by a brief explanation of how these concepts can explain both neural and cultural models of emotional inference. The underlying hypothesis is that emotion recognition and inference devices will evolve from state-of-the-art deep learning models into active inference schemes that go beyond marketing applications and become adjunct to psychiatric practice. Specifically, this paper proposes that a second wave of emotion recognition devices will be equipped with an emotional lexicon (or the ability to epistemically search for one), allowing the device to resolve uncertainty about emotional states by actively eliciting responses from the user and learning from these responses. Following this, a third wave of emotional devices will converge upon the user's generative model, resulting in the machine and human engaging in a reciprocal, prosocial emotional interaction, i.e., sharing a generative model of emotional states.

Rethinking post-traumatic stress disorder - A predictive processing perspective

Predictive processing has become a popular framework in neuroscience and computational psychiatry, where it has provided a new understanding of various mental disorders. Here, we apply the predictive processing account to post-traumatic stress disorder (PTSD). We argue that the experience of a traumatic event in Bayesian terms can be understood as a perceptual hypothesis that is subsequently given a very high a-priori likelihood due to its (life-) threatening significance; thus, this hypothesis is re-selected although it does not fit the actual sensory input. Based on this account, we re-conceptualise the symptom clusters of PTSD through the lens of a predictive processing model. We particularly focus on re-experiencing symptoms as the hallmark symptoms of PTSD, and discuss the occurrence of flashbacks in terms of perceptual and interoceptive inference. This account provides not only a new understanding of the clinical profile of PTSD, but also a unifying framework for the corresponding pathologies at the neurobiological level. Finally, we derive directions for future research and discuss implications for psychological and pharmacological interventions.

Communication between the Anterior Cingulate Cortex and Ventral Tegmental Area during a Cost-Benefit Reversal Task

The anterior cingulate cortex (ACC) is implicated in value-based decision making, anticipation, and adaptation; however, how ACC activity modulates these behaviors is unclear. One possibility is via the ACC's connections with the ventral tegmental area (VTA), a dopaminergic region implicated in motivation and feedback processing. We tested this by monitoring ACC and VTA local field potentials in rats performing a cost-benefit reversal task that elicited both value-based and anticipatory choices. Partial directed coherence analyses revealed that elevated 4-Hz ACC-to-VTA signaling accompanied decisions that appeared to be anticipatory. ACC-to-VTA signaling also occurred post-reversal, consistent with it being involved in the initiation of non-default behavior. An analysis of 4-Hz signals in the other direction (VTA-to-ACC) revealed that it was elevated when the rats committed errors and that this signal was followed by behavioral adaptation. Together, these findings suggest that bidirectional communication between the ACC and VTA supports behavioral flexibility.

Learning action-oriented models through active inference

Converging theories suggest that organisms learn and exploit probabilistic models of their environment. However, it remains unclear how such models can be learned in practice. The open-ended complexity of natural environments means that it is generally infeasible for organisms to model their environment comprehensively. Alternatively, action-oriented models attempt to encode a parsimonious representation of adaptive agent-environment interactions. One approach to learning action-oriented models is to learn online in the presence of goal-directed behaviours. This constrains an agent to behaviourally relevant trajectories, reducing the diversity of the data a model need account for. Unfortunately, this approach can cause models to prematurely converge to sub-optimal solutions, through a process we refer to as a bad-bootstrap. Here, we exploit the normative framework of active inference to show that efficient action-oriented models can be learned by balancing goal-oriented and epistemic (information-seeking) behaviours in a principled manner. We illustrate our approach using a simple agent-based model of bacterial chemotaxis. We first demonstrate that learning via goal-directed behaviour indeed constrains models to behaviorally relevant aspects of the environment, but that this approach is prone to sub-optimal convergence. We then demonstrate that epistemic behaviours facilitate the construction of accurate and comprehensive models, but that these models are not tailored to any specific behavioural niche and are therefore less efficient in their use of data. Finally, we show that active inference agents learn models that are parsimonious, tailored to action, and which avoid bad bootstraps and sub-optimal convergence. Critically, our results indicate that models learned through active inference can support adaptive behaviour in spite of, and indeed because of, their departure from veridical representations of the environment. Our approach provides a principled method for learning adaptive models from limited interactions with an environment, highlighting a route to sample efficient learning algorithms.

A World Unto Itself: Human Communication as Active Inference

Recent theoretical work in developmental psychology suggests that humans are predisposed to align their mental states with those of other individuals. One way this manifests is in cooperative communication; that is, intentional communication aimed at aligning individuals' mental states with respect to events in their shared environment. This idea has received strong empirical support. The purpose of this paper is to extend this account by proposing an integrative model of the biobehavioral dynamics of cooperative communication. Our formulation is based on active inference. Active inference suggests that action-perception cycles operate to minimize uncertainty and optimize an individual's internal model of the world. We propose that humans are characterized by an evolved adaptive prior belief that their mental states are aligned with, or similar to, those of conspecifics (i.e., that 'we are the same sort of creature, inhabiting the same sort of niche'). The use of cooperative communication emerges as the principal means to gather evidence for this belief, allowing for the development of a shared narrative that is used to disambiguate interactants' (hidden and inferred) mental states. Thus, by using cooperative communication, individuals effectively attune to a hermeneutic niche composed, in part, of others' mental states; and, reciprocally, attune the niche to their own ends via epistemic niche construction. This means that niche construction enables features of the niche to encode precise, reliable cues about the deontic or shared value of certain action policies (e.g., the utility of using communicative constructions to disambiguate mental states, given expectations about shared prior beliefs). In turn, the alignment of mental states (prior beliefs) enables the emergence of a novel, contextualizing scale of cultural dynamics that encompasses the actions and mental states of the ensemble of interactants and their shared environment. The dynamics of this contextualizing layer of cultural organization feedback, across scales, to constrain the variability of the prior expectations of the individuals who constitute it. Our theory additionally builds upon the active inference literature by introducing a new set of neurobiologically plausible computational hypotheses for cooperative communication. We conclude with directions for future research.