An Elementary Introduction to Information Geometry

Information geometry analysis example for absolute and relative transmission loss in a shallow ocean

The model manifold, an information geometry tool, is a geometric representation of a model that can quantify the expected information content of modeling parameters. For a normal-mode sound propagation model in a shallow ocean environment, transmission loss (TL) is calculated for a vertical line array and model manifolds are constructed for both absolute and relative TL. For the example presented in this paper, relative TL yields more compact model manifolds with seabed environments that are less statistically distinguishable than manifolds of absolute TL. This example illustrates how model manifolds can be used to improve experimental design for inverse problems.

Minimum Information Variability in Linear Langevin Systems via Model Predictive Control

Controlling the time evolution of a probability distribution that describes the dynamics of a given complex system is a challenging problem. Achieving success in this endeavour will benefit multiple practical scenarios, e.g., controlling mesoscopic systems. Here, we propose a control approach blending the model predictive control technique with insights from information geometry theory. Focusing on linear Langevin systems, we use model predictive control online optimisation capabilities to determine the system inputs that minimise deviations from the geodesic of the information length over time, ensuring dynamics with minimum "geometric information variability". We validate our methodology through numerical experimentation on the Ornstein-Uhlenbeck process and Kramers equation, demonstrating its feasibility. Furthermore, in the context of the Ornstein-Uhlenbeck process, we analyse the impact on the entropy production and entropy rate, providing a physical understanding of the effects of minimum information variability control.

Stochastic Dynamics of Fusion Low-to-High Confinement Mode (L-H) Transition: Correlation and Causal Analyses Using Information Geometry

We investigate the stochastic dynamics of the prey-predator model of the Low-to-High confinement mode (L-H) transition in magnetically confined fusion plasmas. By considering stochastic noise in the turbulence and zonal flows as well as constant and time-varying input power Q, we perform multiple stochastic simulations of over a million trajectories using GPU computing. Due to stochastic noise, some trajectories undergo the L-H transition while others do not, leading to a mixture of H-mode and dithering at a given time and/or input power. One of the consequences of this is that H-mode characteristics appear at a smaller input power QQc as a second peak. The coexisting H-mode and dithering near Q=Qc leads to a prominent bimodal PDF with a gradual L-H transition rather than a sudden transition at Q=Qc and uncertainty in the input power. Also, a time-dependent input power leads to increased variability (dispersion) in stochastic trajectories and a more prominent bimodal PDF. We provide an interpretation of the results using information geometry to elucidate self-regulation between zonal flows, turbulence, and information causality rate to unravel causal relations involved in the L-H transition.

Normalized Power Variance: A new Field Orthogonal to Power in EEG Analysis

To date, electroencephalogram (EEG) has been used in the diagnosis of epilepsy, dementia, and disturbance of consciousness via the inspection of EEG waves and identification of abnormal electrical discharges and slowing of basic waves. In addition, EEG power analysis combined with a source estimation method like exact-low-resolution-brain-electromagnetic-tomography (eLORETA), which calculates the power of cortical electrical activity from EEG data, has been widely used to investigate cortical electrical activity in neuropsychiatric diseases. However, the recently developed field of mathematics "information geometry" indicates that EEG has another dimension orthogonal to power dimension - that of normalized power variance (NPV). In addition, by introducing the idea of information geometry, a significantly faster convergent estimator of NPV was obtained. Research into this NPV coordinate has been limited thus far. In this study, we applied this NPV analysis of eLORETA to idiopathic normal pressure hydrocephalus (iNPH) patients prior to a cerebrospinal fluid (CSF) shunt operation, where traditional power analysis could not detect any difference associated with CSF shunt operation outcome. Our NPV analysis of eLORETA detected significantly higher NPV values at the high convexity area in the beta frequency band between 17 shunt responders and 19 non-responders. Considering our present and past research findings about NPV, we also discuss the advantage of this application of NPV representing a sensitive early warning signal of cortical impairment. Overall, our findings demonstrated that EEG has another dimension - that of NPV, which contains a lot of information about cortical electrical activity that can be useful in clinical practice.

Jeffreys Divergence and Generalized Fisher Information Measures on Fokker-Planck Space-Time Random Field

In this paper, we present the derivation of Jeffreys divergence, generalized Fisher divergence, and the corresponding De Bruijn identities for space-time random field. First, we establish the connection between Jeffreys divergence and generalized Fisher information of a single space-time random field with respect to time and space variables. Furthermore, we obtain the Jeffreys divergence between two space-time random fields obtained by different parameters under the same Fokker-Planck equations. Then, the identities between the partial derivatives of the Jeffreys divergence with respect to space-time variables and the generalized Fisher divergence are found, also known as the De Bruijn identities. Later, at the end of the paper, we present three examples of the Fokker-Planck equations on space-time random fields, identify their density functions, and derive the Jeffreys divergence, generalized Fisher information, generalized Fisher divergence, and their corresponding De Bruijn identities.

An information geometrical evaluation of Shannon information metrics on a discrete n-dimensional digital manifold

The definition and nature of information have perplexed scientists due to its dual nature in measurements. The information is discrete and continuous when evaluated on a metric scale, and the Laplace-Beltrami operator and Gauss-Bonnet Theorem can map one to another. On the other hand, defining the information as a discrete entity on the surface area of an n-dimensional discrete digital manifold provides a unique way of calculating the entropy of a manifold. The software simulation shows that the surface area of the discrete n-dimensional digital manifold is an effectively computable function. Moreover, it also provides the information-geometrical evaluation of Shannon information metrics.

Playing it safe: information constrains collective betting strategies

Every interaction of a living organism with its environment involves the placement of a bet. Armed with partial knowledge about a stochastic world, the organism must decide its next step or near-term strategy, an act that implicitly or explicitly involves the assumption of a model of the world. Better information about environmental statistics can improve the bet quality, but in practice resources for information gathering are always limited. We argue that theories of optimal inference dictate that "complex" models are harder to infer with bounded information and lead to larger prediction errors. Thus, we propose a principle of playing it safe where, given finite information gathering capacity, biological systems should be biased towards simpler models of the world, and thereby to less risky betting strategies. In the framework of Bayesian inference, we show that there is an optimally safe adaptation strategy determined by the Bayesian prior. We then demonstrate that, in the context of stochastic phenotypic switching by bacteria, implementation of our principle of "playing it safe" increases fitness (population growth rate) of the bacterial collective. We suggest that the principle applies broadly to problems of adaptation, learning and evolution, and illuminates the types of environments in which organisms are able to thrive.

Information geometry and Bose-Einstein condensation

It is a long held conjecture in the connection between information geometry (IG) and thermodynamics that the curvature endowed by IG diverges at phase transitions. Recent work on the IG of Bose-Einstein (BE) gases challenged this conjecture by saying that in the limit of fugacity approaching unit-where BE condensation is expected-the curvature does not diverge; rather, it converges to zero. However, as the discontinuous behavior that identifies condensation is only observed at the thermodynamic limit, a study of the IG curvature at a finite number of particles, N, is in order from which the thermodynamic behavior can be observed by taking the thermodynamic limit ( N→∞) posteriorly. This article presents such a study. We find that for a trapped gas, as N increases, the values of curvature decrease proportionally to a power of N, while the temperature at which the maximum value of curvature occurs approaches the usually defined critical temperature. This means that, in the thermodynamic limit, the curvature has a limited value where a phase transition is observed, contradicting the forementioned conjecture.

Information geometry for multiparameter models: new perspectives on the origin of simplicity

Complex models in physics, biology, economics, and engineering are oftensloppy, meaning that the model parameters are not well determined by the model predictions for collective behavior. Many parameter combinations can vary over decades without significant changes in the predictions. This review uses information geometry to explore sloppiness and its deep relation to emergent theories. We introduce themodel manifoldof predictions, whose coordinates are the model parameters. Itshyperribbonstructure explains why only a few parameter combinations matter for the behavior. We review recent rigorous results that connect the hierarchy of hyperribbon widths to approximation theory, and to the smoothness of model predictions under changes of the control variables. We discuss recent geodesic methods to find simpler models on nearby boundaries of the model manifold-emergent theories with fewer parameters that explain the behavior equally well. We discuss a Bayesian prior which optimizes the mutual information between model parameters and experimental data, naturally favoring points on the emergent boundary theories and thus simpler models. We introduce a 'projected maximum likelihood' prior that efficiently approximates this optimal prior, and contrast both to the poor behavior of the traditional Jeffreys prior. We discuss the way the renormalization group coarse-graining in statistical mechanics introduces a flow of the model manifold, and connect stiff and sloppy directions along the model manifold with relevant and irrelevant eigendirections of the renormalization group. Finally, we discuss recently developed 'intensive' embedding methods, allowing one to visualize the predictions of arbitrary probabilistic models as low-dimensional projections of an isometric embedding, and illustrate our method by generating the model manifold of the Ising model.

Information Geometry, Complexity Measures and Data Analysis

In the last several years, a new approach to information theory, called information geometry, has emerged [...].

Information Geometry in Roegenian Economics

We characterise the geometry of the statistical Roegenian manifold that arises from the equilibrium distribution of an income of noninteracting identical economic actors. The main results for ideal income are included in three subsections: partition function in distribution, scalar curvature, and geodesics. Although this system displays no phase transition, its analysis provides an enlightening contrast with the results of Van der Waals Income in Roegenian Economics, where we shall examine the geometry of the economic Van der Waals income, which does exhibit a “monetary policy as liquidity—income” transition. Here we focus on three subsections: canonical partition function, economic limit, and information geometry of the economic Van der Waals manifold.

A Fisher Information-Based Incompatibility Criterion for Quantum Channels

We introduce a new incompatibility criterion for quantum channels based on the notion of (quantum) Fisher information. Our construction is based on a similar criterion for quantum measurements put forward by H. Zhu. We then study the power of the incompatibility criterion in different scenarios. First, we prove the first analytical conditions for the incompatibility of two Schur channels. Then, we study the incompatibility structure of a tuple of depolarizing channels, comparing the newly introduced criterion with the known results from asymmetric quantum cloning.

Information Thermodynamics of the Transition-Path Ensemble

The reaction coordinate describing a transition between reactant and product is a fundamental concept in the theory of chemical reactions. Within transition-path theory, a quantitative definition of the reaction coordinate is found in the committor, which is the probability that a trajectory initiated from a given microstate first reaches the product before the reactant. Here we develop an information-theoretic origin for the committor and show how selecting transition paths from a long ergodic equilibrium trajectory induces entropy production which exactly equals the information that system dynamics provide about the reactivity of trajectories. This equality of entropy production and dynamical information generation also holds at the level of arbitrary individual coordinates, providing parallel measures of the coordinate's relevance to the reaction, each of which is maximized by the committor.

Parameter estimation and uncertainty quantification using information geometry

In this work, we: (i) review likelihood-based inference for parameter estimation and the construction of confidence regions; and (ii) explore the use of techniques from information geometry, including geodesic curves and Riemann scalar curvature, to supplement typical techniques for uncertainty quantification, such as Bayesian methods, profile likelihood, asymptotic analysis and bootstrapping. These techniques from information geometry provide data-independent insights into uncertainty and identifiability, and can be used to inform data collection decisions. All code used in this work to implement the inference and information geometry techniques is available on GitHub.

Conformal Control Tools for Statistical Manifolds and for γ-Manifolds

The theory of statistical manifolds w.r.t. a conformal structure is reviewed in a creative manner and developed. By analogy, the γ-manifolds are introduced. New conformal invariant tools are defined. A necessary condition for the f-conformal equivalence of γ-manifolds is found, extending that for the α-conformal equivalence for statistical manifolds. Certain examples of these new defined geometrical objects are given in the theory of Iinformation.

Principal Component Analysis for Gaussian Process Posteriors

This letter proposes an extension of principal component analysis for gaussian process (GP) posteriors, denoted by GP-PCA. Since GP-PCA estimates a low-dimensional space of GP posteriors, it can be used for metalearning, a framework for improving the performance of target tasks by estimating a structure of a set of tasks. The issue is how to define a structure of a set of GPs with an infinite-dimensional parameter, such as coordinate system and a divergence. In this study, we reduce the infiniteness of GP to the finite-dimensional case under the information geometrical framework by considering a space of GP posteriors that have the same prior. In addition, we propose an approximation method of GP-PCA based on variational inference and demonstrate the effectiveness of GP-PCA as meta-learning through experiments.

Statistical Divergences between Densities of Truncated Exponential Families with Nested Supports: Duo Bregman and Duo Jensen Divergences

By calculating the Kullback–Leibler divergence between two probability measures belonging to different exponential families dominated by the same measure, we obtain a formula that generalizes the ordinary Fenchel–Young divergence. Inspired by this formula, we define the duo Fenchel–Young divergence and report a majorization condition on its pair of strictly convex generators, which guarantees that this divergence is always non-negative. The duo Fenchel–Young divergence is also equivalent to a duo Bregman divergence. We show how to use these duo divergences by calculating the Kullback–Leibler divergence between densities of truncated exponential families with nested supports, and report a formula for the Kullback–Leibler divergence between truncated normal distributions. Finally, we prove that the skewed Bhattacharyya distances between truncated exponential families amount to equivalent skewed duo Jensen divergences.

Weighted Relative Group Entropies and Associated Fisher Metrics

A large family of new α-weighted group entropy functionals is defined and associated Fisher-like metrics are considered. All these notions are well-suited semi-Riemannian tools for the geometrization of entropy-related statistical models, where they may act as sensitive controlling invariants. The main result of the paper establishes a link between such a metric and a canonical one. A sufficient condition is found, in order that the two metrics be conformal (or homothetic). In particular, we recover a recent result, established for α=1 and for non-weighted relative group entropies. Our conformality condition is “universal”, in the sense that it does not depend on the group exponential.

Universal Bound on Energy Cost of Bit Reset in Finite Time

We consider how the energy cost of bit reset scales with the time duration of the protocol. Bit reset necessarily takes place in finite time, where there is an extra penalty on top of the quasistatic work cost derived by Landauer. This extra energy is dissipated as heat in the computer, inducing a fundamental limit on the speed of irreversible computers. We formulate a hardware-independent expression for this limit in the framework of stochastic processes. We derive a closed-form lower bound on the work penalty as a function of the time taken for the protocol and bit reset error. It holds for discrete as well as continuous systems, assuming only that the master equation respects detailed balance.

Approximate Bayesian Inference

This is the Editorial article summarizing the scope of the Special Issue: Approximate Bayesian Inference.