Transfer entropy in physical systems and the arrow of time

A Comparative Study of Causality Detection Methods in Root Cause Diagnosis: From Industrial Processes to Brain Networks

Abstracting causal knowledge from process measurements has become an appealing topic for decades, especially for fault root cause analysis (RCA) based on signals recorded by multiple sensors in a complex system. Although many causality detection methods have been developed and applied in different fields, some research communities may have an idiosyncratic implementation of their preferred methods, with limited accessibility to the wider community. Targeting interested experimental researchers and engineers, this paper provides a comprehensive comparison of data-based causality detection methods in root cause diagnosis across two distinct domains. We provide a possible taxonomy of those methods followed by descriptions of the main motivations of those concepts. Of the two cases we investigated, one is a root cause diagnosis of plant-wide oscillations in an industrial process, while the other is the localization of the epileptogenic focus in a human brain network where the connectivity pattern is transient and even more complex. Considering the differences in various causality detection methods, we designed several sets of experiments so that for each case, a total of 11 methods could be appropriately compared under a unified and reasonable evaluation framework. In each case, these methods were implemented separately and in a standard way to infer causal interactions among multiple variables to thus establish the causal network for RCA. From the cross-domain investigation, several findings are presented along with insights into them, including an interpretative pitfall that warrants caution.

New fluctuation theorems on Maxwell's demon

With increasing interest in the control of systems at the nano- and mesoscopic scales, studies have been focused on the limit of the energy dissipation in an open system by refining the concept of the Maxwell's demon. To uncover the underlying physical principle behind a system controlled by a demon, we prove a previously unexplored set of fluctuation theorems. These fluctuation theorems imply that there exists an intrinsic nonequilibrium state of the system, led by the nonnegative demon-induced dissipative information. A consequence of this analysis is that the bounds of both work and heat are tighter than the limits predicted by the Sagawa-Ueda theorem. We also suggest a possible experimental test of these work and heat bounds.

A Maximum Entropy Model of Bounded Rational Decision-Making with Prior Beliefs and Market Feedback

Bounded rationality is an important consideration stemming from the fact that agents often have limits on their processing abilities, making the assumption of perfect rationality inapplicable to many real tasks. We propose an information-theoretic approach to the inference of agent decisions under Smithian competition. The model explicitly captures the boundedness of agents (limited in their information-processing capacity) as the cost of information acquisition for expanding their prior beliefs. The expansion is measured as the Kullblack–Leibler divergence between posterior decisions and prior beliefs. When information acquisition is free, the homo economicus agent is recovered, while in cases when information acquisition becomes costly, agents instead revert to their prior beliefs. The maximum entropy principle is used to infer least biased decisions based upon the notion of Smithian competition formalised within the Quantal Response Statistical Equilibrium framework. The incorporation of prior beliefs into such a framework allowed us to systematically explore the effects of prior beliefs on decision-making in the presence of market feedback, as well as importantly adding a temporal interpretation to the framework. We verified the proposed model using Australian housing market data, showing how the incorporation of prior knowledge alters the resulting agent decisions. Specifically, it allowed for the separation of past beliefs and utility maximisation behaviour of the agent as well as the analysis into the evolution of agent beliefs.

Neural Estimator of Information for Time-Series Data with Dependency

Novel approaches to estimate information measures using neural networks are well-celebrated in recent years both in the information theory and machine learning communities. These neural-based estimators are shown to converge to the true values when estimating mutual information and conditional mutual information using independent samples. However, if the samples in the dataset are not independent, the consistency of these estimators requires further investigation. This is of particular interest for a more complex measure such as the directed information, which is pivotal in characterizing causality and is meaningful over time-dependent variables. The extension of the convergence proof for such cases is not trivial and demands further assumptions on the data. In this paper, we show that our neural estimator for conditional mutual information is consistent when the dataset is generated with samples of a stationary and ergodic source. In other words, we show that our information estimator using neural networks converges asymptotically to the true value with probability one. Besides universal functional approximation of neural networks, a core lemma to show the convergence is Birkhoff’s ergodic theorem. Additionally, we use the technique to estimate directed information and demonstrate the effectiveness of our approach in simulations.

Thermodynamic uncertainty relations including measurement and feedback

Thermodynamic uncertainty relations quantify how the signal-to-noise ratio of a given observable is constrained by dissipation. Fluctuation relations generalize the second law of thermodynamics to stochastic processes. We show that any fluctuation relation directly implies a thermodynamic uncertainty relation, considerably increasing their range of applicability. In particular, we extend thermodynamic uncertainty relations to scenarios which include measurement and feedback. Since feedback generally breaks time-reversal invariance, the uncertainty relations involve quantities averaged over the forward and the backward experiment defined by the associated fluctuation relation. This implies that the signal-to-noise ratio of a given experiment can in principle become arbitrarily large as long as the corresponding backward experiment compensates, e.g., by being sufficiently noisy. We illustrate our results with the Szilard engine as well as work extraction by free energy reduction in a quantum dot.

Information Thermodynamics for Time Series of Signal-Response Models

The entropy production in stochastic dynamical systems is linked to the structure of their causal representation in terms of Bayesian networks. Such a connection was formalized for bipartite (or multipartite) systems with an integral fluctuation theorem in [Phys. Rev. Lett. 111, 180603 (2013)]. Here we introduce the information thermodynamics for time series, that are non-bipartite in general, and we show that the link between irreversibility and information can only result from an incomplete causal representation. In particular, we consider a backward transfer entropy lower bound to the conditional time series irreversibility that is induced by the absence of feedback in signal-response models. We study such a relation in a linear signal-response model providing analytical solutions, and in a nonlinear biological model of receptor-ligand systems where the time series irreversibility measures the signaling efficiency.

Thermodynamic efficiency of contagions: a statistical mechanical analysis of the SIS epidemic model

We present a novel approach to the study of epidemics on networks as thermodynamic phenomena, quantifying the thermodynamic efficiency of contagions, considered as distributed computational processes. Modelling SIS dynamics on a contact network statistical-mechanically, we follow the maximum entropy (MaxEnt) principle to obtain steady-state distributions and derive, under certain assumptions, relevant thermodynamic quantities both analytically and numerically. In particular, we obtain closed-form solutions for some cases, while interpreting key epidemic variables, such as the reproductive ratio of a SIS model, in a statistical mechanical setting. On the other hand, we consider configuration and free entropy, as well as the Fisher information, in the epidemiological context. This allowed us to identify criticality and distinct phases of epidemic processes. For each of the considered thermodynamic quantities, we compare the analytical solutions informed by the MaxEnt principle with the numerical estimates for SIS epidemics simulated on Watts-Strogatz random graphs.

Semantic information, autonomous agency and non-equilibrium statistical physics

Shannon information theory provides various measures of so-called syntactic information, which reflect the amount of statistical correlation between systems. By contrast, the concept of 'semantic information' refers to those correlations which carry significance or 'meaning' for a given system. Semantic information plays an important role in many fields, including biology, cognitive science and philosophy, and there has been a long-standing interest in formulating a broadly applicable and formal theory of semantic information. In this paper, we introduce such a theory. We define semantic information as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. 'Causal necessity' is defined in terms of counter-factual interventions which scramble correlations between the system and its environment, while 'maintaining existence' is defined in terms of the system's ability to keep itself in a low entropy state. We also use recent results in non-equilibrium statistical physics to analyse semantic information from a thermodynamic point of view. Our framework is grounded in the intrinsic dynamics of a system coupled to an environment, and is applicable to any physical system, living or otherwise. It leads to formal definitions of several concepts that have been intuitively understood to be related to semantic information, including 'value of information', 'semantic content' and 'agency'.

Detailed Fluctuation Relation for Arbitrary Measurement and Feedback Schemes

Fluctuation relations are powerful equalities that hold far from equilibrium. However, the standard approach to include measurement and feedback schemes may become inapplicable in certain situations, including continuous measurements, precise measurements of continuous variables, and feedback induced irreversibility. Here we overcome these shortcomings by providing a recipe for producing detailed fluctuation relations. Based on this recipe, we derive a fluctuation relation which holds for arbitrary measurement and feedback control. The key insight is that fluctuations inferable from the measurement outcomes may be suppressed by postselection. Our detailed fluctuation relation results in a stringent and experimentally accessible inequality on the extractable work, which is saturated when the full entropy production is inferable from the data.

Stochastic Thermodynamic Interpretation of Information Geometry

In recent years, the unified theory of information and thermodynamics has been intensively discussed in the context of stochastic thermodynamics. The unified theory reveals that information theory would be useful to understand nonstationary dynamics of systems far from equilibrium. In this Letter, we have found a new link between stochastic thermodynamics and information theory well-known as information geometry. By applying this link, an information geometric inequality can be interpreted as a thermodynamic uncertainty relationship between speed and thermodynamic cost. We have numerically applied an information geometric inequality to a thermodynamic model of a biochemical enzyme reaction.

Thermodynamics and computation during collective motion near criticality

We study self-organization of collective motion as a thermodynamic phenomenon in the context of the first law of thermodynamics. It is expected that the coherent ordered motion typically self-organises in the presence of changes in the (generalized) internal energy and of (generalized) work done on, or extracted from, the system. We aim to explicitly quantify changes in these two quantities in a system of simulated self-propelled particles and contrast them with changes in the system's configuration entropy. In doing so, we adapt a thermodynamic formulation of the curvatures of the internal energy and the work, with respect to two parameters that control the particles' alignment. This allows us to systematically investigate the behavior of the system by varying the two control parameters to drive the system across a kinetic phase transition. Our results identify critical regimes and show that during the phase transition, where the configuration entropy of the system decreases, the rates of change of the work and of the internal energy also decrease, while their curvatures diverge. Importantly, the reduction of entropy achieved through expenditure of work is shown to peak at criticality. We relate this both to a thermodynamic efficiency and the significance of the increased order with respect to a computational path. Additionally, this study provides an information-geometric interpretation of the curvature of the internal energy as the difference between two curvatures: the curvature of the free entropy, captured by the Fisher information, and the curvature of the configuration entropy.

Minimising the Kullback-Leibler Divergence for Model Selection in Distributed Nonlinear Systems

The Kullback-Leibler (KL) divergence is a fundamental measure of information geometry that is used in a variety of contexts in artificial intelligence. We show that, when system dynamics are given by distributed nonlinear systems, this measure can be decomposed as a function of two information-theoretic measures, transfer entropy and stochastic interaction. More specifically, these measures are applicable when selecting a candidate model for a distributed system, where individual subsystems are coupled via latent variables and observed through a filter. We represent this model as a directed acyclic graph (DAG) that characterises the unidirectional coupling between subsystems. Standard approaches to structure learning are not applicable in this framework due to the hidden variables; however, we can exploit the properties of certain dynamical systems to formulate exact methods based on differential topology. We approach the problem by using reconstruction theorems to derive an analytical expression for the KL divergence of a candidate DAG from the observed dataset. Using this result, we present a scoring function based on transfer entropy to be used as a subroutine in a structure learning algorithm. We then demonstrate its use in recovering the structure of coupled Lorenz and Rössler systems.

Transfer entropy in continuous time, with applications to jump and neural spiking processes

Transfer entropy has been used to quantify the directed flow of information between source and target variables in many complex systems. While transfer entropy was originally formulated in discrete time, in this paper we provide a framework for considering transfer entropy in continuous time systems, based on Radon-Nikodym derivatives between measures of complete path realizations. To describe the information dynamics of individual path realizations, we introduce the pathwise transfer entropy, the expectation of which is the transfer entropy accumulated over a finite time interval. We demonstrate that this formalism permits an instantaneous transfer entropy rate. These properties are analogous to the behavior of physical quantities defined along paths such as work and heat. We use this approach to produce an explicit form for the transfer entropy for pure jump processes, and highlight the simplified form in the specific case of point processes (frequently used in neuroscience to model neural spike trains). Finally, we present two synthetic spiking neuron model examples to exhibit the pertinent features of our formalism, namely, that the information flow for point processes consists of discontinuous jump contributions (at spikes in the target) interrupting a continuously varying contribution (relating to waiting times between target spikes). Numerical schemes based on our formalism promise significant benefits over existing strategies based on discrete time formalisms.

Accuracy of rate coding: When shorter time window and higher spontaneous activity help

It is widely accepted that neuronal firing rates contain a significant amount of information about the stimulus intensity. Nevertheless, theoretical studies on the coding accuracy inferred from the exact spike counting distributions are rare. We present an analysis based on the number of observed spikes assuming the stochastic perfect integrate-and-fire model with a change point, representing the stimulus onset, for which we calculate the corresponding Fisher information to investigate the accuracy of rate coding. We analyze the effect of changing the duration of the time window and the influence of several parameters of the model, in particular the level of the presynaptic spontaneous activity and the level of random fluctuation of the membrane potential, which can be interpreted as noise of the system. The results show that the Fisher information is nonmonotonic with respect to the length of the observation period. This counterintuitive result is caused by the discrete nature of the count of spikes. We observe also that the signal can be enhanced by noise, since the Fisher information is nonmonotonic with respect to the level of spontaneous activity and, in some cases, also with respect to the level of fluctuation of the membrane potential.