Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data

Beyond Pearson's Correlation: Modern Nonparametric Independence Tests for Psychological Research

When examining whether two continuous variables are associated, tests based on Pearson's, Kendall's, and Spearman's correlation coefficients are typically used. This paper explores modern nonparametric independence tests as an alternative, which, unlike traditional tests, have the ability to potentially detect any type of relationship. In addition to existing modern nonparametric independence tests, we developed and considered two novel variants of existing tests, most notably the Heller-Heller-Gorfine-Pearson (HHG-Pearson) test. We conducted a simulation study to compare traditional independence tests, such as Pearson's correlation, and the modern nonparametric independence tests in situations commonly encountered in psychological research. As expected, no test had the highest power across all relationships. However, the distance correlation and the HHG-Pearson tests were found to have substantially greater power than all traditional tests for many relationships and only slightly less power in the worst case. A similar pattern was found in favor of the HHG-Pearson test compared to the distance correlation test. However, given that distance correlation performed better for linear relationships and is more widely accepted, we suggest considering its use in place or additional to traditional methods when there is no prior knowledge of the relationship type, as is often the case in psychological research.

Similar teleconnection patterns of ENSO-NAO and ENSO-precipitation in Colombia: linear and non-linear relationships

The Central-Pacific (CP) and Eastern-Pacific (EP) types of El Niño-Southern Oscillation (ENSO) and their ocean-atmosphere effect cause diverse responses in the hydroclimatological patterns of specific regions. Given the impact of ENSO diversity on the North Atlantic Oscillation (NAO), this study aimed to determine the relationship between the ENSO-NAO teleconnection and the ENSO-influenced precipitation patterns in Colombia during the December-February period. Precipitation data from 1981 to 2023, obtained from the Climate Hazards Group (CHIRPS), were analyzed using nine ENSO and NAO indices spanning from 1951 to 2023. Using Pearson's correlation and mutual information (MI) techniques, nine scenarios were devised, encompassing the CP and EP ENSO events, neutral years, and volcanic eruptions. The results suggest a shift in the direction of the ENSO-NAO relationship when distinguishing between the CP and EP events. Higher linear correlations were observed in the CP ENSO scenarios (r > 0.65) using the MEI and BEST indices, while lower correlations were observed when considering EP events along with the Niño 3 and Niño 1.2 indices. MI show difference in relationships based on the event type and the ENSO index used. Notably, an increase in the non-linear relationship was observed for the EP scenarios with respect to correlation. Both teleconnections followed a similar pattern, exhibiting a more substantial impact during CP ENSO events. This highlights the significance of investigating the impacts of ENSO on hydrometeorological variables in the context of adapting to climate change, while acknowledging the intricate diversity inherent to the ENSO phenomenon.

Functional sufficient dimension reduction through information maximization with application to classification

Considering the case where the response variable is a categorical variable and the predictor is a random function, two novel functional sufficient dimensional reduction (FSDR) methods are proposed based on mutual information and square loss mutual information. Compared to the classical FSDR methods, such as functional sliced inverse regression and functional sliced average variance estimation, the proposed methods are appealing because they are capable of estimating multiple effective dimension reduction directions in the case of a relatively small number of categories, especially for the binary response. Moreover, the proposed methods do not require the restrictive linear conditional mean assumption and the constant covariance assumption. They avoid the inverse problem of the covariance operator which is often encountered in the functional sufficient dimension reduction. The functional principal component analysis with truncation be used as a regularization mechanism. Under some mild conditions, the statistical consistency of the proposed methods is established. Simulation studies and real data analyzes are used to evaluate the finite sample properties of our methods.

Mutual information based weighted variance approach for uncertainty quantification of climate projections

Future climate projections are a vital source of information that aid in deriving effective mitigation and adaptation measures. Due to the inherent uncertainty in these climate projections, quantification of uncertainty is essential for increasing its credibility in policymaking. While quantifying the uncertainty, often the possible dependency between the General Circulation Models (GCMs) due to their shared common model code, literature, ideas of representation processes, parameterization schemes, evaluation datasets etc., are ignored. As this will lead to wrong conclusions, the inter-model dependency and the respective independence weights need to be considered, for a realistic quantification of uncertainty. Here, we present the detailed step-wise methodology of a "mutual information based independence weight" framework, that accounts for the linear and nonlinear dependence between GCMs and the equitability property.•A brief illustration of the utility of this method is provided by applying it to the multi-model ensemble of 20 GCMs.•The weighted variance approach seemingly reduces the uncertainty about one GCM given the knowledge of another.

Quantifying information of intracellular signaling: progress with machine learning

Cells convey information about their extracellular environment to their core functional machineries. Studying the capacity of intracellular signaling pathways to transmit information addresses fundamental questions about living systems. Here, we review how information-theoretic approaches have been used to quantify information transmission by signaling pathways that are functionally pleiotropic and subject to molecular stochasticity. We describe how recent advances in machine learning have been leveraged to address the challenges of complex temporal trajectory datasets and how these have contributed to our understanding of how cells employ temporal coding to appropriately adapt to environmental perturbations.

Uniform Partitioning of Data Grid for Association Detection

Inferring appropriate information from large datasets has become important. In particular, identifying relationships among variables in these datasets has far-reaching impacts. In this article, we introduce the uniform information coefficient (UIC), which measures the amount of dependence between two multidimensional variables and is able to detect both linear and non-linear associations. Our proposed UIC is inspired by the maximal information coefficient (MIC) [1].; however, the MIC was originally designed to measure dependence between two one-dimensional variables. Unlike the MIC calculation that depends on the type of association between two variables, we show that the UIC calculation is less computationally expensive and more robust to the type of association between two variables. The UIC achieves this by replacing the dynamic programming step in the MIC calculation with a simpler technique based on the uniform partitioning of the data grid. This computational efficiency comes at the cost of not maximizing the information coefficient as done by the MIC algorithm. We present theoretical guarantees for the performance of the UIC and a variety of experiments to demonstrate its quality in detecting associations.

Connectivity Analysis for Multivariate Time Series: Correlation vs. Causality

The study of the interdependence relationships of the variables of an examined system is of great importance and remains a challenging task. There are two distinct cases of interdependence. In the first case, the variables evolve in synchrony, connections are undirected and the connectivity is examined based on symmetric measures, such as correlation. In the second case, a variable drives another one and they are connected with a causal relationship. Therefore, directed connections entail the determination of the interrelationships based on causality measures. The main open question that arises is the following: can symmetric correlation measures or directional causality measures be applied to infer the connectivity network of an examined system? Using simulations, we demonstrate the performance of different connectivity measures in case of contemporaneous or/and temporal dependencies. Results suggest the sensitivity of correlation measures when temporal dependencies exist in the data. On the other hand, causality measures do not spuriously indicate causal effects when data present only contemporaneous dependencies. Finally, the necessity of introducing effective instantaneous causality measures is highlighted since they are able to handle both contemporaneous and causal effects at the same time. Results based on instantaneous causality measures are promising; however, further investigation is required in order to achieve an overall satisfactory performance.

Dependence between Structural and Electronic Properties of CsPbI3: Unsupervised Machine Learning of Nonadiabatic Molecular Dynamics

Using unsupervised machine learning on the trajectories from a nonadiabatic molecular dynamics simulation with time-dependent Kohn-Sham density functional theory, we elucidated the structural parameters with the largest influence on nonradiative recombination of charge carriers in CsPbI₃, which forms the basis for solar energy and optoelectronic applications. The I-I-I angles between PbI₆ octahedra, followed by the Cs-I distance, have the strongest impact on the bandgap and the nonadiabatic coupling. The importance of the Cs-I distance is unexpected, because Cs does not contribute to electron and hole wave functions. The nonadiabatic coupling is most influenced by static properties, which is also surprising, given its explicit dependence on atomic velocities.

Estimating Transfer Entropy in Continuous Time Between Neural Spike Trains or Other Event-Based Data

Transfer entropy (TE) is a widely used measure of directed information flows in a number of domains including neuroscience. Many real-world time series for which we are interested in information flows come in the form of (near) instantaneous events occurring over time. Examples include the spiking of biological neurons, trades on stock markets and posts to social media, amongst myriad other systems involving events in continuous time throughout the natural and social sciences. However, there exist severe limitations to the current approach to TE estimation on such event-based data via discretising the time series into time bins: it is not consistent, has high bias, converges slowly and cannot simultaneously capture relationships that occur with very fine time precision as well as those that occur over long time intervals. Building on recent work which derived a theoretical framework for TE in continuous time, we present an estimation framework for TE on event-based data and develop a k-nearest-neighbours estimator within this framework. This estimator is provably consistent, has favourable bias properties and converges orders of magnitude more quickly than the current state-of-the-art in discrete-time estimation on synthetic examples. We demonstrate failures of the traditionally-used source-time-shift method for null surrogate generation. In order to overcome these failures, we develop a local permutation scheme for generating surrogate time series conforming to the appropriate null hypothesis in order to test for the statistical significance of the TE and, as such, test for the conditional independence between the history of one point process and the updates of another. Our approach is shown to be capable of correctly rejecting or accepting the null hypothesis of conditional independence even in the presence of strong pairwise time-directed correlations. This capacity to accurately test for conditional independence is further demonstrated on models of a spiking neural circuit inspired by the pyloric circuit of the crustacean stomatogastric ganglion, succeeding where previous related estimators have failed.

Quantifying the Predictability of Visual Scanpaths Using Active Information Storage

Entropy-based measures are an important tool for studying human gaze behavior under various conditions. In particular, gaze transition entropy (GTE) is a popular method to quantify the predictability of a visual scanpath as the entropy of transitions between fixations and has been shown to correlate with changes in task demand or changes in observer state. Measuring scanpath predictability is thus a promising approach to identifying viewers' cognitive states in behavioral experiments or gaze-based applications. However, GTE does not account for temporal dependencies beyond two consecutive fixations and may thus underestimate the actual predictability of the current fixation given past gaze behavior. Instead, we propose to quantify scanpath predictability by estimating the active information storage (AIS), which can account for dependencies spanning multiple fixations. AIS is calculated as the mutual information between a processes' multivariate past state and its next value. It is thus able to measure how much information a sequence of past fixations provides about the next fixation, hence covering a longer temporal horizon. Applying the proposed approach, we were able to distinguish between induced observer states based on estimated AIS, providing first evidence that AIS may be used in the inference of user states to improve human-machine interaction.

Information-Theoretic Representation Learning for Positive-Unlabeled Classification

Recent advances in weakly supervised classification allow us to train a classifier from only positive and unlabeled (PU) data. However, existing PU classification methods typically require an accurate estimate of the class-prior probability, a critical bottleneck particularly for high-dimensional data. This problem has been commonly addressed by applying principal component analysis in advance, but such unsupervised dimension reduction can collapse the underlying class structure. In this letter, we propose a novel representation learning method from PU data based on the information-maximization principle. Our method does not require class-prior estimation and thus can be used as a preprocessing method for PU classification. Through experiments, we demonstrate that our method, combined with deep neural networks, highly improves the accuracy of PU class-prior estimation, leading to state-of-the-art PU classification performance.

A Novel Just-in-Time Learning Strategy for Soft Sensing with Improved Similarity Measure Based on Mutual Information and PLS

In modern industrial process control, just-in-time learning (JITL)-based soft sensors have been widely applied. An accurate similarity measure is crucial in JITL-based soft sensor modeling since it is not only the basis for selecting the nearest neighbor samples but also determines sample weights. In recent years, JITL similarity measure methods have been greatly enriched, including methods based on Euclidean distance, weighted Euclidean distance, correlation, etc. However, due to the different influence of input variables on output, the complex nonlinear relationship between input and output, the collinearity between input variables, and other complex factors, the above similarity measure methods may become inaccurate. In this paper, a new similarity measure method is proposed by combining mutual information (MI) and partial least squares (PLS). A two-stage calculation framework, including a training stage and a prediction stage, was designed in this study to reduce the online computational burden. In the prediction stage, to establish the local model, an improved locally weighted PLS (LWPLS) with variables and samples double-weighted was adopted. The above operations constitute a novel JITL modeling strategy, which is named MI-PLS-LWPLS. By comparison with other related JITL methods, the effectiveness of the MI-PLS-LWPLS method was verified through case studies on both a synthetic Friedman dataset and a real industrial dataset.

Bundled Causal History Interaction

Complex systems arise as a result of the nonlinear interactions between components. In particular, the evolutionary dynamics of a multivariate system encodes the ways in which different variables interact with each other individually or in groups. One fundamental question that remains unanswered is: How do two non-overlapping multivariate subsets of variables interact to causally determine the outcome of a specific variable? Here, we provide an information-based approach to address this problem. We delineate the temporal interactions between the bundles in a probabilistic graphical model. The strength of the interactions, captured by partial information decomposition, then exposes complex behavior of dependencies and memory within the system. The proposed approach successfully illustrated complex dependence between cations and anions as determinants of pH in an observed stream chemistry system. In the studied catchment, the dynamics of pH is a result of both cations and anions through mainly synergistic effects of the two and their individual influences as well. This example demonstrates the potentially broad applicability of the approach, establishing the foundation to study the interaction between groups of variables in a range of complex systems.

Estimating information in time-varying signals

Across diverse biological systems-ranging from neural networks to intracellular signaling and genetic regulatory networks-the information about changes in the environment is frequently encoded in the full temporal dynamics of the network nodes. A pressing data-analysis challenge has thus been to efficiently estimate the amount of information that these dynamics convey from experimental data. Here we develop and evaluate decoding-based estimation methods to lower bound the mutual information about a finite set of inputs, encoded in single-cell high-dimensional time series data. For biological reaction networks governed by the chemical Master equation, we derive model-based information approximations and analytical upper bounds, against which we benchmark our proposed model-free decoding estimators. In contrast to the frequently-used k-nearest-neighbor estimator, decoding-based estimators robustly extract a large fraction of the available information from high-dimensional trajectories with a realistic number of data samples. We apply these estimators to previously published data on Erk and Ca2+ signaling in mammalian cells and to yeast stress-response, and find that substantial amount of information about environmental state can be encoded by non-trivial response statistics even in stationary signals. We argue that these single-cell, decoding-based information estimates, rather than the commonly-used tests for significant differences between selected population response statistics, provide a proper and unbiased measure for the performance of biological signaling networks.

Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback-Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.

K-th Nearest Neighbor (KNN) Entropy Estimates of Complexity and Integration from Ongoing and Stimulus-Evoked Electroencephalographic (EEG) Recordings of the Human Brain

Information-theoretic measures for quantifying multivariate statistical dependence have proven useful for the study of the unity and diversity of the human brain. Two such measures–integration, I(X), and interaction complexity, C_I(X)–have been previously applied to electroencephalographic (EEG) signals recorded during ongoing wakeful brain states. Here, I(X) and C_I(X) were computed for empirical and simulated visually-elicited alpha-range (8–13 Hz) EEG signals. Integration and complexity of evoked (stimulus-locked) and induced (non-stimulus-locked) EEG responses were assessed using nonparametric k-th nearest neighbor (KNN) entropy estimation, which is robust to the nonstationarity of stimulus-elicited EEG signals. KNN-based I(X) and C_I(X) were also computed for the alpha-range EEG of ongoing wakeful brain states. I(X) and C_I(X) patterns differentiated between induced and evoked EEG signals and replicated previous wakeful EEG findings obtained using Gaussian-based entropy estimators. Absolute levels of I(X) and C_I(X) were related to absolute levels of alpha-range EEG power and phase synchronization, but stimulus-related changes in the information-theoretic and other EEG properties were independent. These findings support the hypothesis that visual perception and ongoing wakeful mental states emerge from complex, dynamical interaction among segregated and integrated brain networks operating near an optimal balance between order and disorder.

Separating normosmic and anosmic patients based on entropy evaluation of olfactory event-related potentials

OBJECTIVE

Methods based on electroencephalography (EEG) are used to evaluate brain responses to odors which is challenging due to the relatively low signal-to-noise ratio. This is especially difficult in patients with olfactory loss. In the present study, we aim to establish a method to separate functionally anosmic and normosmic individuals by means of recordings of olfactory event-related potentials (OERP) using an automated tool. Therefore, Shannon entropy was adopted to examine the complexity of the averaged electrophysiological responses.

METHODS

A total of 102 participants received 60 rose-like odorous stimuli at an inter-stimulus interval of 10 s. Olfactory-related brain activity was investigated within three time-windows of equal length; pre-, during-, and post-stimulus.

RESULTS

Based on entropy analysis, patients were correctly diagnosed for anosmia with a 75% success rate.

CONCLUSION

This novel approach can be expected to help clinicians to identify patients with anosmia or patients with early symptoms of neurodegenerative disorders.

SIGNIFICANCE

There is no automated diagnostic tool for anosmic and normosmic patients using OERP. However, detectability of OERP in patients with functional anosmia has been reported to be in the range of 50%.

Optimization of relative parameters in transfer entropy estimation and application to corticomuscular coupling in humans

BACKGROUND

As a non-modeled information theoretical measure, the transfer entropy (TE) could be applied to quantitatively analyze the linear and nonlinear coupling characteristics between two observations. However, the parameters selection of TE (the parameters used in state space reconstruction and estimating Shannon entropy) has a serious influence on the accuracy of its results.

NEW METHOD

In this study, the hybrid particle swarm optimization (HPSO) was applied to improve the accuracy of TE by optimizing its parameters. In HPSO, the TE calculation and significant analysis were integrated into the fitness function, and the optimal parameters group within the parameter space could be automatically found through an iteration process.

RESULTS

The TE results computed under the parameters optimized by HPSO (HPSO-TE), was assessed with a numerical non-linear model, the neural mass model and the recorded electroencephalogram (EEG) and electromyogram (EMG) signals. Compared with TE, HPSO-TE could reduce the 'false positive' in non-linear model, and 'spurious coupling', i.e. two nonzero TEs for unidirectionally coupled systems, especially when coupling strength was weak. The robustness against noise and long time-delay was improved. Moreover, the experimental data analysis showed HPSO-TE revealed the dominant direction (EEG → EMG) in corticomuscular coupling, and had higher values than TE which showed the same dominant direction.

COMPARISON WITH EXISTING METHOD

The implication of HPSO improved the accuracy of TE in estimating the coupling strength and direction.

CONCLUSIONS

The efficiency of TE could be improved by HPSO for estimating coupling relationships, especially for weakly coupled, strong noisy and long time-delay series.

Jackknife approach to the estimation of mutual information

Quantifying the dependence between two random variables is a fundamental issue in data analysis, and thus many measures have been proposed. Recent studies have focused on the renowned mutual information (MI) [Reshef DN, et al. (2011) Science 334:1518-1524]. However, "Unfortunately, reliably estimating mutual information from finite continuous data remains a significant and unresolved problem" [Kinney JB, Atwal GS (2014) Proc Natl Acad Sci USA 111:3354-3359]. In this paper, we examine the kernel estimation of MI and show that the bandwidths involved should be equalized. We consider a jackknife version of the kernel estimate with equalized bandwidth and allow the bandwidth to vary over an interval. We estimate the MI by the largest value among these kernel estimates and establish the associated theoretical underpinnings.

Cross-Sectoral Information Transfer in the Chinese Stock Market around Its Crash in 2015

This paper applies effective transfer entropy to research the information transfer in the Chinese stock market around its crash in 2015. According to the market states, the entire period is divided into four sub-phases: the tranquil, bull, crash, and post-crash periods. Kernel density estimation is used to calculate the effective transfer entropy. Then, the information transfer network is constructed. Nodes’ centralities and the directed maximum spanning trees of the networks are analyzed. The results show that, in the tranquil period, the information transfer is weak in the market. In the bull period, the strength and scope of the information transfer increases. The utility sector outputs a great deal of information and is the hub node for the information flow. In the crash period, the information transfer grows further. The market efficiency in this period is worse than that in the other three sub-periods. The information technology sector is the biggest information source, while the consumer staples sector receives the most information. The interactions of the sectors become more direct. In the post-crash period, information transfer declines but is still stronger than the tranquil time. The financial sector receives the largest amount of information and is the pivot node.

Prediction of microsleeps using pairwise joint entropy and mutual information between EEG channels

Microsleeps are involuntary and brief instances of complete loss of responsiveness, typically of 0.5-15 s duration. They adversely affect performance in extended attention-driven jobs and can be fatal. Our aim was to predict microsleeps from 16 channel EEG signals. Two information theoretic concepts - pairwise joint entropy and mutual information - were independently used to continuously extract features from EEG signals. k-nearest neighbor (kNN) with k = 3 was used to calculate both joint entropy and mutual information. Highly correlated features were discarded and the rest were ranked using Fisher score followed by an average of 3-fold cross-validation area under the curve of the receiver operating characteristic (AUC_ROC). Leave-one-out method (LOOM) was performed to test the performance of microsleep prediction system on independent data. The best prediction for 0.25 s ahead was AUCROC, sensitivity, precision, geometric mean (GM), and φ of 0.93, 0.68, 0.33, 0.75, and 0.38 respectively with joint entropy using single linear discriminant analysis (LDA) classifier.

Information-Theoretic Bounds and Approximations in Neural Population Coding

While Shannon's mutual information has widespread applications in many disciplines, for practical applications it is often difficult to calculate its value accurately for high-dimensional variables because of the curse of dimensionality. This article focuses on effective approximation methods for evaluating mutual information in the context of neural population coding. For large but finite neural populations, we derive several information-theoretic asymptotic bounds and approximation formulas that remain valid in high-dimensional spaces. We prove that optimizing the population density distribution based on these approximation formulas is a convex optimization problem that allows efficient numerical solutions. Numerical simulation results confirmed that our asymptotic formulas were highly accurate for approximating mutual information for large neural populations. In special cases, the approximation formulas are exactly equal to the true mutual information. We also discuss techniques of variable transformation and dimensionality reduction to facilitate computation of the approximations.

Information Entropy Suggests Stronger Nonlinear Associations between Hydro-Meteorological Variables and ENSO

Understanding the teleconnections between hydro-meteorological data and the El Niño-Southern Oscillation cycle (ENSO) is an important step towards developing flood early warning systems. In this study, the concept of mutual information (MI) was applied using marginal and joint information entropy to quantify the linear and non-linear relationship between annual streamflow, extreme precipitation indices over Mekong river basin, and ENSO. We primarily used Pearson correlation as a linear association metric for comparison with mutual information. The analysis was performed at four hydro-meteorological stations located on the mainstream Mekong river basin. It was observed that the nonlinear correlation information is comparatively higher between the large-scale climate index and local hydro-meteorology data in comparison to the traditional linear correlation information. The spatial analysis was carried out using all the grid points in the river basin, which suggests a spatial dependence structure between precipitation extremes and ENSO. Overall, this study suggests that mutual information approach can further detect more meaningful connections between large-scale climate indices and hydro-meteorological variables at different spatio-temporal scales. Application of nonlinear mutual information metric can be an efficient tool to better understand hydro-climatic variables dynamics resulting in improved climate-informed adaptation strategies.

Analysis of Resting-State fMRI Topological Graph Theory Properties in Methamphetamine Drug Users Applying Box-Counting Fractal Dimension

Introduction:

Graph theoretical analysis of functional Magnetic Resonance Imaging (fMRI) data has provided new measures of mapping human brain in vivo. Of all methods to measure the functional connectivity between regions, Linear Correlation (LC) calculation of activity time series of the brain regions as a linear measure is considered the most ubiquitous one. The strength of the dependence obligatory for graph construction and analysis is consistently underestimated by LC, because not all the bivariate distributions, but only the marginals are Gaussian. In a number of studies, Mutual Information (MI) has been employed, as a similarity measure between each two time series of the brain regions, a pure nonlinear measure. Owing to the complex fractal organization of the brain indicating self-similarity, more information on the brain can be revealed by fMRI Fractal Dimension (FD) analysis.

Methods:

In the present paper, Box-Counting Fractal Dimension (BCFD) is introduced for graph theoretical analysis of fMRI data in 17 methamphetamine drug users and 18 normal controls. Then, BCFD performance was evaluated compared to those of LC and MI methods. Moreover, the global topological graph properties of the brain networks inclusive of global efficiency, clustering coefficient and characteristic path length in addict subjects were investigated too.

Results:

Compared to normal subjects by using statistical tests (P<0.05), topological graph properties were postulated to be disrupted significantly during the resting-state fMRI.

Conclusion:

Based on the results, analyzing the graph topological properties (representing the brain networks) based on BCFD is a more reliable method than LC and MI.

Breakdown of local information processing may underlie isoflurane anesthesia effects

The disruption of coupling between brain areas has been suggested as the mechanism underlying loss of consciousness in anesthesia. This hypothesis has been tested previously by measuring the information transfer between brain areas, and by taking reduced information transfer as a proxy for decoupling. Yet, information transfer is a function of the amount of information available in the information source—such that transfer decreases even for unchanged coupling when less source information is available. Therefore, we reconsidered past interpretations of reduced information transfer as a sign of decoupling, and asked whether impaired local information processing leads to a loss of information transfer. An important prediction of this alternative hypothesis is that changes in locally available information (signal entropy) should be at least as pronounced as changes in information transfer. We tested this prediction by recording local field potentials in two ferrets after administration of isoflurane in concentrations of 0.0%, 0.5%, and 1.0%. We found strong decreases in the source entropy under isoflurane in area V1 and the prefrontal cortex (PFC)—as predicted by our alternative hypothesis. The decrease in source entropy was stronger in PFC compared to V1. Information transfer between V1 and PFC was reduced bidirectionally, but with a stronger decrease from PFC to V1. This links the stronger decrease in information transfer to the stronger decrease in source entropy—suggesting reduced source entropy reduces information transfer. This conclusion fits the observation that the synaptic targets of isoflurane are located in local cortical circuits rather than on the synapses formed by interareal axonal projections. Thus, changes in information transfer under isoflurane seem to be a consequence of changes in local processing more than of decoupling between brain areas. We suggest that source entropy changes must be considered whenever interpreting changes in information transfer as decoupling.

Currently we do not understand how anesthesia leads to loss of consciousness (LOC). One popular idea is that we loose consciousness when brain areas lose their ability to communicate with each other–as anesthetics might interrupt transmission on nerve fibers coupling them. This idea has been tested by measuring the amount of information transferred between brain areas, and taking this transfer to reflect the coupling itself. Yet, information that isn’t available in the source area can’t be transferred to a target. Hence, the decreases in information transfer could be related to less information being available in the source, rather than to a decoupling. We tested this possibility measuring the information available in source brain areas and found that it decreased under isoflurane anesthesia. In addition, a stronger decrease in source information lead to a stronger decrease of the information transfered. Thus, the input to the connection between brain areas determined the communicated information, not the strength of the coupling (which would result in a stronger decrease in the target). We suggest that interrupted information processing within brain areas has an important contribution to LOC, and should be focused on more in attempts to understand loss of consciousness under anesthesia.

A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula

We begin by reviewing the statistical framework of information theory as applicable to neuroimaging data analysis. A major factor hindering wider adoption of this framework in neuroimaging is the difficulty of estimating information theoretic quantities in practice. We present a novel estimation technique that combines the statistical theory of copulas with the closed form solution for the entropy of Gaussian variables. This results in a general, computationally efficient, flexible, and robust multivariate statistical framework that provides effect sizes on a common meaningful scale, allows for unified treatment of discrete, continuous, unidimensional and multidimensional variables, and enables direct comparisons of representations from behavioral and brain responses across any recording modality. We validate the use of this estimate as a statistical test within a neuroimaging context, considering both discrete stimulus classes and continuous stimulus features. We also present examples of analyses facilitated by these developments, including application of multivariate analyses to MEG planar magnetic field gradients, and pairwise temporal interactions in evoked EEG responses. We show the benefit of considering the instantaneous temporal derivative together with the raw values of M/EEG signals as a multivariate response, how we can separately quantify modulations of amplitude and direction for vector quantities, and how we can measure the emergence of novel information over time in evoked responses. Open-source Matlab and Python code implementing the new methods accompanies this article. Hum Brain Mapp 38:1541-1573, 2017. © 2016 Wiley Periodicals, Inc.

Discovering Pair-wise Synergies in Microarray Data

Informative gene selection can have important implications for the improvement of cancer diagnosis and the identification of new drug targets. Individual-gene-ranking methods ignore interactions between genes. Furthermore, popular pair-wise gene evaluation methods, e.g. TSP and TSG, are helpless for discovering pair-wise interactions. Several efforts to discover pair-wise synergy have been made based on the information approach, such as EMBP and FeatKNN. However, the methods which are employed to estimate mutual information, e.g. binarization, histogram-based and KNN estimators, depend on known data or domain characteristics. Recently, Reshef et al. proposed a novel maximal information coefficient (MIC) measure to capture a wide range of associations between two variables that has the property of generality. An extension from MIC(X; Y) to MIC(X₁; X₂; Y) is therefore desired. We developed an approximation algorithm for estimating MIC(X₁; X₂; Y) where Y is a discrete variable. MIC(X₁; X₂; Y) is employed to detect pair-wise synergy in simulation and cancer microarray data. The results indicate that MIC(X₁; X₂; Y) also has the property of generality. It can discover synergic genes that are undetectable by reference feature selection methods such as MIC(X; Y) and TSG. Synergic genes can distinguish different phenotypes. Finally, the biological relevance of these synergic genes is validated with GO annotation and OUgene database.

Inferring time-varying brain connectivity graph based on a new method for link estimation

Causal interaction estimation among neuronal groups plays an important role in the assessment of brain functions. These directional relations can be best illustrated by means of graphical modeling which is a mathematical representation of a network. Here, we propose an efficient framework to derive a graphical model for the statistical analysis of multivariate processes from observed time series in a data-driven pipeline to explore the interregional brain interactions. A major part of this analysis is devoted to the graph link estimation, which is a measure capable of dealing with the multivariate analysis obstacles. In this paper, we use the Transfer Entropy (TE) measure and focus on its calculation that requires efficient estimation of high dimensional conditional probability distributions. Our method is based on the simplification of high dimensional parts of the conventional TE definition and especially devoted to the reduction of estimation dimension through searching for the most informative contents of the high dimensional parts. To this end, we exploit the causal Markov properties for time series graphs and prove that only a specified subset of involved variables plays an important role in multivariate TE estimation. We demonstrate the performance of our method for stationary processes using some numerical simulated examples as well as real neurophysiological data.

Detecting Microbial Dysbiosis Associated with Pediatric Crohn Disease Despite the High Variability of the Gut Microbiota

The relationship between the host and its microbiota is challenging to understand because both microbial communities and their environments are highly variable. We have developed a set of techniques based on population dynamics and information theory to address this challenge. These methods identify additional bacterial taxa associated with pediatric Crohn disease and can detect significant changes in microbial communities with fewer samples than previous statistical approaches required. We have also substantially improved the accuracy of the diagnosis based on the microbiota from stool samples, and we found that the ecological niche of a microbe predicts its role in Crohn disease. Bacteria typically residing in the lumen of healthy individuals decrease in disease, whereas bacteria typically residing on the mucosa of healthy individuals increase in disease. Our results also show that the associations with Crohn disease are evolutionarily conserved and provide a mutual information-based method to depict dysbiosis.

Resected Brain Tissue, Seizure Onset Zone and Quantitative EEG Measures: Towards Prediction of Post-Surgical Seizure Control

BACKGROUND

Epilepsy surgery is a potentially curative treatment option for pharmacoresistent patients. If non-invasive methods alone do not allow to delineate the epileptogenic brain areas the surgical candidates undergo long-term monitoring with intracranial EEG. Visual EEG analysis is then used to identify the seizure onset zone for targeted resection as a standard procedure.

METHODS

Despite of its great potential to assess the epileptogenicty of brain tissue, quantitative EEG analysis has not yet found its way into routine clinical practice. To demonstrate that quantitative EEG may yield clinically highly relevant information we retrospectively investigated how post-operative seizure control is associated with four selected EEG measures evaluated in the resected brain tissue and the seizure onset zone. Importantly, the exact spatial location of the intracranial electrodes was determined by coregistration of pre-operative MRI and post-implantation CT and coregistration with post-resection MRI was used to delineate the extent of tissue resection. Using data-driven thresholding, quantitative EEG results were separated into normally contributing and salient channels.

RESULTS

In patients with favorable post-surgical seizure control a significantly larger fraction of salient channels in three of the four quantitative EEG measures was resected than in patients with unfavorable outcome in terms of seizure control (median over the whole peri-ictal recordings). The same statistics revealed no association with post-operative seizure control when EEG channels contributing to the seizure onset zone were studied.

CONCLUSIONS

We conclude that quantitative EEG measures provide clinically relevant and objective markers of target tissue, which may be used to optimize epilepsy surgery. The finding that differentiation between favorable and unfavorable outcome was better for the fraction of salient values in the resected brain tissue than in the seizure onset zone is consistent with growing evidence that spatially extended networks might be more relevant for seizure generation, evolution and termination than a single highly localized brain region (i.e. a "focus") where seizures start.

Procedures for the Comparative Testing of Noninvasive Neuroassessment Devices

A sequential process for comparison testing of noninvasive neuroassessment devices is presented. Comparison testing of devices in a clinical population should be preceded by computational research and reliability testing with healthy populations, as opposed to proceeding immediately to testing with clinical participants. A five-step process is outlined as follows: 1. Complete a preliminary literature review identifying candidate measures. 2. Conduct systematic simulation studies to determine the computational properties and data requirements of candidate measures. 3. Establish the test-retest reliability of each measure in a healthy comparison population and the clinical population of interest. 4. Investigate the clinical validity of reliable measures in appropriately defined clinical populations. 5. Complete device usability assessment (weight, simplicity of use, cost effectiveness, ruggedness) only for devices and measures that are promising after steps 1 through 4 are completed. Usability may be considered throughout the device evaluation process but such considerations are subordinate to the higher priorities addressed in steps 1 through 4.

Objective reduction based on nonlinear correlation information entropy

It is hard to obtain the entire solution set of a many-objective optimization problem (MaOP) by multi-objective evolutionary algorithms (MOEAs) because of the difficulties brought by the large number of objectives. However, the redundancy of objectives exists in some problems with correlated objectives (linearly or nonlinearly). Objective reduction can be used to decrease the difficulties of some MaOPs. In this paper, we propose a novel objective reduction approach based on nonlinear correlation information entropy (NCIE). It uses the NCIE matrix to measure the linear and nonlinear correlation between objectives and a simple method to select the most conflicting objectives during the execution of MOEAs. We embed our approach into both Pareto-based and indicator-based MOEAs to analyze the impact of our reduction method on the performance of these algorithms. The results show that our approach significantly improves the performance of Pareto-based MOEAs on both reducible and irreducible MaOPs, but does not much help the performance of indicator-based MOEAs.

Equitability, mutual information, and the maximal information coefficient

How should one quantify the strength of association between two random variables without bias for relationships of a specific form? Despite its conceptual simplicity, this notion of statistical "equitability" has yet to receive a definitive mathematical formalization. Here we argue that equitability is properly formalized by a self-consistency condition closely related to Data Processing Inequality. Mutual information, a fundamental quantity in information theory, is shown to satisfy this equitability criterion. These findings are at odds with the recent work of Reshef et al. [Reshef DN, et al. (2011) Science 334(6062):1518-1524], which proposed an alternative definition of equitability and introduced a new statistic, the "maximal information coefficient" (MIC), said to satisfy equitability in contradistinction to mutual information. These conclusions, however, were supported only with limited simulation evidence, not with mathematical arguments. Upon revisiting these claims, we prove that the mathematical definition of equitability proposed by Reshef et al. cannot be satisfied by any (nontrivial) dependence measure. We also identify artifacts in the reported simulation evidence. When these artifacts are removed, estimates of mutual information are found to be more equitable than estimates of MIC. Mutual information is also observed to have consistently higher statistical power than MIC. We conclude that estimating mutual information provides a natural (and often practical) way to equitably quantify statistical associations in large datasets.

Time domain measures of inter-channel EEG correlations: a comparison of linear, nonparametric and nonlinear measures

Correlations between ten-channel EEGs obtained from thirteen healthy adult participants were investigated. Signals were obtained in two behavioral states: eyes open no task and eyes closed no task. Four time domain measures were compared: Pearson product moment correlation, Spearman rank order correlation, Kendall rank order correlation and mutual information. The psychophysiological utility of each measure was assessed by determining its ability to discriminate between conditions. The sensitivity to epoch length was assessed by repeating calculations with 1, 2, 3, …, 8 s epochs. The robustness to noise was assessed by performing calculations with noise corrupted versions of the original signals (SNRs of 0, 5 and 10 dB). Three results were obtained in these calculations. First, mutual information effectively discriminated between states with less data. Pearson, Spearman and Kendall failed to discriminate between states with a 1 s epoch, while a statistically significant separation was obtained with mutual information. Second, at all epoch durations tested, the measure of between-state discrimination was greater for mutual information. Third, discrimination based on mutual information was more robust to noise. The limitations of this study are discussed. Further comparisons should be made with frequency domain measures, with measures constructed with embedded data and with the maximal information coefficient.

Parametric inference in the large data limit using maximally informative models

Motivated by data-rich experiments in transcriptional regulation and sensory neuroscience, we consider the following general problem in statistical inference: when exposed to a high-dimensional signal S, a system of interest computes a representation R of that signal, which is then observed through a noisy measurement M. From a large number of signals and measurements, we wish to infer the "filter" that maps S to R. However, the standard method for solving such problems, likelihood-based inference, requires perfect a priori knowledge of the "noise function" mapping R to M. In practice such noise functions are usually known only approximately, if at all, and using an incorrect noise function will typically bias the inferred filter. Here we show that in the large data limit, this need for a precharacterized noise function can be circumvented by searching for filters that instead maximize the mutual information I[M; R] between observed measurements and predicted representations. Moreover, if the correct filter lies within the space of filters being explored, maximizing mutual information becomes equivalent to simultaneously maximizing every dependence measure that satisfies the data processing inequality. It is important to note that maximizing mutual information will typically leave a small number of directions in parameter space unconstrained. We term these directions diffeomorphic modes and present an equation that allows these modes to be derived systematically. The presence of diffeomorphic modes reflects a fundamental and nontrivial substructure within parameter space, one that is obscured by standard likelihood-based inference.

To control false positives in gene-gene interaction analysis: two novel conditional entropy-based approaches

Genome-wide analysis of gene-gene interactions has been recognized as a powerful avenue to identify the missing genetic components that can not be detected by using current single-point association analysis. Recently, several model-free methods (e.g. the commonly used information based metrics and several logistic regression-based metrics) were developed for detecting non-linear dependence between genetic loci, but they are potentially at the risk of inflated false positive error, in particular when the main effects at one or both loci are salient. In this study, we proposed two conditional entropy-based metrics to challenge this limitation. Extensive simulations demonstrated that the two proposed metrics, provided the disease is rare, could maintain consistently correct false positive rate. In the scenarios for a common disease, our proposed metrics achieved better or comparable control of false positive error, compared to four previously proposed model-free metrics. In terms of power, our methods outperformed several competing metrics in a range of common disease models. Furthermore, in real data analyses, both metrics succeeded in detecting interactions and were competitive with the originally reported results or the logistic regression approaches. In conclusion, the proposed conditional entropy-based metrics are promising as alternatives to current model-based approaches for detecting genuine epistatic effects.

Spatial and Electronic Correlations in the PE545 Light-Harvesting Complex

The recent discovery of long-lasting quantum coherence effects in photosynthetic pigment-protein complexes has challenged our view of the role that protein motions play in light-harvesting processes. Several groups have suggested that correlated fluctuations involving the pigments site energies and couplings could be at the origin of such unexpected behavior. Here we combine molecular dynamics simulations with quantum mechanics/molecular mechanics calculations to analyze the degree of correlated fluctuations in the PE545 complex of Rhodomonas sp. strain CS24. We find that correlations between the motions of the chromophores, which are significantly assisted by the water solvent, do not translate into appreciable site energy correlations but do lead to significant cross-correlations of energies and couplings. Such behavior, not observed in a recent study on the Fenna-Mathews-Olson complex, seems to provide phycobiliproteins with an additional fundamental mechanism to control quantum coherence and light-harvesting efficiency compared with chlorophyll-containing complexes.

Transfer entropy estimation and directional coupling change detection in biomedical time series

Background

The detection of change in magnitude of directional coupling between two non-linear time series is a common subject of interest in the biomedical domain, including studies involving the respiratory chemoreflex system. Although transfer entropy is a useful tool in this avenue, no study to date has investigated how different transfer entropy estimation methods perform in typical biomedical applications featuring small sample size and presence of outliers.

Methods

With respect to detection of increased coupling strength, we compared three transfer entropy estimation techniques using both simulated time series and respiratory recordings from lambs. The following estimation methods were analyzed: fixed-binning with ranking, kernel density estimation (KDE), and the Darbellay-Vajda (D-V) adaptive partitioning algorithm extended to three dimensions. In the simulated experiment, sample size was varied from 50 to 200, while coupling strength was increased. In order to introduce outliers, the heavy-tailed Laplace distribution was utilized. In the lamb experiment, the objective was to detect increased respiratory-related chemosensitivity to O₂ and CO₂ induced by a drug, domperidone. Specifically, the separate influence of end-tidal PO₂ and PCO₂ on minute ventilation (V˙E) before and after administration of domperidone was analyzed.

Results

In the simulation, KDE detected increased coupling strength at the lowest SNR among the three methods. In the lamb experiment, D-V partitioning resulted in the statistically strongest increase in transfer entropy post-domperidone for PO2→V˙E. In addition, D-V partitioning was the only method that could detect an increase in transfer entropy for PCO2→V˙E, in agreement with experimental findings.

Conclusions

Transfer entropy is capable of detecting directional coupling changes in non-linear biomedical time series analysis featuring a small number of observations and presence of outliers. The results of this study suggest that fixed-binning, even with ranking, is too primitive, and although there is no clear winner between KDE and D-V partitioning, the reader should note that KDE requires more computational time and extensive parameter selection than D-V partitioning. We hope this study provides a guideline for selection of an appropriate transfer entropy estimation method.