Information estimation using nonparametric copulas

Specialized structure of neural population codes in parietal cortex outputs

Do cortical neurons that send axonal projections to the same target area form specialized population codes for transmitting information? We used calcium imaging in mouse posterior parietal cortex (PPC), retrograde labeling, and statistical multivariate models to address this question during a delayed match-to-sample task. We found that PPC broadcasts sensory, choice, and locomotion signals widely, but sensory information is enriched in the output to anterior cingulate cortex. Neurons projecting to the same area have elevated pairwise activity correlations. These correlations are structured as information-limiting and information-enhancing interaction networks that collectively enhance information levels. This network structure is unique to sub-populations projecting to the same target and strikingly absent in surrounding neural populations with unidentified projections. Furthermore, this structure is only present when mice make correct, but not incorrect, behavioral choices. Therefore, cortical neurons comprising an output pathway form uniquely structured population codes that enhance information transmission to guide accurate behavior.

An information-theoretic quantification of the content of communication between brain regions

Quantifying the amount, content and direction of communication between brain regions is key to understanding brain function. Traditional methods to analyze brain activity based on the Wiener-Granger causality principle quantify the overall information propagated by neural activity between simultaneously recorded brain regions, but do not reveal the information flow about specific features of interest (such as sensory stimuli). Here, we develop a new information theoretic measure termed Feature-specific Information Transfer (FIT), quantifying how much information about a specific feature flows between two regions. FIT merges the Wiener-Granger causality principle with information-content specificity. We first derive FIT and prove analytically its key properties. We then illustrate and test them with simulations of neural activity, demonstrating that FIT identifies, within the total information flowing between regions, the information that is transmitted about specific features. We then analyze three neural datasets obtained with different recording methods, magneto- and electro-encephalography, and spiking activity, to demonstrate the ability of FIT to uncover the content and direction of information flow between brain regions beyond what can be discerned with traditional anaytical methods. FIT can improve our understanding of how brain regions communicate by uncovering previously hidden feature-specific information flow.

The Role of Thermodynamic and Informational Entropy in Improving Real Estate Valuation Methods

Price, Cost and Income (PCI) are distinct economic indicators intrinsically linked to the values they denote. These observables take center stage in the multi-criteria decision-making process that enables economic agents to convey subjective utilities of market-exchanged commodities objectively. The valuation of these commodities heavily relies on PCI-based empirical observables and their supported methodologies. This valuation measure's accuracy is critical, as it influences subsequent decisions within the market chain. However, measurement errors often arise due to inherent uncertainties in the value state, impacting economic agents' wealth, particularly when trading significant commodities such as real estate properties. This paper addresses this issue by incorporating entropy measurements into real estate valuation. This mathematical technique adjusts and integrates triadic PCI estimates, improving the final stage of appraisal systems where definitive value decisions are crucial. Employing entropy within the appraisal system can also aid market agents in devising informed production/trading strategies for optimal returns. The results from our practical demonstration indicate promising implications. The entropy's integration with PCI estimates significantly improved the value measurement's precision and reduced economic decision-making errors.

Computational methods to study information processing in neural circuits

The brain is an information processing machine and thus naturally lends itself to be studied using computational tools based on the principles of information theory. For this reason, computational methods based on or inspired by information theory have been a cornerstone of practical and conceptual progress in neuroscience. In this Review, we address how concepts and computational tools related to information theory are spurring the development of principled theories of information processing in neural circuits and the development of influential mathematical methods for the analyses of neural population recordings. We review how these computational approaches reveal mechanisms of essential functions performed by neural circuits. These functions include efficiently encoding sensory information and facilitating the transmission of information to downstream brain areas to inform and guide behavior. Finally, we discuss how further progress and insights can be achieved, in particular by studying how competing requirements of neural encoding and readout may be optimally traded off to optimize neural information processing.

The structures and functions of correlations in neural population codes

The collective activity of a population of neurons, beyond the properties of individual cells, is crucial for many brain functions. A fundamental question is how activity correlations between neurons affect how neural populations process information. Over the past 30 years, major progress has been made on how the levels and structures of correlations shape the encoding of information in population codes. Correlations influence population coding through the organization of pairwise-activity correlations with respect to the similarity of tuning of individual neurons, by their stimulus modulation and by the presence of higher-order correlations. Recent work has shown that correlations also profoundly shape other important functions performed by neural populations, including generating codes across multiple timescales and facilitating information transmission to, and readout by, downstream brain areas to guide behaviour. Here, we review this recent work and discuss how the structures of correlations can have opposite effects on the different functions of neural populations, thus creating trade-offs and constraints for the structure-function relationships of population codes. Further, we present ideas on how to combine large-scale simultaneous recordings of neural populations, computational models, analyses of behaviour, optogenetics and anatomy to unravel how the structures of correlations might be optimized to serve multiple functions.

Ranking the information content of distance measures

Real-world data typically contain a large number of features that are often heterogeneous in nature, relevance, and also units of measure. When assessing the similarity between data points, one can build various distance measures using subsets of these features. Finding a small set of features that still retains sufficient information about the dataset is important for the successful application of many statistical learning approaches. We introduce a statistical test that can assess the relative information retained when using 2 different distance measures, and determine if they are equivalent, independent, or if one is more informative than the other. This ranking can in turn be used to identify the most informative distance measure and, therefore, the most informative set of features, out of a pool of candidates. To illustrate the general applicability of our approach, we show that it reproduces the known importance ranking of policy variables for Covid-19 control, and also identifies compact yet informative descriptors for atomic structures. We further provide initial evidence that the information asymmetry measured by the proposed test can be used to infer relationships of causality between the features of a dataset. The method is general and should be applicable to many branches of science.

Multivariate Gaussian Copula Mutual Information to Estimate Functional Connectivity with Less Random Architecture

Recognition of a brain region’s interaction is an important field in neuroscience. Most studies use the Pearson correlation to find the interaction between the regions. According to the experimental evidence, there is a nonlinear dependence between the activities of different brain regions that is ignored by Pearson correlation as a linear measure. Typically, the average activity of each region is used as input because it is a univariate measure. This dimensional reduction, i.e., averaging, leads to a loss of spatial information across voxels within the region. In this study, we propose using an information-theoretic measure, multivariate mutual information (mvMI), as a nonlinear dependence to find the interaction between regions. This measure, which has been recently proposed, simplifies the mutual information calculation complexity using the Gaussian copula. Using simulated data, we show that the using this measure overcomes the mentioned limitations. Additionally using the real resting-state fMRI data, we compare the level of significance and randomness of graphs constructed using different methods. Our results indicate that the proposed method estimates the functional connectivity more significantly and leads to a smaller number of random connections than the common measure, Pearson correlation. Moreover, we find that the similarity of the estimated functional networks of the individuals is higher when the proposed method is used.

Inferring a Property of a Large System from a Small Number of Samples

Inferring the value of a property of a large stochastic system is a difficult task when the number of samples is insufficient to reliably estimate the probability distribution. The Bayesian estimator of the property of interest requires the knowledge of the prior distribution, and in many situations, it is not clear which prior should be used. Several estimators have been developed so far in which the proposed prior us individually tailored for each property of interest; such is the case, for example, for the entropy, the amount of mutual information, or the correlation between pairs of variables. In this paper, we propose a general framework to select priors that is valid for arbitrary properties. We first demonstrate that only certain aspects of the prior distribution actually affect the inference process. We then expand the sought prior as a linear combination of a one-dimensional family of indexed priors, each of which is obtained through a maximum entropy approach with constrained mean values of the property under study. In many cases of interest, only one or very few components of the expansion turn out to contribute to the Bayesian estimator, so it is often valid to only keep a single component. The relevant component is selected by the data, so no handcrafted priors are required. We test the performance of this approximation with a few paradigmatic examples and show that it performs well in comparison to the ad-hoc methods previously proposed in the literature. Our method highlights the connection between Bayesian inference and equilibrium statistical mechanics, since the most relevant component of the expansion can be argued to be that with the right temperature.

Parametric Copula-GP model for analyzing multidimensional neuronal and behavioral relationships

One of the main goals of current systems neuroscience is to understand how neuronal populations integrate sensory information to inform behavior. However, estimating stimulus or behavioral information that is encoded in high-dimensional neuronal populations is challenging. We propose a method based on parametric copulas which allows modeling joint distributions of neuronal and behavioral variables characterized by different statistics and timescales. To account for temporal or spatial changes in dependencies between variables, we model varying copula parameters by means of Gaussian Processes (GP). We validate the resulting Copula-GP framework on synthetic data and on neuronal and behavioral recordings obtained in awake mice. We show that the use of a parametric description of the high-dimensional dependence structure in our method provides better accuracy in mutual information estimation in higher dimensions compared to other non-parametric methods. Moreover, by quantifying the redundancy between neuronal and behavioral variables, our model exposed the location of the reward zone in an unsupervised manner (i.e., without using any explicit cues about the task structure). These results demonstrate that the Copula-GP framework is particularly useful for the analysis of complex multidimensional relationships between neuronal, sensory and behavioral variables.

Understanding the relationship between a set of variables is a common problem in many fields, such as weather forecast or stock market data. In neuroscience, one of the main challenges is to characterize the dependencies between neuronal activity, sensory stimuli and behavioral outputs. A method of choice for modeling such statistical dependencies is based on copulas, which disentangle dependencies from single variable statistics. To account for changes in dependencies, we model changes in copula parameters by means of Gaussian Processes, conditioned on a task-related variable. The novelty of our approach includes 1) explicit modeling of the dependencies; and 2) combining different copulas to describe experimentally observed variability. We validate the goodness-of-fit as well as information estimates on synthetic data and on recordings from the visual cortex of mice performing a behavioral task. Our parametric model demonstrates significantly better performance in describing high dimensional dependencies compared to other commonly used techniques. We demonstrate that our model can estimate information and predict behaviorally-relevant parameters of the task without providing any explicit cues to the model. Our results indicate that our model is interpretable in the context of neuroscience applications, scalable to large datasets and suitable for accurate statistical modeling and information estimation.

Entropy Estimation Using a Linguistic Zipf-Mandelbrot-Li Model for Natural Sequences

Entropy estimation faces numerous challenges when applied to various real-world problems. Our interest is in divergence and entropy estimation algorithms which are capable of rapid estimation for natural sequence data such as human and synthetic languages. This typically requires a large amount of data; however, we propose a new approach which is based on a new rank-based analytic Zipf–Mandelbrot–Li probabilistic model. Unlike previous approaches, which do not consider the nature of the probability distribution in relation to language; here, we introduce a novel analytic Zipfian model which includes linguistic constraints. This provides more accurate distributions for natural sequences such as natural or synthetic emergent languages. Results are given which indicates the performance of the proposed ZML model. We derive an entropy estimation method which incorporates the linguistic constraint-based Zipf–Mandelbrot–Li into a new non-equiprobable coincidence counting algorithm which is shown to be effective for tasks such as entropy rate estimation with limited data.

A Quick and Easy Way to Estimate Entropy and Mutual Information for Neuroscience

Calculations of entropy of a signal or mutual information between two variables are valuable analytical tools in the field of neuroscience. They can be applied to all types of data, capture non-linear interactions and are model independent. Yet the limited size and number of recordings one can collect in a series of experiments makes their calculation highly prone to sampling bias. Mathematical methods to overcome this so-called "sampling disaster" exist, but require significant expertise, great time and computational costs. As such, there is a need for a simple, unbiased and computationally efficient tool for estimating the level of entropy and mutual information. In this article, we propose that application of entropy-encoding compression algorithms widely used in text and image compression fulfill these requirements. By simply saving the signal in PNG picture format and measuring the size of the file on the hard drive, we can estimate entropy changes through different conditions. Furthermore, with some simple modifications of the PNG file, we can also estimate the evolution of mutual information between a stimulus and the observed responses through different conditions. We first demonstrate the applicability of this method using white-noise-like signals. Then, while this method can be used in all kind of experimental conditions, we provide examples of its application in patch-clamp recordings, detection of place cells and histological data. Although this method does not give an absolute value of entropy or mutual information, it is mathematically correct, and its simplicity and broad use make it a powerful tool for their estimation through experiments.

Predicting synchronous firing of large neural populations from sequential recordings

A major goal in neuroscience is to understand how populations of neurons code for stimuli or actions. While the number of neurons that can be recorded simultaneously is increasing at a fast pace, in most cases these recordings cannot access a complete population: some neurons that carry relevant information remain unrecorded. In particular, it is hard to simultaneously record all the neurons of the same type in a given area. Recent progress have made possible to profile each recorded neuron in a given area thanks to genetic and physiological tools, and to pool together recordings from neurons of the same type across different experimental sessions. However, it is unclear how to infer the activity of a full population of neurons of the same type from these sequential recordings. Neural networks exhibit collective behaviour, e.g. noise correlations and synchronous activity, that are not directly captured by a conditionally-independent model that would just put together the spike trains from sequential recordings. Here we show that we can infer the activity of a full population of retina ganglion cells from sequential recordings, using a novel method based on copula distributions and maximum entropy modeling. From just the spiking response of each ganglion cell to a repeated stimulus, and a few pairwise recordings, we could predict the noise correlations using copulas, and then the full activity of a large population of ganglion cells of the same type using maximum entropy modeling. Remarkably, we could generalize to predict the population responses to different stimuli with similar light conditions and even to different experiments. We could therefore use our method to construct a very large population merging cells' responses from different experiments. We predicted that synchronous activity in ganglion cell populations saturates only for patches larger than 1.5mm in radius, beyond what is today experimentally accessible.

Using intersection information to map stimulus information transfer within neural networks

Analytical tools that estimate the directed information flow between simultaneously recorded neural populations, such as directed information or Granger causality, typically focus on measuring how much information is exchanged between such populations. However, understanding how sensory information is processed through the brain and how it is used to generate behaviors requires estimating specifically the amount of stimulus information that is transmitted. Here we use the concept of intersection information to make progress on how to perform this measure. We develop the concept of transmitted intersection information, which measures how much of the stimulus information present in one population at a certain time is transmitted to a second population at a later time. We show that this measure of stimulus-specific information transfer has several appealing properties, such as being non-negative, and being bounded by the amount of stimulus information present in each of the two populations and by the total amount of information transmitted between the two populations. Applying this measure to simulated neurons or pools of neurons connected by feed-forward synapses, we show that it can discern cases when the information transmitted from one population to another is about specific stimulus features encoded by the sending population from cases in which the information transmitted is not about the stimuli. We also show that this measure has a good statistical sensitivity from trial numbers that can be collected in real data. Our results highlight the promise of using the concept of intersection information to map stimulus-specific information transfer across neural populations.

Estimating the Mutual Information between Two Discrete, Asymmetric Variables with Limited Samples

Determining the strength of nonlinear, statistical dependencies between two variables is a crucial matter in many research fields. The established measure for quantifying such relations is the mutual information. However, estimating mutual information from limited samples is a challenging task. Since the mutual information is the difference of two entropies, the existing Bayesian estimators of entropy may be used to estimate information. This procedure, however, is still biased in the severely under-sampled regime. Here, we propose an alternative estimator that is applicable to those cases in which the marginal distribution of one of the two variables—the one with minimal entropy—is well sampled. The other variable, as well as the joint and conditional distributions, can be severely undersampled. We obtain a consistent estimator that presents very low bias, outperforming previous methods even when the sampled data contain few coincidences. As with other Bayesian estimators, our proposal focuses on the strength of the interaction between the two variables, without seeking to model the specific way in which they are related. A distinctive property of our method is that the main data statistics determining the amount of mutual information is the inhomogeneity of the conditional distribution of the low-entropy variable in those states in which the large-entropy variable registers coincidences.

Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback-Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.