Entropy and information in neural spike trains: progress on the sampling problem

Millisecond-scale motor coding precedes sensorimotor learning in songbirds

A key goal of the nervous system in young animals is to learn motor skills. Songbirds learn to sing as juveniles, providing a unique opportunity to identify the neural correlates of skill acquisition. Prior studies have shown that spike rate variability in vocal motor cortex decreases substantially during song acquisition, suggesting a transition from rate-based neural control to the millisecond-precise motor codes known to underlie adult vocal performance. By distinguishing how the ensemble of spike patterns fired by cortical neurons (the "neural vocabulary") and the relationship between spike patterns and song acoustics (the "neural code") change during song acquisition, we quantified how vocal control changes across learning in juvenile Bengalese finches. We found that despite the expected drop in rate variability (a learning-related change in spike vocabulary), the precision of the neural code in the youngest singers is the same as in adults, with 1-2 ms variations in spike timing transduced into quantifiably different behaviors. In contrast, fluctuations in firing rates on longer timescales fail to affect the motor output in both juvenile and adult animals. The consistent presence of millisecond-scale motor coding during changing levels of spike rate and behavioral variability suggests that learning-related changes in cortical activity reflect the brain's changing its spiking vocabulary to better match the underlying motor code, rather than a change in the precision of the code itself.

Sampling bias corrections for accurate neural measures of redundant, unique, and synergistic information

Shannon Information theory has long been a tool of choice to measure empirically how populations of neurons in the brain encode information about cognitive variables. Recently, Partial Information Decomposition (PID) has emerged as principled way to break down this information into components identifying not only the unique information carried by each neuron, but also whether relationships between neurons generate synergistic or redundant information. While it has been long recognized that Shannon information measures on neural activity suffer from a (mostly upward) limited sampling estimation bias, this issue has largely been ignored in the burgeoning field of PID analysis of neural activity. We used simulations to investigate the limited sampling bias of PID computed from discrete probabilities (suited to describe neural spiking activity). We found that PID suffers from a large bias that is uneven across components, with synergy by far the most biased. Using approximate analytical expansions, we found that the bias of synergy increases quadratically with the number of discrete responses of each neuron, whereas the bias of unique and redundant information increase only linearly or sub-linearly. Based on the understanding of the PID bias properties, we developed simple yet effective procedures that correct for the bias effectively, and that improve greatly the PID estimation with respect to current state-of-the-art procedures. We apply these PID bias correction procedures to datasets of 53117 pairs neurons in auditory cortex, posterior parietal cortex and hippocampus of mice performing cognitive tasks, deriving precise estimates and bounds of how synergy and redundancy vary across these brain regions.

Shared input and recurrency in neural networks for metabolically efficient information transmission

Shared input to a population of neurons induces noise correlations, which can decrease the information carried by a population activity. Inhibitory feedback in recurrent neural networks can reduce the noise correlations and thus increase the information carried by the population activity. However, the activity of inhibitory neurons is costly. This inhibitory feedback decreases the gain of the population. Thus, depolarization of its neurons requires stronger excitatory synaptic input, which is associated with higher ATP consumption. Given that the goal of neural populations is to transmit as much information as possible at minimal metabolic costs, it is unclear whether the increased information transmission reliability provided by inhibitory feedback compensates for the additional costs. We analyze this problem in a network of leaky integrate-and-fire neurons receiving correlated input. By maximizing mutual information with metabolic cost constraints, we show that there is an optimal strength of recurrent connections in the network, which maximizes the value of mutual information-per-cost. For higher values of input correlation, the mutual information-per-cost is higher for recurrent networks with inhibitory feedback compared to feedforward networks without any inhibitory neurons. Our results, therefore, show that the optimal synaptic strength of a recurrent network can be inferred from metabolically efficient coding arguments and that decorrelation of the input by inhibitory feedback compensates for the associated increased metabolic costs.

Entropy Estimators for Markovian Sequences: A Comparative Analysis

Entropy estimation is a fundamental problem in information theory that has applications in various fields, including physics, biology, and computer science. Estimating the entropy of discrete sequences can be challenging due to limited data and the lack of unbiased estimators. Most existing entropy estimators are designed for sequences of independent events and their performances vary depending on the system being studied and the available data size. In this work, we compare different entropy estimators and their performance when applied to Markovian sequences. Specifically, we analyze both binary Markovian sequences and Markovian systems in the undersampled regime. We calculate the bias, standard deviation, and mean squared error for some of the most widely employed estimators. We discuss the limitations of entropy estimation as a function of the transition probabilities of the Markov processes and the sample size. Overall, this paper provides a comprehensive comparison of entropy estimators and their performance in estimating entropy for systems with memory, which can be useful for researchers and practitioners in various fields.

Physical Constraints in Intracellular Signaling: The Cost of Sending a Bit

Many biological processes require timely communication between molecular components. Cells employ diverse physical channels to this end, transmitting information through diffusion, electrical depolarization, and mechanical waves among other strategies. Here we bound the energetic cost of transmitting information through these physical channels, in k_{B}T/bit, as a function of the size of the sender and receiver, their spatial separation, and the communication latency. These calculations provide an estimate for the energy costs associated with information processing arising from the physical constraints of the cellular environment, which we find to be many orders of magnitude larger than unity in natural units. From these calculations, we construct a phase diagram indicating where each strategy is most efficient. Our results suggest that intracellular information transfer may constitute a substantial energetic cost. This provides a new tool for understanding tradeoffs in cellular network function.

Low-probability states, data statistics, and entropy estimation

A fundamental problem in the analysis of complex systems is getting a reliable estimate of the entropy of their probability distributions over the state space. This is difficult because unsampled states can contribute substantially to the entropy, while they do not contribute to the maximum likelihood estimator of entropy, which replaces probabilities by the observed frequencies. Bayesian estimators overcome this obstacle by introducing a model of the low-probability tail of the probability distribution. Which statistical features of the observed data determine the model of the tail, and hence the output of such estimators, remains unclear. Here we show that well-known entropy estimators for probability distributions on discrete state spaces model the structure of the low-probability tail based largely on a few statistics of the data: the sample size, the maximum likelihood estimate, the number of coincidences among the samples, and the dispersion of the coincidences. We derive approximate analytical entropy estimators for undersampled distributions based on these statistics, and we use the results to propose an intuitive understanding of how the Bayesian entropy estimators work.

The Role of Thermodynamic and Informational Entropy in Improving Real Estate Valuation Methods

Price, Cost and Income (PCI) are distinct economic indicators intrinsically linked to the values they denote. These observables take center stage in the multi-criteria decision-making process that enables economic agents to convey subjective utilities of market-exchanged commodities objectively. The valuation of these commodities heavily relies on PCI-based empirical observables and their supported methodologies. This valuation measure's accuracy is critical, as it influences subsequent decisions within the market chain. However, measurement errors often arise due to inherent uncertainties in the value state, impacting economic agents' wealth, particularly when trading significant commodities such as real estate properties. This paper addresses this issue by incorporating entropy measurements into real estate valuation. This mathematical technique adjusts and integrates triadic PCI estimates, improving the final stage of appraisal systems where definitive value decisions are crucial. Employing entropy within the appraisal system can also aid market agents in devising informed production/trading strategies for optimal returns. The results from our practical demonstration indicate promising implications. The entropy's integration with PCI estimates significantly improved the value measurement's precision and reduced economic decision-making errors.

An information theoretic method to resolve millisecond-scale spike timing precision in a comprehensive motor program

Sensory inputs in nervous systems are often encoded at the millisecond scale in a precise spike timing code. There is now growing evidence in behaviors ranging from slow breathing to rapid flight for the prevalence of precise timing encoding in motor systems. Despite this, we largely do not know at what scale timing matters in these circuits due to the difficulty of recording a complete set of spike-resolved motor signals and assessing spike timing precision for encoding continuous motor signals. We also do not know if the precision scale varies depending on the functional role of different motor units. We introduce a method to estimate spike timing precision in motor circuits using continuous MI estimation at increasing levels of added uniform noise. This method can assess spike timing precision at fine scales for encoding rich motor output variation. We demonstrate the advantages of this approach compared to a previously established discrete information theoretic method of assessing spike timing precision. We use this method to analyze the precision in a nearly complete, spike resolved recording of the 10 primary wing muscles control flight in an agile hawk moth, Manduca sexta. Tethered moths visually tracked a robotic flower producing a range of turning (yaw) torques. We know that all 10 muscles in this motor program encode the majority of information about yaw torque in spike timings, but we do not know whether individual muscles encode motor information at different levels of precision. We demonstrate that the scale of temporal precision in all motor units in this insect flight circuit is at the sub-millisecond or millisecond-scale, with variation in precision scale present between muscle types. This method can be applied broadly to estimate spike timing precision in sensory and motor circuits in both invertebrates and vertebrates.

Pareto-Optimal Clustering with the Primal Deterministic Information Bottleneck

At the heart of both lossy compression and clustering is a trade-off between the fidelity and size of the learned representation. Our goal is to map out and study the Pareto frontier that quantifies this trade-off. We focus on the optimization of the Deterministic Information Bottleneck (DIB) objective over the space of hard clusterings. To this end, we introduce the primal DIB problem, which we show results in a much richer frontier than its previously studied Lagrangian relaxation when optimized over discrete search spaces. We present an algorithm for mapping out the Pareto frontier of the primal DIB trade-off that is also applicable to other two-objective clustering problems. We study general properties of the Pareto frontier, and we give both analytic and numerical evidence for logarithmic sparsity of the frontier in general. We provide evidence that our algorithm has polynomial scaling despite the super-exponential search space, and additionally, we propose a modification to the algorithm that can be used where sampling noise is expected to be significant. Finally, we use our algorithm to map the DIB frontier of three different tasks: compressing the English alphabet, extracting informative color classes from natural images, and compressing a group theory-inspired dataset, revealing interesting features of frontier, and demonstrating how the structure of the frontier can be used for model selection with a focus on points previously hidden by the cloak of the convex hull.

Inferring a Property of a Large System from a Small Number of Samples

Inferring the value of a property of a large stochastic system is a difficult task when the number of samples is insufficient to reliably estimate the probability distribution. The Bayesian estimator of the property of interest requires the knowledge of the prior distribution, and in many situations, it is not clear which prior should be used. Several estimators have been developed so far in which the proposed prior us individually tailored for each property of interest; such is the case, for example, for the entropy, the amount of mutual information, or the correlation between pairs of variables. In this paper, we propose a general framework to select priors that is valid for arbitrary properties. We first demonstrate that only certain aspects of the prior distribution actually affect the inference process. We then expand the sought prior as a linear combination of a one-dimensional family of indexed priors, each of which is obtained through a maximum entropy approach with constrained mean values of the property under study. In many cases of interest, only one or very few components of the expansion turn out to contribute to the Bayesian estimator, so it is often valid to only keep a single component. The relevant component is selected by the data, so no handcrafted priors are required. We test the performance of this approximation with a few paradigmatic examples and show that it performs well in comparison to the ad-hoc methods previously proposed in the literature. Our method highlights the connection between Bayesian inference and equilibrium statistical mechanics, since the most relevant component of the expansion can be argued to be that with the right temperature.

A Quick and Easy Way to Estimate Entropy and Mutual Information for Neuroscience

Calculations of entropy of a signal or mutual information between two variables are valuable analytical tools in the field of neuroscience. They can be applied to all types of data, capture non-linear interactions and are model independent. Yet the limited size and number of recordings one can collect in a series of experiments makes their calculation highly prone to sampling bias. Mathematical methods to overcome this so-called "sampling disaster" exist, but require significant expertise, great time and computational costs. As such, there is a need for a simple, unbiased and computationally efficient tool for estimating the level of entropy and mutual information. In this article, we propose that application of entropy-encoding compression algorithms widely used in text and image compression fulfill these requirements. By simply saving the signal in PNG picture format and measuring the size of the file on the hard drive, we can estimate entropy changes through different conditions. Furthermore, with some simple modifications of the PNG file, we can also estimate the evolution of mutual information between a stimulus and the observed responses through different conditions. We first demonstrate the applicability of this method using white-noise-like signals. Then, while this method can be used in all kind of experimental conditions, we provide examples of its application in patch-clamp recordings, detection of place cells and histological data. Although this method does not give an absolute value of entropy or mutual information, it is mathematically correct, and its simplicity and broad use make it a powerful tool for their estimation through experiments.

Variability and Randomness of the Instantaneous Firing Rate

The apparent stochastic nature of neuronal activity significantly affects the reliability of neuronal coding. To quantify the encountered fluctuations, both in neural data and simulations, the notions of variability and randomness of inter-spike intervals have been proposed and studied. In this article we focus on the concept of the instantaneous firing rate, which is also based on the spike timing. We use several classical statistical models of neuronal activity and we study the corresponding probability distributions of the instantaneous firing rate. To characterize the firing rate variability and randomness under different spiking regimes, we use different indices of statistical dispersion. We find that the relationship between the variability of interspike intervals and the instantaneous firing rate is not straightforward in general. Counter-intuitively, an increase in the randomness (based on entropy) of spike times may either decrease or increase the randomness of instantaneous firing rate, in dependence on the neuronal firing model. Finally, we apply our methods to experimental data, establishing that instantaneous rate analysis can indeed provide additional information about the spiking activity.

Embedding optimization reveals long-lasting history dependence in neural spiking activity

Information processing can leave distinct footprints on the statistics of neural spiking. For example, efficient coding minimizes the statistical dependencies on the spiking history, while temporal integration of information may require the maintenance of information over different timescales. To investigate these footprints, we developed a novel approach to quantify history dependence within the spiking of a single neuron, using the mutual information between the entire past and current spiking. This measure captures how much past information is necessary to predict current spiking. In contrast, classical time-lagged measures of temporal dependence like the autocorrelation capture how long-potentially redundant-past information can still be read out. Strikingly, we find for model neurons that our method disentangles the strength and timescale of history dependence, whereas the two are mixed in classical approaches. When applying the method to experimental data, which are necessarily of limited size, a reliable estimation of mutual information is only possible for a coarse temporal binning of past spiking, a so-called past embedding. To still account for the vastly different spiking statistics and potentially long history dependence of living neurons, we developed an embedding-optimization approach that does not only vary the number and size, but also an exponential stretching of past bins. For extra-cellular spike recordings, we found that the strength and timescale of history dependence indeed can vary independently across experimental preparations. While hippocampus indicated strong and long history dependence, in visual cortex it was weak and short, while in vitro the history dependence was strong but short. This work enables an information-theoretic characterization of history dependence in recorded spike trains, which captures a footprint of information processing that is beyond time-lagged measures of temporal dependence. To facilitate the application of the method, we provide practical guidelines and a toolbox.

A model integrating multiple processes of synchronization and coherence for information instantiation within a cortical area

What is the form of dynamic, e.g., sensory, information in the mammalian cortex? Information in the cortex is modeled as a coherence map of a mixed chimera state of synchronous, phasic, and disordered minicolumns. The theoretical model is built on neurophysiological evidence. Complex spatiotemporal information is instantiated through a system of interacting biological processes that generate a synchronized cortical area, a coherent aperture. Minicolumn elements are grouped in macrocolumns in an array analogous to a phased-array radar, modeled as an aperture, a "hole through which radiant energy flows." Coherence maps in a cortical area transform inputs from multiple sources into outputs to multiple targets, while reducing complexity and entropy. Coherent apertures can assume extremely large numbers of different information states as coherence maps, which can be communicated among apertures with corresponding very large bandwidths. The coherent aperture model incorporates considerable reported research, integrating five conceptually and mathematically independent processes: 1) a damped Kuramoto network model, 2) a pumped area field potential, 3) the gating of nearly coincident spikes, 4) the coherence of activity across cortical lamina, and 5) complex information formed through functions in macrocolumns. Biological processes and their interactions are described in equations and a functional circuit such that the mathematical pieces can be assembled the same way the neurophysiological ones are. The model can be conceptually convolved over the specifics of local cortical areas within and across species. A coherent aperture becomes a node in a graph of cortical areas with a corresponding distribution of information.

Behavior of information flow near criticality

Recent experiments have indicated that many biological systems self-organize near their critical point, which hints at a common design principle. While it has been suggested that information transmission is optimized near the critical point, it remains unclear how information transmission depends on the dynamics of the input signal, the distance over which the information needs to be transmitted, and the distance to the critical point. Here we employ stochastic simulations of a driven two-dimensional Ising system and study the instantaneous mutual information and the information transmission rate between a driven input spin and an output spin. The instantaneous mutual information varies nonmonotonically with the temperature but increases monotonically with the correlation time of the input signal. In contrast, there exists not only an optimal temperature but also an optimal finite input correlation time that maximizes the information transmission rate. This global optimum arises from a fundamental trade-off between the need to maximize the frequency of independent input messages, the necessity to respond fast to changes in the input, and the need to respond reliably to these changes. The optimal temperature lies above the critical point but moves toward it as the distance between the input and output spin is increased.

Entropy, mutual information, and systematic measures of structured spiking neural networks

The aim of this paper is to investigate various information-theoretic measures, including entropy, mutual information, and some systematic measures that are based on mutual information, for a class of structured spiking neuronal networks. In order to analyze and compute these information-theoretic measures for large networks, we coarse-grained the data by ignoring the order of spikes that fall into the same small time bin. The resultant coarse-grained entropy mainly captures the information contained in the rhythm produced by a local population of the network. We first show that these information theoretical measures are well-defined and computable by proving stochastic stability and the law of large numbers. Then we use three neuronal network examples, from simple to complex, to investigate these information-theoretic measures. Several analytical and computational results about properties of these information-theoretic measures are given.

On the Use of Correlation and MI as a Measure of Metabolite-Metabolite Association for Network Differential Connectivity Analysis

Metabolite differential connectivity analysis has been successful in investigating potential molecular mechanisms underlying different conditions in biological systems. Correlation and Mutual Information (MI) are two of the most common measures to quantify association and for building metabolite-metabolite association networks and to calculate differential connectivity. In this study, we investigated the performance of correlation and MI to identify significantly differentially connected metabolites. These association measures were compared on (i) 23 publicly available metabolomic data sets and 7 data sets from other fields, (ii) simulated data with known correlation structures, and (iii) data generated using a dynamic metabolic model to simulate real-life observed metabolite concentration profiles. In all cases, we found more differentially connected metabolites when using correlation indices as a measure for association than MI. We also observed that different MI estimation algorithms resulted in difference in performance when applied to data generated using a dynamic model. We concluded that there is no significant benefit in using MI as a replacement for standard Pearson's or Spearman's correlation when the application is to quantify and detect differentially connected metabolites.

Flexible Multielectrode Arrays With 2-D and 3-D Contacts for In Vivo Electromyography Recording

We present a system for recording in vivo electromyographic (EMG) signals from songbirds using hybrid polyimide-polydimethylsiloxane (PDMS) flexible multielectrode arrays (MEAs). 2-D electrodes with a diameter of 200, 125, and 50 μm and a center-to-center pitch of 300, 200, and 100 μm, respectively, were fabricated. 3-D MEAs were fabricated using a photoresist reflow process to obtain hemispherical domes utilized to form the 3-D electrodes. Biocompatibility and flexibility of the arrays were ensured by using polyimide and PDMS as the materials of choice for the arrays. EMG activity was recorded from the expiratory muscle group of anesthetized songbirds using the fabricated 2-D and 3-D arrays. Air pressure data were also recorded simultaneously from the air sac of the songbird. Together, EMG recordings and air pressure measurements can be used to characterize how the nervous system controls breathing and other motor behaviors. Such technologies can in turn provide unique insights into motor control in a range of species, including humans. An improvement of over 7× in the signal-to-noise ratio (SNR) is observed with the utilization of 3-D MEAs in comparison to 2-D MEAs.

Topological Information Data Analysis

This paper presents methods that quantify the structure of statistical interactions within a given data set, and were applied in a previous article. It establishes new results on the k-multivariate mutual-information (Ik) inspired by the topological formulation of Information introduced in a serie of studies. In particular, we show that the vanishing of all Ik for 2≤k≤n of n random variables is equivalent to their statistical independence. Pursuing the work of Hu Kuo Ting and Te Sun Han, we show that information functions provide co-ordinates for binary variables, and that they are analytically independent from the probability simplex for any set of finite variables. The maximal positive Ik identifies the variables that co-vary the most in the population, whereas the minimal negative Ik identifies synergistic clusters and the variables that differentiate–segregate the most in the population. Finite data size effects and estimation biases severely constrain the effective computation of the information topology on data, and we provide simple statistical tests for the undersampling bias and the k-dependences. We give an example of application of these methods to genetic expression and unsupervised cell-type classification. The methods unravel biologically relevant subtypes, with a sample size of 41 genes and with few errors. It establishes generic basic methods to quantify the epigenetic information storage and a unified epigenetic unsupervised learning formalism. We propose that higher-order statistical interactions and non-identically distributed variables are constitutive characteristics of biological systems that should be estimated in order to unravel their significant statistical structure and diversity. The topological information data analysis presented here allows for precisely estimating this higher-order structure characteristic of biological systems.

Information processing in the LGN: a comparison of neural codes and cell types

To understand how anatomy and physiology allow an organism to perform its function, it is important to know how information that is transmitted by spikes in the brain is received and encoded. A natural question is whether the spike rate alone encodes the information about a stimulus (rate code), or additional information is contained in the temporal pattern of the spikes (temporal code). Here we address this question using data from the cat Lateral Geniculate Nucleus (LGN), which is the visual portion of the thalamus, through which visual information from the retina is communicated to the visual cortex. We analyzed the responses of LGN neurons to spatially homogeneous spots of various sizes with temporally random luminance modulation. We compared the Firing Rate with the Shannon Information Transmission Rate , which quantifies the information contained in the temporal relationships between spikes. We found that the behavior of these two rates can differ quantitatively. This suggests that the energy used for spiking does not translate directly into the information to be transmitted. We also compared Firing Rates with Information Rates for X-ON and X-OFF cells. We found that, for X-ON cells the Firing Rate and Information Rate often behave in a completely different way, while for X-OFF cells these rates are much more highly correlated. Our results suggest that for X-ON cells a more efficient "temporal code" is employed, while for X-OFF cells a straightforward "rate code" is used, which is more reliable and is correlated with energy consumption.

Empirical Estimation of Information Measures: A Literature Guide

We give a brief survey of the literature on the empirical estimation of entropy, differential entropy, relative entropy, mutual information and related information measures. While those quantities are of central importance in information theory, universal algorithms for their estimation are increasingly important in data science, machine learning, biology, neuroscience, economics, language, and other experimental sciences.

Estimating the Mutual Information between Two Discrete, Asymmetric Variables with Limited Samples

Determining the strength of nonlinear, statistical dependencies between two variables is a crucial matter in many research fields. The established measure for quantifying such relations is the mutual information. However, estimating mutual information from limited samples is a challenging task. Since the mutual information is the difference of two entropies, the existing Bayesian estimators of entropy may be used to estimate information. This procedure, however, is still biased in the severely under-sampled regime. Here, we propose an alternative estimator that is applicable to those cases in which the marginal distribution of one of the two variables—the one with minimal entropy—is well sampled. The other variable, as well as the joint and conditional distributions, can be severely undersampled. We obtain a consistent estimator that presents very low bias, outperforming previous methods even when the sampled data contain few coincidences. As with other Bayesian estimators, our proposal focuses on the strength of the interaction between the two variables, without seeking to model the specific way in which they are related. A distinctive property of our method is that the main data statistics determining the amount of mutual information is the inhomogeneity of the conditional distribution of the low-entropy variable in those states in which the large-entropy variable registers coincidences.

Quantifying Information Conveyed by Large Neuronal Populations

Quantifying mutual information between inputs and outputs of a large neural circuit is an important open problem in both machine learning and neuroscience. However, evaluation of the mutual information is known to be generally intractable for large systems due to the exponential growth in the number of terms that need to be evaluated. Here we show how information contained in the responses of large neural populations can be effectively computed provided the input-output functions of individual neurons can be measured and approximated by a logistic function applied to a potentially nonlinear function of the stimulus. Neural responses in this model can remain sensitive to multiple stimulus components. We show that the mutual information in this model can be effectively approximated as a sum of lower-dimensional conditional mutual information terms. The approximations become exact in the limit of large neural populations and for certain conditions on the distribution of receptive fields across the neural population. We empirically find that these approximations continue to work well even when the conditions on the receptive field distributions are not fulfilled. The computing cost for the proposed methods grows linearly in the dimension of the input and compares favorably with other approximations.

Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback-Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.

Calculating the Mutual Information between Two Spike Trains

It is difficult to estimate the mutual information between spike trains because established methods require more data than are usually available. Kozachenko-Leonenko estimators promise to solve this problem but include a smoothing parameter that must be set. We propose here that the smoothing parameter can be selected by maximizing the estimated unbiased mutual information. This is tested on fictive data and shown to work very well.

Fabrication and Characterization of 3D Multi-Electrode Array on Flexible Substrate for In Vivo EMG Recording from Expiratory Muscle of Songbird

This work presents fabrication and characterization of flexible three-dimensional (3D) multi-electrode arrays (MEAs) capable of high signal-to-noise (SNR) electromyogram (EMG) recordings from the expiratory muscle of a songbird. The fabrication utilizes a photoresist reflow process to obtain 3D structures to serve as the electrodes. A polyimide base with a PDMS top insulation was utilized to ensure flexibility and biocompatibility of the fabricated 3D MEA devices. SNR measurements from the fabricated 3D electrode show up to a 7x improvement as compared to the 2D MEAs.

Assessing the Relevance of Specific Response Features in the Neural Code

The study of the neural code aims at deciphering how the nervous system maps external stimuli into neural activity—the encoding phase—and subsequently transforms such activity into adequate responses to the original stimuli—the decoding phase. Several information-theoretical methods have been proposed to assess the relevance of individual response features, as for example, the spike count of a given neuron, or the amount of correlation in the activity of two cells. These methods work under the premise that the relevance of a feature is reflected in the information loss that is induced by eliminating the feature from the response. The alternative methods differ in the procedure by which the tested feature is removed, and the algorithm with which the lost information is calculated. Here we compare these methods, and show that more often than not, each method assigns a different relevance to the tested feature. We demonstrate that the differences are both quantitative and qualitative, and connect them with the method employed to remove the tested feature, as well as the procedure to calculate the lost information. By studying a collection of carefully designed examples, and working on analytic derivations, we identify the conditions under which the relevance of features diagnosed by different methods can be ranked, or sometimes even equated. The condition for equality involves both the amount and the type of information contributed by the tested feature. We conclude that the quest for relevant response features is more delicate than previously thought, and may yield to multiple answers depending on methodological subtleties.

Information estimation using nonparametric copulas

Estimation of mutual information between random variables has become crucial in a range of fields, from physics to neuroscience to finance. Estimating information accurately over a wide range of conditions relies on the development of flexible methods to describe statistical dependencies among variables, without imposing potentially invalid assumptions on the data. Such methods are needed in cases that lack prior knowledge of their statistical properties and that have limited sample numbers. Here we propose a powerful and generally applicable information estimator based on non-parametric copulas. This estimator, called the non-parametric copula-based estimator (NPC), is tailored to take into account detailed stochastic relationships in the data independently of the data's marginal distributions. The NPC estimator can be used both for continuous and discrete numerical variables and thus provides a single framework for the mutual information estimation of both continuous and discrete data. By extensive validation on artificial samples drawn from various statistical distributions, we found that the NPC estimator compares well against commonly used alternatives. Unlike methods not based on copulas, it allows an estimation of information that is robust to changes of the details of the marginal distributions. Unlike parametric copula methods, it remains accurate regardless of the precise form of the interactions between the variables. In addition, the NPC estimator had accurate information estimates even at low sample numbers, in comparison to alternative estimators. The NPC estimator therefore provides a good balance between general applicability to arbitrarily shaped statistical dependencies in the data and shows accurate and robust performance when working with small sample sizes. We anticipate that the non-parametric copula information estimator will be a powerful tool in estimating mutual information between a broad range of data.

Inferring interaction partners from protein sequences using mutual information

Functional protein-protein interactions are crucial in most cellular processes. They enable multi-protein complexes to assemble and to remain stable, and they allow signal transduction in various pathways. Functional interactions between proteins result in coevolution between the interacting partners, and thus in correlations between their sequences. Pairwise maximum-entropy based models have enabled successful inference of pairs of amino-acid residues that are in contact in the three-dimensional structure of multi-protein complexes, starting from the correlations in the sequence data of known interaction partners. Recently, algorithms inspired by these methods have been developed to identify which proteins are functional interaction partners among the paralogous proteins of two families, starting from sequence data alone. Here, we demonstrate that a slightly higher performance for partner identification can be reached by an approximate maximization of the mutual information between the sequence alignments of the two protein families. Our mutual information-based method also provides signatures of the existence of interactions between protein families. These results stand in contrast with structure prediction of proteins and of multi-protein complexes from sequence data, where pairwise maximum-entropy based global statistical models substantially improve performance compared to mutual information. Our findings entail that the statistical dependences allowing interaction partner prediction from sequence data are not restricted to the residue pairs that are in direct contact at the interface between the partner proteins.

Neural coding: A single neuron's perspective

What any sensory neuron knows about the world is one of the cardinal questions in Neuroscience. Information from the sensory periphery travels across synaptically coupled neurons as each neuron encodes information by varying the rate and timing of its action potentials (spikes). Spatiotemporally correlated changes in this spiking regimen across neuronal populations are the neural basis of sensory representations. In the somatosensory cortex, however, spiking of individual (or pairs of) cortical neurons is only minimally informative about the world. Recent studies showed that one solution neurons implement to counteract this information loss is adapting their rate of information transfer to the ongoing synaptic activity by changing the membrane potential at which spike is generated. Here we first introduce the principles of information flow from the sensory periphery to the primary sensory cortex in a model sensory (whisker) system, and subsequently discuss how the adaptive spike threshold gates the intracellular information transfer from the somatic post-synaptic potential to action potentials, controlling the information content of communication across somatosensory cortical neurons.

Learning from the past: A reverberation of past errors in the cerebellar climbing fiber signal

The cerebellum allows us to rapidly adjust motor behavior to the needs of the situation. It is commonly assumed that cerebellum-based motor learning is guided by the difference between the desired and the actual behavior, i.e., by error information. Not only immediate but also future behavior will benefit from an error because it induces lasting changes of parallel fiber synapses on Purkinje cells (PCs), whose output mediates the behavioral adjustments. Olivary climbing fibers, likewise connecting with PCs, are thought to transport information on instant errors needed for the synaptic modification yet not to contribute to error memory. Here, we report work on monkeys tested in a saccadic learning paradigm that challenges this concept. We demonstrate not only a clear complex spikes (CS) signature of the error at the time of its occurrence but also a reverberation of this signature much later, before a new manifestation of the behavior, suitable to improve it.

Information-theoretic analysis of realistic odor plumes: What cues are useful for determining location?

Many species rely on olfaction to navigate towards food sources or mates. Olfactory navigation is a challenging task since odor environments are typically turbulent. While time-averaged odor concentration varies smoothly with the distance to the source, instaneous concentrations are intermittent and obtaining stable averages takes longer than the typical intervals between animals’ navigation decisions. How to effectively sample from the odor distribution to determine sampling location is the focus in this article. To investigate which sampling strategies are most informative about the location of an odor source, we recorded three naturalistic stimuli with planar lased-induced fluorescence and used an information-theoretic approach to quantify the information that different sampling strategies provide about sampling location. Specifically, we compared multiple sampling strategies based on a fixed number of coding bits for encoding the olfactory stimulus. When the coding bits were all allocated to representing odor concentration at a single sensor, information rapidly saturated. Using the same number of coding bits in two sensors provides more information, as does coding multiple samples at different times. When accumulating multiple samples at a fixed location, the temporal sequence does not yield a large amount of information and can be averaged with minimal loss. Furthermore, we show that histogram-equalization is not the most efficient way to use coding bits when using the olfactory sample to determine location.

Navigating towards a food source or mating partner based on an animals’ sense of smell is a difficult task due to the complex spatiotemporal distribution of odor molecules. The most basic aspect of this task is the acquisition of samples from the environment. It is clear that odor concentration does not vary smoothly across space in many natural foraging environments. Using data from three different naturalistic environments, we compare different sampling strategies and assess their efficacy in determining the sources’ location. Our findings show that coarsely encoding the concentration of samples at separate sensors and/or multiple times provides more information than encoding fewer samples with higher resolution. Furthermore, coding resources should be focused on discriminating rare high-concentration odor samples, which are very informative about the sampling location. Such a nonlinear transformation can be implemented biologically by the receptor binding kinetics that bind odorants as a first stage of the sampling process. A further implication is that animals as well as computational models of algorithms can operate efficiently with a coarse representation of the odor concentration.

Rhythmic activities of the brain: Quantifying the high complexity of beta and gamma oscillations during visuomotor tasks

Electroencephalography (EEG) signals depict the electrical activity that takes place at the surface of the brain and provide an important tool for understanding a variety of cognitive processes. The EEG is the product of synchronized activity of the brain, and variations in EEG oscillations patterns reflect the underlying changes in neuronal synchrony. Our aim is to characterize the complexity of the EEG rhythmic oscillations bands when the subjects perform a visuomotor or imagined cognitive tasks (imagined movement), providing a causal mapping of the dynamical rhythmic activities of the brain as a measure of attentional investment. We estimate the intrinsic correlational structure of the signals within the causality entropy-complexity plane H×C, where the enhanced complexity in the gamma 1, gamma 2, and beta 1 bands allows us to distinguish motor-visual memory tasks from control conditions. We identify the dynamics of the gamma 1, gamma 2, and beta 1 rhythmic oscillations within the zone of a chaotic dissipative behavior, whereas in contrast the beta 2 band shows a much higher level of entropy and a significant low level of complexity that correspond to a non-invertible cubic map. Our findings enhance the importance of the gamma band during attention in perceptual feature binding during the visuomotor/imagery tasks.

A Tutorial for Information Theory in Neuroscience

Understanding how neural systems integrate, encode, and compute information is central to understanding brain function. Frequently, data from neuroscience experiments are multivariate, the interactions between the variables are nonlinear, and the landscape of hypothesized or possible interactions between variables is extremely broad. Information theory is well suited to address these types of data, as it possesses multivariate analysis tools, it can be applied to many different types of data, it can capture nonlinear interactions, and it does not require assumptions about the structure of the underlying data (i.e., it is model independent). In this article, we walk through the mathematics of information theory along with common logistical problems associated with data type, data binning, data quantity requirements, bias, and significance testing. Next, we analyze models inspired by canonical neuroscience experiments to improve understanding and demonstrate the strengths of information theory analyses. To facilitate the use of information theory analyses, and an understanding of how these analyses are implemented, we also provide a free MATLAB software package that can be applied to a wide range of data from neuroscience experiments, as well as from other fields of study.

Mutual Information and Information Gating in Synfire Chains

Coherent neuronal activity is believed to underlie the transfer and processing of information in the brain. Coherent activity in the form of synchronous firing and oscillations has been measured in many brain regions and has been correlated with enhanced feature processing and other sensory and cognitive functions. In the theoretical context, synfire chains and the transfer of transient activity packets in feedforward networks have been appealed to in order to describe coherent spiking and information transfer. Recently, it has been demonstrated that the classical synfire chain architecture, with the addition of suitably timed gating currents, can support the graded transfer of mean firing rates in feedforward networks (called synfire-gated synfire chains—SGSCs). Here we study information propagation in SGSCs by examining mutual information as a function of layer number in a feedforward network. We explore the effects of gating and noise on information transfer in synfire chains and demonstrate that asymptotically, two main regions exist in parameter space where information may be propagated and its propagation is controlled by pulse-gating: a large region where binary codes may be propagated, and a smaller region near a cusp in parameter space that supports graded propagation across many layers.

Dual Coding Theory Explains Biphasic Collective Computation in Neural Decision-Making

A central question in cognitive neuroscience is how unitary, coherent decisions at the whole organism level can arise from the distributed behavior of a large population of neurons with only partially overlapping information. We address this issue by studying neural spiking behavior recorded from a multielectrode array with 169 channels during a visual motion direction discrimination task. It is well known that in this task there are two distinct phases in neural spiking behavior. Here we show Phase I is a distributed or incompressible phase in which uncertainty about the decision is substantially reduced by pooling information from many cells. Phase II is a redundant or compressible phase in which numerous single cells contain all the information present at the population level in Phase I, such that the firing behavior of a single cell is enough to predict the subject's decision. Using an empirically grounded dynamical modeling framework, we show that in Phase I large cell populations with low redundancy produce a slow timescale of information aggregation through critical slowing down near a symmetry-breaking transition. Our model indicates that increasing collective amplification in Phase II leads naturally to a faster timescale of information pooling and consensus formation. Based on our results and others in the literature, we propose that a general feature of collective computation is a "coding duality" in which there are accumulation and consensus formation processes distinguished by different timescales.

Breakdown of local information processing may underlie isoflurane anesthesia effects

The disruption of coupling between brain areas has been suggested as the mechanism underlying loss of consciousness in anesthesia. This hypothesis has been tested previously by measuring the information transfer between brain areas, and by taking reduced information transfer as a proxy for decoupling. Yet, information transfer is a function of the amount of information available in the information source—such that transfer decreases even for unchanged coupling when less source information is available. Therefore, we reconsidered past interpretations of reduced information transfer as a sign of decoupling, and asked whether impaired local information processing leads to a loss of information transfer. An important prediction of this alternative hypothesis is that changes in locally available information (signal entropy) should be at least as pronounced as changes in information transfer. We tested this prediction by recording local field potentials in two ferrets after administration of isoflurane in concentrations of 0.0%, 0.5%, and 1.0%. We found strong decreases in the source entropy under isoflurane in area V1 and the prefrontal cortex (PFC)—as predicted by our alternative hypothesis. The decrease in source entropy was stronger in PFC compared to V1. Information transfer between V1 and PFC was reduced bidirectionally, but with a stronger decrease from PFC to V1. This links the stronger decrease in information transfer to the stronger decrease in source entropy—suggesting reduced source entropy reduces information transfer. This conclusion fits the observation that the synaptic targets of isoflurane are located in local cortical circuits rather than on the synapses formed by interareal axonal projections. Thus, changes in information transfer under isoflurane seem to be a consequence of changes in local processing more than of decoupling between brain areas. We suggest that source entropy changes must be considered whenever interpreting changes in information transfer as decoupling.

Currently we do not understand how anesthesia leads to loss of consciousness (LOC). One popular idea is that we loose consciousness when brain areas lose their ability to communicate with each other–as anesthetics might interrupt transmission on nerve fibers coupling them. This idea has been tested by measuring the amount of information transferred between brain areas, and taking this transfer to reflect the coupling itself. Yet, information that isn’t available in the source area can’t be transferred to a target. Hence, the decreases in information transfer could be related to less information being available in the source, rather than to a decoupling. We tested this possibility measuring the information available in source brain areas and found that it decreased under isoflurane anesthesia. In addition, a stronger decrease in source information lead to a stronger decrease of the information transfered. Thus, the input to the connection between brain areas determined the communicated information, not the strength of the coupling (which would result in a stronger decrease in the target). We suggest that interrupted information processing within brain areas has an important contribution to LOC, and should be focused on more in attempts to understand loss of consciousness under anesthesia.

Distributing tasks via multiple input pathways increases cellular survival in stress

Improving in one aspect of a task can undermine performance in another, but how such opposing demands play out in single cells and impact on fitness is mostly unknown. Here we study budding yeast in dynamic environments of hyperosmotic stress and show how the corresponding signalling network increases cellular survival both by assigning the requirements of high response speed and high response accuracy to two separate input pathways and by having these pathways interact to converge on Hog1, a p38 MAP kinase. Cells with only the less accurate, reflex-like pathway are fitter in sudden stress, whereas cells with only the slow, more accurate pathway are fitter in increasing but fluctuating stress. Our results demonstrate that cellular signalling is vulnerable to trade-offs in performance, but that these trade-offs can be mitigated by assigning the opposing tasks to different signalling subnetworks. Such division of labour could function broadly within cellular signal transduction.

DOI:http://dx.doi.org/10.7554/eLife.21415.001

The faster we do tasks the harder it is to do them well. For example, when we wish to judge if, say, a cup, is too hot, we first quickly withdraw our hand after touching it: we know that the cup is hot but not how much. Next we hold a finger against the cup to accurately judge its temperature. Such speed-accuracy trade-offs are studied widely in fields ranging from neuroscience to engineering, but their consequences for single cells are unknown. This is despite the fact that when cells are exposed to stress they must respond both quickly (to survive) and accurately (to reduce how many resources they consume).

One way of stressing yeast cells is to place them in a syrupy substance called sorbitol. This causes the cells to lose water, shrink in size, and launch a stress response to regain volume. If the cells respond inappropriately to the situation, they may die.

The signalling network that produces the stress response is unusual in that it has a Y-shaped structure, where the two ‘arms’ of the Y are the input pathways. Although it was known that one input pathway responds to stress faster than the other, the advantages of having two inputs in the signalling network were not understood. Granados, Crane et al. thought that the differences in speed and the Y-shaped structure could allow the cell to respond to stress with both speed and accuracy.

To investigate this theory, Granados, Crane et al. used a microscope to study individual yeast cells that had been exposed to sorbitol. Combining these results with a mathematical model of the cell signalling network revealed that a mutant yeast cell that only has one of the input pathways specializes in speed but is inaccurate, similar to a reflex-like response. In contrast, a mutant with only the other pathway specializes in accuracy, being slower but matching the level of the cell’s response to the level of stress placed on it. This trade-off is reflected in rates of cell survival: the first mutant survives best in sudden shocks of stress; the second mutant survives best in gradually increasing stress. Normal yeast cells that have both input pathways survive more often than either mutant.

Overall, the results presented by Granados, Crane et al. reveal principles behind cellular decision-making that should hold true in more complex organisms and could be exploited by synthetic biologists to programme cells with new behaviours.

DOI:http://dx.doi.org/10.7554/eLife.21415.002

A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula

We begin by reviewing the statistical framework of information theory as applicable to neuroimaging data analysis. A major factor hindering wider adoption of this framework in neuroimaging is the difficulty of estimating information theoretic quantities in practice. We present a novel estimation technique that combines the statistical theory of copulas with the closed form solution for the entropy of Gaussian variables. This results in a general, computationally efficient, flexible, and robust multivariate statistical framework that provides effect sizes on a common meaningful scale, allows for unified treatment of discrete, continuous, unidimensional and multidimensional variables, and enables direct comparisons of representations from behavioral and brain responses across any recording modality. We validate the use of this estimate as a statistical test within a neuroimaging context, considering both discrete stimulus classes and continuous stimulus features. We also present examples of analyses facilitated by these developments, including application of multivariate analyses to MEG planar magnetic field gradients, and pairwise temporal interactions in evoked EEG responses. We show the benefit of considering the instantaneous temporal derivative together with the raw values of M/EEG signals as a multivariate response, how we can separately quantify modulations of amplitude and direction for vector quantities, and how we can measure the emergence of novel information over time in evoked responses. Open-source Matlab and Python code implementing the new methods accompanies this article. Hum Brain Mapp 38:1541-1573, 2017. © 2016 Wiley Periodicals, Inc.

Motor control by precisely timed spike patterns

A fundamental problem in neuroscience is understanding how sequences of action potentials ("spikes") encode information about sensory signals and motor outputs. Although traditional theories assume that this information is conveyed by the total number of spikes fired within a specified time interval (spike rate), recent studies have shown that additional information is carried by the millisecond-scale timing patterns of action potentials (spike timing). However, it is unknown whether or how subtle differences in spike timing drive differences in perception or behavior, leaving it unclear whether the information in spike timing actually plays a role in brain function. By examining the activity of individual motor units (the muscle fibers innervated by a single motor neuron) and manipulating patterns of activation of these neurons, we provide both correlative and causal evidence that the nervous system uses millisecond-scale variations in the timing of spikes within multispike patterns to control a vertebrate behavior-namely, respiration in the Bengalese finch, a songbird. These findings suggest that a fundamental assumption of current theories of motor coding requires revision.

Analysis of Power Laws, Shape Collapses, and Neural Complexity: New Techniques and MATLAB Support via the NCC Toolbox

Neural systems include interactions that occur across many scales. Two divergent methods for characterizing such interactions have drawn on the physical analysis of critical phenomena and the mathematical study of information. Inferring criticality in neural systems has traditionally rested on fitting power laws to the property distributions of "neural avalanches" (contiguous bursts of activity), but the fractal nature of avalanche shapes has recently emerged as another signature of criticality. On the other hand, neural complexity, an information theoretic measure, has been used to capture the interplay between the functional localization of brain regions and their integration for higher cognitive functions. Unfortunately, treatments of all three methods-power-law fitting, avalanche shape collapse, and neural complexity-have suffered from shortcomings. Empirical data often contain biases that introduce deviations from true power law in the tail and head of the distribution, but deviations in the tail have often been unconsidered; avalanche shape collapse has required manual parameter tuning; and the estimation of neural complexity has relied on small data sets or statistical assumptions for the sake of computational efficiency. In this paper we present technical advancements in the analysis of criticality and complexity in neural systems. We use maximum-likelihood estimation to automatically fit power laws with left and right cutoffs, present the first automated shape collapse algorithm, and describe new techniques to account for large numbers of neural variables and small data sets in the calculation of neural complexity. In order to facilitate future research in criticality and complexity, we have made the software utilized in this analysis freely available online in the MATLAB NCC (Neural Complexity and Criticality) Toolbox.

Rényi entropy, abundance distribution, and the equivalence of ensembles

Distributions of abundances or frequencies play an important role in many fields of science, from biology to sociology, as does the Rényi entropy, which measures the diversity of a statistical ensemble. We derive a mathematical relation between the abundance distribution and the Rényi entropy, by analogy with the equivalence of ensembles in thermodynamics. The abundance distribution is mapped onto the density of states, and the Rényi entropy to the free energy. The two quantities are related in the thermodynamic limit by a Legendre transform, by virtue of the equivalence between the micro-canonical and canonical ensembles. In this limit, we show how the Rényi entropy can be constructed geometrically from rank-frequency plots. This mapping predicts that non-concave regions of the rank-frequency curve should result in kinks in the Rényi entropy as a function of its order. We illustrate our results on simple examples, and emphasize the limitations of the equivalence of ensembles when a thermodynamic limit is not well defined. Our results help choose reliable diversity measures based on the experimental accuracy of the abundance distributions in particular frequency ranges.

Characterizing Protease Specificity: How Many Substrates Do We Need?

Calculation of cleavage entropies allows to quantify, map and compare protease substrate specificity by an information entropy based approach. The metric intrinsically depends on the number of experimentally determined substrates (data points). Thus a statistical analysis of its numerical stability is crucial to estimate the systematic error made by estimating specificity based on a limited number of substrates. In this contribution, we show the mathematical basis for estimating the uncertainty in cleavage entropies. Sets of cleavage entropies are calculated using experimental cleavage data and modeled extreme cases. By analyzing the underlying mathematics and applying statistical tools, a linear dependence of the metric in respect to 1/n was found. This allows us to extrapolate the values to an infinite number of samples and to estimate the errors. Analyzing the errors, a minimum number of 30 substrates was found to be necessary to characterize substrate specificity, in terms of amino acid variability, for a protease (S4-S4’) with an uncertainty of 5 percent. Therefore, we encourage experimental researchers in the protease field to record specificity profiles of novel proteases aiming to identify at least 30 peptide substrates of maximum sequence diversity. We expect a full characterization of protease specificity helpful to rationalize biological functions of proteases and to assist rational drug design.

Information Flow through a Model of the C. elegans Klinotaxis Circuit

Understanding how information about external stimuli is transformed into behavior is one of the central goals of neuroscience. Here we characterize the information flow through a complete sensorimotor circuit: from stimulus, to sensory neurons, to interneurons, to motor neurons, to muscles, to motion. Specifically, we apply a recently developed framework for quantifying information flow to a previously published ensemble of models of salt klinotaxis in the nematode worm Caenorhabditis elegans. Despite large variations in the neural parameters of individual circuits, we found that the overall information flow architecture circuit is remarkably consistent across the ensemble. This suggests structural connectivity is not necessarily predictive of effective connectivity. It also suggests information flow analysis captures general principles of operation for the klinotaxis circuit. In addition, information flow analysis reveals several key principles underlying how the models operate: (1) Interneuron class AIY is responsible for integrating information about positive and negative changes in concentration, and exhibits a strong left/right information asymmetry. (2) Gap junctions play a crucial role in the transfer of information responsible for the information symmetry observed in interneuron class AIZ. (3) Neck motor neuron class SMB implements an information gating mechanism that underlies the circuit’s state-dependent response. (4) The neck carries more information about small changes in concentration than about large ones, and more information about positive changes in concentration than about negative ones. Thus, not all directions of movement are equally informative for the worm. Each of these findings corresponds to hypotheses that could potentially be tested in the worm. Knowing the results of these experiments would greatly refine our understanding of the neural circuit underlying klinotaxis.

Time resolution dependence of information measures for spiking neurons: scaling and universality

The mutual information between stimulus and spike-train response is commonly used to monitor neural coding efficiency, but neuronal computation broadly conceived requires more refined and targeted information measures of input-output joint processes. A first step toward that larger goal is to develop information measures for individual output processes, including information generation (entropy rate), stored information (statistical complexity), predictable information (excess entropy), and active information accumulation (bound information rate). We calculate these for spike trains generated by a variety of noise-driven integrate-and-fire neurons as a function of time resolution and for alternating renewal processes. We show that their time-resolution dependence reveals coarse-grained structural properties of interspike interval statistics; e.g., τ-entropy rates that diverge less quickly than the firing rate indicated by interspike interval correlations. We also find evidence that the excess entropy and regularized statistical complexity of different types of integrate-and-fire neurons are universal in the continuous-time limit in the sense that they do not depend on mechanism details. This suggests a surprising simplicity in the spike trains generated by these model neurons. Interestingly, neurons with gamma-distributed ISIs and neurons whose spike trains are alternating renewal processes do not fall into the same universality class. These results lead to two conclusions. First, the dependence of information measures on time resolution reveals mechanistic details about spike train generation. Second, information measures can be used as model selection tools for analyzing spike train processes.