Mutual information against correlations in binary communication channels

Non-Equilibrium Enhancement of Classical Information Transmission

Information transmission plays a crucial role across various fields, including physics, engineering, biology, and society. The efficiency of this transmission is quantified by mutual information and its associated information capacity. While studies in closed systems have yielded significant progress, understanding the impact of non-equilibrium effects on open systems remains a challenge. These effects, characterized by the exchange of energy, information, and materials with the external environment, can influence both mutual information and information capacity. Here, we delve into this challenge by exploring non-equilibrium effects using the memoryless channel model, a cornerstone of information channel coding theories and methodology development. Our findings reveal that mutual information exhibits a convex relationship with non-equilibriumness, quantified by the non-equilibrium strength in transmission probabilities. Notably, channel information capacity is enhanced by non-equilibrium effects. Furthermore, we demonstrate that non-equilibrium thermodynamic cost, characterized by the entropy production rate, can actually improve both mutual information and information channel capacity, leading to a boost in overall information transmission efficiency. Our numerical results support our conclusions.

Intracortical functional connectivity dynamics induced by reflex seizures

Functional connectivity analysis is gaining more interest due to its promising clinical applications. To study network mechanisms underlying seizure termination and postictal depression, we explore dynamics of interhemispheric functional connectivity near the offset of focal and bilateral seizures in the experimental model of reflex audiogenic epilepsy. In the model, seizures and spreading depression are induced by sound stimulation of genetically predisposed rodents. We characterize temporal evolution of seizure-associated coupling dynamics in the frontoparietal cortex during late ictal, immediate postictal and interictal resting states, using two measures applied to local field potentials recorded in awake epileptic rats. Signals were analyzed with mean phase coherence index in delta (1-4 Hz), theta (4-10 Hz) beta (10-25 Hz) and gamma (25-50 Hz) frequency bands and mutual information function. The study shows that reflex seizures elicit highly dynamic changes in interhemispheric functional coupling with seizure-, region- and frequency-specific patterns of increased and decreased connectivity during late ictal and immediate postictal periods. Also, secondary generalization of recurrent seizures (kindling) is associated with pronounced alterations in resting-state functional connectivity - an early wideband decrease and a subsequent beta-gamma increase. The findings show that intracortical functional connectivity is dynamically modified in response to seizures on short and long timescales, suggesting the existence of activity-dependent plastic network alterations that may promote or prevent seizure propagation within the cortex and underlie postictal behavioral impairments.

Diversity and interdisciplinarity: Should variety, balance and disparity be combined as a product or better as a sum? An information-theoretical and statistical estimation approach

Diversity is a central concept not only in ecology, but also in the social sciences and in bibliometrics. The discussion about an adequate measure of diversity is strongly driven by the work of Rao (Sankhyā Indian J Stat Series A 44:1-22, 1982) and Stirling (J R Soc Interface 4:707-719, 2007). It is to the credit of Leydesdorff (Scientometr 116:2113-2121, 2018) to have proposed a decisive improvement with regard to an inconsistency in the Rao-Sterling-diversity indicator that Rousseau (Scientometr 116:645-653, 2018) had pointed out. With recourse to Shannon's probabilistically based entropy concept, in this contribution the three components of diversity “variety”, “balance”, and “disparity” are to be reconceptualized as entropy masses that add up to an overall diversity indicator dive. Diversity can thus be interpreted as the degree of uncertainty or unpredictability. For "disparity", for example, the concept of mutual information is used. However, probabilities must be estimated statistically. A basic estimation strategy (cross tables) and a more sophisticated one (parametric statistical model) are presented. This overall probability-theoretical based concept is applied exemplarily to data on research output types of funded research projects in UK that were the subject of the Metric Tide Report (REF 2014) and ex-ante evaluation data of a research funding organization. As expected, research output types depend on the research area, with journal articles having the strongest individual balance among the output types, i.e., being represented in almost all research areas. For the ex-ante evaluation data of 1,221 funded projects the diversity components were statistically estimated. The overall diversity of the projects in terms of entropy is 55.5% of the maximal possible entropy.

Evaluation of Non-Uniform Image Quality Caused by Anode Heel Effect in Digital Radiography Using Mutual Information

Anode heel effects are known to cause non-uniform image quality, but no method has been proposed to evaluate the non-uniform image quality caused by the heel effect. Therefore, the purpose of this study was to evaluate non-uniform image quality in digital radiographs using a novel circular step-wedge (CSW) phantom and normalized mutual information (nMI). All X-ray images were acquired from a digital radiography system equipped with a CsI flat panel detector. A new acrylic CSW phantom was imaged ten times at various kVp and mAs to evaluate overall and non-uniform image quality with nMI metrics. For comparisons, a conventional contrast-detail resolution phantom was imaged ten times at identical exposure parameters to evaluate overall image quality with visible ratio (VR) metrics, and the phantom was placed in different orientations to assess non-uniform image quality. In addition, heel effect correction (HEC) was executed to elucidate the impact of its effect on image quality. The results showed that both nMI and VR metrics significantly changed with kVp and mAs, and had a significant positive correlation. The positive correlation is suggestive that the nMI metrics have a similar performance to the VR metrics in assessing the overall image quality of digital radiographs. The nMI metrics significantly changed with orientations and also significantly increased after HEC in the anode direction. However, the VR metrics did not change significantly with orientations or with HEC. The results indicate that the nMI metrics were more sensitive than the VR metrics with regards to non-uniform image quality caused by the anode heel effect. In conclusion, the proposed nMI metrics with a CSW phantom outperformed the conventional VR metrics in detecting non-uniform image quality caused by the heel effect, and thus are suitable for quantitatively evaluating non-uniform image quality in digital radiographs with and without HEC.

Signal Fluctuations and the Information Transmission Rates in Binary Communication Channels

In the nervous system, information is conveyed by sequence of action potentials, called spikes-trains. As MacKay and McCulloch suggested, spike-trains can be represented as bits sequences coming from Information Sources (IS). Previously, we studied relations between spikes’ Information Transmission Rates (ITR) and their correlations, and frequencies. Now, I concentrate on the problem of how spikes fluctuations affect ITR. The IS are typically modeled as stationary stochastic processes, which I consider here as two-state Markov processes. As a spike-trains’ fluctuation measure, I assume the standard deviation σ, which measures the average fluctuation of spikes around the average spike frequency. I found that the character of ITR and signal fluctuations relation strongly depends on the parameter s being a sum of transitions probabilities from a no spike state to spike state. The estimate of the Information Transmission Rate was found by expressions depending on the values of signal fluctuations and parameter s. It turned out that for smaller s<1, the quotient ITRσ has a maximum and can tend to zero depending on transition probabilities, while for 1<s, the ITRσ is separated from 0. Additionally, it was also shown that ITR quotient by variance behaves in a completely different way. Similar behavior was observed when classical Shannon entropy terms in the Markov entropy formula are replaced by their approximation with polynomials. My results suggest that in a noisier environment (1<s), to get appropriate reliability and efficiency of transmission, IS with higher tendency of transition from the no spike to spike state should be applied. Such selection of appropriate parameters plays an important role in designing learning mechanisms to obtain networks with higher performance.

Causal Network Inference for Neural Ensemble Activity

Interactions among cellular components forming a mesoscopic scale brain network (microcircuit) display characteristic neural dynamics. Analysis of microcircuits provides a system-level understanding of the neurobiology of health and disease. Causal discovery aims to detect causal relationships among variables based on observational data. A key barrier in causal discovery is the high dimensionality of the variable space. A method called Causal Inference for Microcircuits (CAIM) is proposed to reconstruct causal networks from calcium imaging or electrophysiology time series. CAIM combines neural recording, Bayesian network modeling, and neuron clustering. Validation experiments based on simulated data and a real-world reaching task dataset demonstrated that CAIM accurately revealed causal relationships among neural clusters.

Upper Limit on the Thermodynamic Information Content of an Action Potential

In computational neuroscience, spiking neurons are often analyzed as computing devices that register bits of information, with each action potential carrying at most one bit of Shannon entropy. Here, I question this interpretation by using Landauer's principle to estimate an upper limit for the quantity of thermodynamic information that can be processed within a single action potential in a typical mammalian neuron. A straightforward calculation shows that an action potential in a typical mammalian cortical pyramidal cell can process up to approximately 3.4 · 1011 bits of thermodynamic information, or about 4.9 · 1011 bits of Shannon entropy. This result suggests that an action potential can, in principle, carry much more than a single bit of Shannon entropy.

Limitations to Estimating Mutual Information in Large Neural Populations

Information theory provides a powerful framework to analyse the representation of sensory stimuli in neural population activity. However, estimating the quantities involved such as entropy and mutual information from finite samples is notoriously hard and any direct estimate is known to be heavily biased. This is especially true when considering large neural populations. We study a simple model of sensory processing and show through a combinatorial argument that, with high probability, for large neural populations any finite number of samples of neural activity in response to a set of stimuli is mutually distinct. As a consequence, the mutual information when estimated directly from empirical histograms will be equal to the stimulus entropy. Importantly, this is the case irrespective of the precise relation between stimulus and neural activity and corresponds to a maximal bias. This argument is general and applies to any application of information theory, where the state space is large and one relies on empirical histograms. Overall, this work highlights the need for alternative approaches for an information theoretic analysis when dealing with large neural populations.

Information processing in the LGN: a comparison of neural codes and cell types

To understand how anatomy and physiology allow an organism to perform its function, it is important to know how information that is transmitted by spikes in the brain is received and encoded. A natural question is whether the spike rate alone encodes the information about a stimulus (rate code), or additional information is contained in the temporal pattern of the spikes (temporal code). Here we address this question using data from the cat Lateral Geniculate Nucleus (LGN), which is the visual portion of the thalamus, through which visual information from the retina is communicated to the visual cortex. We analyzed the responses of LGN neurons to spatially homogeneous spots of various sizes with temporally random luminance modulation. We compared the Firing Rate with the Shannon Information Transmission Rate , which quantifies the information contained in the temporal relationships between spikes. We found that the behavior of these two rates can differ quantitatively. This suggests that the energy used for spiking does not translate directly into the information to be transmitted. We also compared Firing Rates with Information Rates for X-ON and X-OFF cells. We found that, for X-ON cells the Firing Rate and Information Rate often behave in a completely different way, while for X-OFF cells these rates are much more highly correlated. Our results suggest that for X-ON cells a more efficient "temporal code" is employed, while for X-OFF cells a straightforward "rate code" is used, which is more reliable and is correlated with energy consumption.

How Far can Neural Correlations Reduce Uncertainty? Comparison of Information Transmission Rates for Markov and Bernoulli Processes

The nature of neural codes is central to neuroscience. Do neurons encode information through relatively slow changes in the firing rates of individual spikes (rate code) or by the precise timing of every spike (temporal code)? Here we compare the loss of information due to correlations for these two possible neural codes. The essence of Shannon's definition of information is to combine information with uncertainty: the higher the uncertainty of a given event, the more information is conveyed by that event. Correlations can reduce uncertainty or the amount of information, but by how much? In this paper we address this question by a direct comparison of the information per symbol conveyed by the words coming from a binary Markov source (temporal code) with the information per symbol coming from the corresponding Bernoulli source (uncorrelated, rate code). In a previous paper we found that a crucial role in the relation between information transmission rates (ITRs) and firing rates is played by a parameter s, which is the sum of transition probabilities from the no-spike state to the spike state and vice versa. We found that in this case too a crucial role is played by the same parameter s. We calculated the maximal and minimal bounds of the quotient of ITRs for these sources. Next, making use of the entropy grouping axiom, we determined the loss of information in a Markov source compared with the information in the corresponding Bernoulli source for a given word length. Our results show that in the case of correlated signals the loss of information is relatively small, and thus temporal codes, which are more energetically efficient, can replace rate codes effectively. These results were confirmed by experiments.

Critical noise can make the minority candidate win: The U.S. presidential election cases

A national voting population, when segmented into groups such as, for example, different states, can yield a counterintuitive scenario in which the winner may not necessarily get the highest number of total votes. A recent example is the 2016 presidential election in the United States. We model the situation by using interacting opinion dynamics models, and we look at the effect of coarse graining near the critical points where the spatial fluctuations are high. We establish that the sole effect of coarse graining, which mimics the "winner take all" electoral college system in the United States, can give rise to finite probabilities of events in which a minority candidate wins even in the large size limit near the critical point. The overall probabilities of victory of the minority candidate can be predicted from the models, which indicate that one may expect more instances of minority candidate winning in the future.