How Far can Neural Correlations Reduce Uncertainty? Comparison of Information Transmission Rates for Markov and Bernoulli Processes

Equivalence of information production and generalised entropies in complex processes

Complex systems with strong correlations and fat-tailed distribution functions have been argued to be incompatible with the Boltzmann-Gibbs entropy framework and alternatives, so-called generalised entropies, were proposed and studied. Here we show, that this perceived incompatibility is actually a misconception. For a broad class of processes, Boltzmann entropy -the log multiplicity- remains the valid entropy concept. However, for non-i.i.d. processes, Boltzmann entropy is not of Shannon form, -k∑ipi log pi, but takes the shape of generalised entropies. We derive this result for all processes that can be asymptotically mapped to adjoint representations reversibly where processes are i.i.d. In these representations the information production is given by the Shannon entropy. Over the original sampling space this yields functionals identical to generalised entropies. The problem of constructing adequate context-sensitive entropy functionals therefore can be translated into the much simpler problem of finding adjoint representations. The method provides a comprehensive framework for a statistical physics of strongly correlated systems and complex processes.

A Perspective on Information Optimality in a Neural Circuit and Other Biological Systems

The nematode worm Caenorhabditis elegans has a relatively simple neural system for analysis of information transmission from sensory organ to muscle fiber. Consequently, this study includes an example of a neural circuit from the nematode worm, and a procedure is shown for measuring its information optimality by use of a logic gate model. This approach is useful where the assumptions are applicable for a neural circuit, and also for choosing between competing mathematical hypotheses that explain the function of a neural circuit. In this latter case, the logic gate model can estimate computational complexity and distinguish which of the mathematical models require fewer computations. In addition, the concept of information optimality is generalized to other biological systems, along with an extended discussion of its role in genetic-based pathways of organisms.

Towards the Idea of Molecular Brains

How can single cells without nervous systems perform complex behaviours such as habituation, associative learning and decision making, which are considered the hallmark of animals with a brain? Are there molecular systems that underlie cognitive properties equivalent to those of the brain? This review follows the development of the idea of molecular brains from Darwin’s “root brain hypothesis”, through bacterial chemotaxis, to the recent discovery of neuron-like r-protein networks in the ribosome. By combining a structural biology view with a Bayesian brain approach, this review explores the evolutionary labyrinth of information processing systems across scales. Ribosomal protein networks open a window into what were probably the earliest signalling systems to emerge before the radiation of the three kingdoms. While ribosomal networks are characterised by long-lasting interactions between their protein nodes, cell signalling networks are essentially based on transient interactions. As a corollary, while signals propagated in persistent networks may be ephemeral, networks whose interactions are transient constrain signals diffusing into the cytoplasm to be durable in time, such as post-translational modifications of proteins or second messenger synthesis. The duration and nature of the signals, in turn, implies different mechanisms for the integration of multiple signals and decision making. Evolution then reinvented networks with persistent interactions with the development of nervous systems in metazoans. Ribosomal protein networks and simple nervous systems display architectural and functional analogies whose comparison could suggest scale invariance in information processing. At the molecular level, the significant complexification of eukaryotic ribosomal protein networks is associated with a burst in the acquisition of new conserved aromatic amino acids. Knowing that aromatic residues play a critical role in allosteric receptors and channels, this observation suggests a general role of π systems and their interactions with charged amino acids in multiple signal integration and information processing. We think that these findings may provide the molecular basis for designing future computers with organic processors.

Movement Analysis for Neurological and Musculoskeletal Disorders Using Graph Convolutional Neural Network

Using optical motion capture and wearable sensors is a common way to analyze impaired movement in individuals with neurological and musculoskeletal disorders. However, using optical motion sensors and wearable sensors is expensive and often requires highly trained professionals to identify specific impairments. In this work, we proposed a graph convolutional neural network that mimics the intuition of physical therapists to identify patient-specific impairments based on video of a patient. In addition, two modeling approaches are compared: a graph convolutional network applied solely on skeleton input data and a graph convolutional network accompanied with a 1-dimensional convolutional neural network (1D-CNN). Experiments on the dataset showed that the proposed method not only improves the correlation of the predicted gait measure with the ground truth value (speed = 0.791, gait deviation index (GDI) = 0.792) but also enables faster training with fewer parameters. In conclusion, the proposed method shows that the possibility of using video-based data to treat neurological and musculoskeletal disorders with acceptable accuracy instead of depending on the expensive and labor-intensive optical motion capture systems.

A Low-Power Spiking Neural Network Chip Based on a Compact LIF Neuron and Binary Exponential Charge Injector Synapse Circuits

To realize a large-scale Spiking Neural Network (SNN) on hardware for mobile applications, area and power optimized electronic circuit design is critical. In this work, an area and power optimized hardware implementation of a large-scale SNN for real time IoT applications is presented. The analog Complementary Metal Oxide Semiconductor (CMOS) implementation incorporates neuron and synaptic circuits optimized for area and power consumption. The asynchronous neuronal circuits implemented benefit from higher energy efficiency and higher sensitivity. The proposed synapse circuit based on Binary Exponential Charge Injector (BECI) saves area and power consumption, and provides design scalability for higher resolutions. The SNN model implemented is optimized for 9 × 9 pixel input image and minimum bit-width weights that can satisfy target accuracy, occupies less area and power consumption. Moreover, the spiking neural network is replicated in full digital implementation for area and power comparisons. The SNN chip integrated from neuron and synapse circuits is capable of pattern recognition. The proposed SNN chip is fabricated using 180 nm CMOS process, which occupies a 3.6 mm² chip core area, and achieves a classification accuracy of 94.66% for the MNIST dataset. The proposed SNN chip consumes an average power of 1.06 mW—20 times lower than the digital implementation.

Comparison of Sneo-Based Neural Spike Detection Algorithms for Implantable Multi-Transistor Array Biosensors

Real-time neural spike detection is an important step in understanding neurological activities and developing brain-silicon interfaces. Recent approaches exploit minimally invasive sensing techniques based on implanted complementary metal-oxide semiconductors (CMOS) multi transistors arrays (MTAs) that limit the damage of the neural tissue and provide high spatial resolution. Unfortunately, MTAs result in low signal-to-noise ratios due to the weak capacitive coupling between the nearby neurons and the sensor and the high noise power coming from the analog front-end. In this paper we investigate the performance achievable by using spike detection algorithms for MTAs, based on some variants of the smoothed non-linear energy operator (SNEO). We show that detection performance benefits from the correlation of the signals detected by the MTA pixels, but degrades when a high firing rate of neurons occurs. We present and compare different approaches and noise estimation techniques for the SNEO, aimed at increasing the detection accuracy at low SNR and making it less dependent on neurons firing rates. The algorithms are tested by using synthetic neural signals obtained with a modified version of NEUROCUBE generator. The proposed approaches outperform the SNEO, showing a more than 20% increase on averaged sensitivity at 0 dB and reduced dependence on the neuronal firing rate.

Signal Fluctuations and the Information Transmission Rates in Binary Communication Channels

In the nervous system, information is conveyed by sequence of action potentials, called spikes-trains. As MacKay and McCulloch suggested, spike-trains can be represented as bits sequences coming from Information Sources (IS). Previously, we studied relations between spikes’ Information Transmission Rates (ITR) and their correlations, and frequencies. Now, I concentrate on the problem of how spikes fluctuations affect ITR. The IS are typically modeled as stationary stochastic processes, which I consider here as two-state Markov processes. As a spike-trains’ fluctuation measure, I assume the standard deviation σ, which measures the average fluctuation of spikes around the average spike frequency. I found that the character of ITR and signal fluctuations relation strongly depends on the parameter s being a sum of transitions probabilities from a no spike state to spike state. The estimate of the Information Transmission Rate was found by expressions depending on the values of signal fluctuations and parameter s. It turned out that for smaller s<1, the quotient ITRσ has a maximum and can tend to zero depending on transition probabilities, while for 1<s, the ITRσ is separated from 0. Additionally, it was also shown that ITR quotient by variance behaves in a completely different way. Similar behavior was observed when classical Shannon entropy terms in the Markov entropy formula are replaced by their approximation with polynomials. My results suggest that in a noisier environment (1<s), to get appropriate reliability and efficiency of transmission, IS with higher tendency of transition from the no spike to spike state should be applied. Such selection of appropriate parameters plays an important role in designing learning mechanisms to obtain networks with higher performance.

Design and Implementation of a Spiking Neural Network with Integrate-and-Fire Neuron Model for Pattern Recognition

In contrast to the previous artificial neural networks (ANNs), spiking neural networks (SNNs) work based on temporal coding approaches. In the proposed SNN, the number of neurons, neuron models, encoding method, and learning algorithm design are described in a correct and pellucid fashion. It is also discussed that optimizing the SNN parameters based on physiology, and maximizing the information they pass leads to a more robust network. In this paper, inspired by the "center-surround" structure of the receptive fields in the retina, and the amount of overlap that they have, a robust SNN is implemented. It is based on the Integrate-and-Fire (IF) neuron model and uses the time-to-first-spike coding to train the network by a newly proposed method. The Iris and MNIST datasets were employed to evaluate the performance of the proposed network whose accuracy, with 60 input neurons, was 96.33% on the Iris dataset. The network was trained in only 45 iterations indicating its reasonable convergence rate. For the MNIST dataset, when the gray level of each pixel was considered as input to the network, 600 input neurons were required, and the accuracy of the network was 90.5%. Next, 14 structural features were used as input. Therefore, the number of input neurons decreased to 210, and accuracy increased up to 95%, meaning that an SNN with fewer input neurons and good skill was implemented. Also, the ABIDE1 dataset is applied to the proposed SNN. Of the 184 data, 79 are used for healthy people and 105 for people with autism. One of the characteristics that can differentiate between these two classes is the entropy of the existing data. Therefore, Shannon entropy is used for feature extraction. Applying these values to the proposed SNN, an accuracy of 84.42% was achieved by only 120 iterations, which is a good result compared to the recent results.

Information Processing in the Brain as Optimal Entropy Transport: A Theoretical Approach

We consider brain activity from an information theoretic perspective. We analyze the information processing in the brain, considering the optimality of Shannon entropy transport using the Monge–Kantorovich framework. It is proposed that some of these processes satisfy an optimal transport of informational entropy condition. This optimality condition allows us to derive an equation of the Monge–Ampère type for the information flow that accounts for the branching structure of neurons via the linearization of this equation. Based on this fact, we discuss a version of Murray’s law in this context.