Estimating functions of probability distributions from a finite set of samples

Optimal instruments and models for noisy chaos

Analysis of finite, noisy time series data leads to modern statistical inference methods. Here we adapt Bayesian inference for applied symbolic dynamics. We show that reconciling Kolmogorov's maximum-entropy partition with the methods of Bayesian model selection requires the use of two separate optimizations. First, instrument design produces a maximum-entropy symbolic representation of time series data. Second, Bayesian model comparison with a uniform prior selects a minimum-entropy model, with respect to the considered Markov chain orders, of the symbolic data. We illustrate these steps using a binary partition of time series data from the logistic and Henon maps as well as the Rössler and Lorenz attractors with dynamical noise. In each case we demonstrate the inference of effectively generating partitions and kth-order Markov chain models.

A multistage knowledge-based system for EEG seizure detection in newborn infants

OBJECTIVE

Automatic seizure detection has attracted attention as a method to obtain valuable information concerning the duration, timing, and frequency of seizures. Methods currently used to detect EEG seizures in adults show high false detection rates in neonates because they lack information about specific age-dependent features of normal and pathological EEG and artifacts. This paper describes a novel multistage knowledge-based seizure detection system for newborn infants to identify and classify normal and pathological newborn EEGs as well as seizures with a reduced false detection rate.

METHODS

We developed the system in a way to make comprehensive use of spatial and temporal contextual information obtained from multichannel EEGs. The system development consists of six major stages: (i) EEG data collection and bandpass filtering; (ii) automatic artifact detection; (iii) feature extraction from segments of non-seizure and seizure activities; (iv) feature selection via the relevance and redundancy analysis; (v) EEG classification and pattern recognition using a trained multilayer back-propagation neural network; and (v) knowledge-based decision-making to examine each of possible EEG patterns from a multi-channel perspective. The system was developed and tested with the EEG recordings of 10 newborns aged between 39 and 42 weeks.

RESULTS

The overall sensitivity, selectivity, and average detection rate of the system were 74%, 70.1%, and 79.7%, respectively. The average false detection of 1.55/h was also achieved by the system with a feature reduction up to 80%.

CONCLUSIONS

The expert rule-based decision-making subsystem accompanying the classifier helped to reduce the false detection rate, reject a wide variety of artifacts, and discriminate various patterns of EEG.

SIGNIFICANCE

This paper may serve as a guide for the selection of discriminative features to improve the accuracy of conventional seizure detection systems for routine clinical EEG interpretation and brain activity monitoring in newborns especially those hospitalized in the neonatal intensive care units.

Estimating information rates with confidence intervals in neural spike trains

Information theory provides a natural set of statistics to quantify the amount of knowledge a neuron conveys about a stimulus. A related work (Kennel, Shlens, Abarbanel, & Chichilnisky, 2005) demonstrated how to reliably estimate, with a Bayesian confidence interval, the entropy rate from a discrete, observed time series. We extend this method to measure the rate of novel information that a neural spike train encodes about a stimulus--the average and specific mutual information rates. Our estimator makes few assumptions about the underlying neural dynamics, shows excellent performance in experimentally relevant regimes, and uniquely provides confidence intervals bounding the range of information rates compatible with the observed spike train. We validate this estimator with simulations of spike trains and highlight how stimulus parameters affect its convergence in bias and variance. Finally, we apply these ideas to a recording from a guinea pig retinal ganglion cell and compare results to a simple linear decoder.

Measurements of protein sequence-structure correlations

Correlations between protein structures and amino acid sequences are widely used for protein structure prediction. For example, secondary structure predictors generally use correlations between a secondary structure sequence and corresponding primary structure sequence, whereas threading algorithms and similar tertiary structure predictors typically incorporate interresidue contact potentials. To investigate the relative importance of these sequence-structure interactions, we measured the mutual information among the primary structure, secondary structure and side-chain surface exposure, both for adjacent residues along the amino acid sequence and for tertiary structure contacts between residues distantly separated along the backbone. We found that local interactions along the amino acid chain are far more important than non-local contacts and that correlations between proximate amino acids are essentially uninformative. This suggests that knowledge-based contact potentials may be less important for structure predication than is generally believed.

Approaches to Information-Theoretic Analysis of Neural Activity

Understanding how neurons represent, process, and manipulate information is one of the main goals of neuroscience. These issues are fundamentally abstract, and information theory plays a key role in formalizing and addressing them. However, application of information theory to experimental data is fraught with many challenges. Meeting these challenges has led to a variety of innovative analytical techniques, with complementary domains of applicability, assumptions, and goals.

Information processing in noisy burster models of sensory neurons

Processing of external stimuli by sensory neurons often involves bursting, when epochs of fast firing alternate with intervals of quiescence. In particular, sensory neurons of electroreceptors in paddlefish (Polyodon spathula) undergo bursting when stimulated externally with broad-band noise, but otherwise fire spontaneously in a quasiperiodic tonic manner. We use a simple phenomenological model for noise-induced bursting to quantify analytically, by means of the Kullback entropy and Fisher information, the gain in information transfer and electroreceptor sensitivity for external noisy stimuli. A good agreement between theoretical predictions, numerical simulations and experimental data is shown.

Estimating entropy rates with Bayesian confidence intervals

The entropy rate quantifies the amount of uncertainty or disorder produced by any dynamical system. In a spiking neuron, this uncertainty translates into the amount of information potentially encoded and thus the subject of intense theoretical and experimental investigation. Estimating this quantity in observed, experimental data is difficult and requires a judicious selection of probabilistic models, balancing between two opposing biases. We use a model weighting principle originally developed for lossless data compression, following the minimum description length principle. This weighting yields a direct estimator of the entropy rate, which, compared to existing methods, exhibits significantly less bias and converges faster in simulation. With Monte Carlo techinques, we estimate a Bayesian confidence interval for the entropy rate. In related work, we apply these ideas to estimate the information rates between sensory stimuli and neural responses in experimental data (Shlens, Kennel, Abarbanel, & Chichilnisky, in preparation).

Estimating the temporal interval entropy of neuronal discharge

To better understand the role of timing in the function of the nervous system, we have developed a methodology that allows the entropy of neuronal discharge activity to be estimated from a spike train record when it may be assumed that successive interspike intervals are temporally uncorrelated. The so-called interval entropy obtained by this methodology is based on an implicit enumeration of all possible spike trains that are statistically indistinguishable from a given spike train. The interval entropy is calculated from an analytic distribution whose parameters are obtained by maximum likelihood estimation from the interval probability distribution associated with a given spike train. We show that this approach reveals features of neuronal discharge not seen with two alternative methods of entropy estimation. The methodology allows for validation of the obtained data models by calculation of confidence intervals for the parameters of the analytic distribution and the testing of the significance of the fit between the observed and analytic interval distributions by means of Kolmogorov-Smirnov and Anderson-Darling statistics. The method is demonstrated by analysis of two different data sets: simulated spike trains evoked by either Poissonian or near-synchronous pulsed activation of a model cerebellar Purkinje neuron and spike trains obtained by extracellular recording from spontaneously discharging cultured rat hippocampal neurons.

Entropy and information in neural spike trains: progress on the sampling problem

The major problem in information theoretic analysis of neural responses and other biological data is the reliable estimation of entropy-like quantities from small samples. We apply a recently introduced Bayesian entropy estimator to synthetic data inspired by experiments, and to real experimental spike trains. The estimator performs admirably even very deep in the undersampled regime, where other techniques fail. This opens new possibilities for the information theoretic analysis of experiments, and may be of general interest as an example of learning from limited data.

Estimation of Entropy and Mutual Information

We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods.

Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.

Hidden Markov models that use predicted local structure for fold recognition: alphabets of backbone geometry

An important problem in computational biology is predicting the structure of the large number of putative proteins discovered by genome sequencing projects. Fold-recognition methods attempt to solve the problem by relating the target proteins to known structures, searching for template proteins homologous to the target. Remote homologs that may have significant structural similarity are often not detectable by sequence similarities alone. To address this, we incorporated predicted local structure, a generalization of secondary structure, into two-track profile hidden Markov models (HMMs). We did not rely on a simple helix-strand-coil definition of secondary structure, but experimented with a variety of local structure descriptions, following a principled protocol to establish which descriptions are most useful for improving fold recognition and alignment quality. On a test set of 1298 nonhomologous proteins, HMMs incorporating a 3-letter STRIDE alphabet improved fold recognition accuracy by 15% over amino-acid-only HMMs and 23% over PSI-BLAST, measured by ROC-65 numbers. We compared two-track HMMs to amino-acid-only HMMs on a difficult alignment test set of 200 protein pairs (structurally similar with 3-24% sequence identity). HMMs with a 6-letter STRIDE secondary track improved alignment quality by 62%, relative to DALI structural alignments, while HMMs with an STR track (an expanded DSSP alphabet that subdivides strands into six states) improved by 40% relative to CE.

Information-theoretic dissection of pairwise contact potentials

Pairwise contact potentials have a long, successful history in protein structure prediction. They provide an easily-estimated representation of many attributes of protein structures, such as the hydrophobic effect. In order to improve on existing potentials, one should develop a clear understanding of precisely what information they convey. Here, using mutual information, we quantified the information in amino acid potentials, and the importance of hydropathy, charge, disulfide bonding, and burial. Sampling error in mutual information was controlled for by estimating how much information cannot be attributed to sampling bias. We found the information in amino acid contacts to be modest: 0.04 bits per contact. Of that, only 0.01 bits of information could not be attributed to hydropathy, charge, disulfide bonding, or burial.

Estimating probabilities from experimental frequencies

Estimating the probability distribution q governing the behavior of a certain variable by sampling its value a finite number of times most typically involves an error. Successive measurements allow the construction of a histogram, or frequency count f, of each of the possible outcomes. In this work, the probability that the true distribution be q, given that the frequency count f was sampled, is studied. Such a probability may be written as a Gibbs distribution. A thermodynamic potential, which allows an easy evaluation of the mean Kullback-Leibler divergence between the true and measured distribution, is defined. For a large number of samples, the expectation value of any function of q is expanded in powers of the inverse number of samples. As an example, the moments, the entropy, and the mutual information are analyzed.

DNA segmentation through the Bayesian approach

We present a new approach to DNA segmentation into compositionally homogeneous blocks. The Bayesian estimator, which is applicable for both short and long segments, is used to obtain the measure of homogeneity. An exact optimal segmentation is found via the dynamic programming technique. After completion of the segmentation procedure, the sequence composition on different scales can be analyzed with filtration of boundaries via the partition function approach.

The study of correlation structures of DNA sequences: a critical review

The study of correlation structure in the primary sequences of DNA is reviewed. The issues reviewed include: symmetries among 16 base-base correlation functions; accurate estimation of correlation measures; the relationship between 1/f and Lorentzian spectra; heterogeneity in DNA sequences; different modeling strategies of the correlation structure of DNA sequences; the difference of correlation structure between coding and non-coding regions (besides the period-3 pattern); and source of broad distribution of domain sizes. Although some of the results remain controversial, a body of work on this topic constitutes a good starting point for future studies.

Entropy estimation of symbol sequences

We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our interest is in describing their convergence with sequence length, assuming no limits for the space and time complexities of the compression algorithms. A scaling law is proposed for extrapolation from finite sample lengths. This is applied to sequences of dynamical systems in non-trivial chaotic regimes, a 1-D cellular automaton, and to written English texts. (c)1996 American Institute of Physics.