Error-probability noise benefits in threshold neural signal detection

Noise can speed backpropagation learning and deep bidirectional pretraining

We show that the backpropagation algorithm is a special case of the generalized Expectation-Maximization (EM) algorithm for iterative maximum likelihood estimation. We then apply the recent result that carefully chosen noise can speed the average convergence of the EM algorithm as it climbs a hill of probability or log-likelihood. Then injecting such noise can speed the average convergence of the backpropagation algorithm for both the training and pretraining of multilayer neural networks. The beneficial noise adds to the hidden and visible neurons and related parameters. The noise also applies to regularized regression networks. This beneficial noise is just that noise that makes the current signal more probable. We show that such noise also tends to improve classification accuracy. The geometry of the noise-benefit region depends on the probability structure of the neurons in a given layer. The noise-benefit region in noise space lies above the noisy-EM (NEM) hyperplane for classification and involves a hypersphere for regression. Simulations demonstrate these noise benefits using MNIST digit classification. The NEM noise benefits substantially exceed those of simply adding blind noise to the neural network. We further prove that the noise speed-up applies to the deep bidirectional pretraining of neural-network bidirectional associative memories (BAMs) or their functionally equivalent restricted Boltzmann machines. We then show that learning with basic contrastive divergence also reduces to generalized EM for an energy-based network probability. The optimal noise adds to the input visible neurons of a BAM in stacked layers of trained BAMs. Global stability of generalized BAMs guarantees rapid convergence in pretraining where neural signals feed back between contiguous layers. Bipolar coding of inputs further improves pretraining performance.

Noise-enhanced convolutional neural networks

Injecting carefully chosen noise can speed convergence in the backpropagation training of a convolutional neural network (CNN). The Noisy CNN algorithm speeds training on average because the backpropagation algorithm is a special case of the generalized expectation-maximization (EM) algorithm and because such carefully chosen noise always speeds up the EM algorithm on average. The CNN framework gives a practical way to learn and recognize images because backpropagation scales with training data. It has only linear time complexity in the number of training samples. The Noisy CNN algorithm finds a special separating hyperplane in the network's noise space. The hyperplane arises from the likelihood-based positivity condition that noise-boosts the EM algorithm. The hyperplane cuts through a uniform-noise hypercube or Gaussian ball in the noise space depending on the type of noise used. Noise chosen from above the hyperplane speeds training on average. Noise chosen from below slows it on average. The algorithm can inject noise anywhere in the multilayered network. Adding noise to the output neurons reduced the average per-iteration training-set cross entropy by 39% on a standard MNIST image test set of handwritten digits. It also reduced the average per-iteration training-set classification error by 47%. Adding noise to the hidden layers can also reduce these performance measures. The noise benefit is most pronounced for smaller data sets because the largest EM hill-climbing gains tend to occur in the first few iterations. This noise effect can assist random sampling from large data sets because it allows a smaller random sample to give the same or better performance than a noiseless sample gives.

Noise-benefit forbidden-interval theorems for threshold signal detectors based on cross correlations

We show that the main forbidden interval theorems of stochastic resonance hold for a correlation performance measure. Earlier theorems held only for performance measures based on mutual information or the probability of error detection. Forbidden interval theorems ensure that a threshold signal detector benefits from deliberately added noise if the average noise does not lie in an interval that depends on the threshold value. We first show that this result holds for correlation for all finite-variance noise and for all forms of infinite-variance stable noise. A second forbidden-interval theorem gives necessary and sufficient conditions for a local noise benefit in a bipolar signal system when the noise comes from a location-scale family. A third theorem gives a general condition for a local noise benefit for arbitrary signals with finite second moments and for location-scale noise. This result also extends forbidden intervals to forbidden bands of parameters. A fourth theorem gives necessary and sufficient conditions for a local noise benefit when both the independent signal and noise are normal. A final theorem derives necessary and sufficient conditions for forbidden bands when using arrays of threshold detectors for arbitrary signals and location-scale noise.

Stochastic resonance with colored noise for neural signal detection

We analyze signal detection with nonlinear test statistics in the presence of colored noise. In the limits of small signal and weak noise correlation, the optimal test statistic and its performance are derived under general conditions, especially concerning the type of noise. We also analyze, for a threshold nonlinearity–a key component of a neural model, the conditions for noise-enhanced performance, establishing that colored noise is superior to white noise for detection. For a parallel array of nonlinear elements, approximating neurons, we demonstrate even broader conditions allowing noise-enhanced detection, via a form of suprathreshold stochastic resonance.

Optimal sensor selection for noisy binary detection in stochastic pooling networks

Stochastic Pooling Networks (SPNs) are a useful model for understanding and explaining how naturally occurring encoding of stochastic processes can occur in sensor systems ranging from macroscopic social networks to neuron populations and nanoscale electronics. Due to the interaction of nonlinearity, random noise, and redundancy, SPNs support various unexpected emergent features, such as suprathreshold stochastic resonance, but most existing mathematical results are restricted to the simplest case where all sensors in a network are identical. Nevertheless, numerical results on information transmission have shown that in the presence of independent noise, the optimal configuration of a SPN is such that there should be partial heterogeneity in sensor parameters, such that the optimal solution includes clusters of identical sensors, where each cluster has different parameter values. In this paper, we consider a SPN model of a binary hypothesis detection task and show mathematically that the optimal solution for a specific bound on detection performance is also given by clustered heterogeneity, such that measurements made by sensors with identical parameters either should all be excluded from the detection decision or all included. We also derive an algorithm for numerically finding the optimal solution and illustrate its utility with several examples, including a model of parallel sensory neurons with Poisson firing characteristics.

Noise-enhanced clustering and competitive learning algorithms

Noise can provably speed up convergence in many centroid-based clustering algorithms. This includes the popular k-means clustering algorithm. The clustering noise benefit follows from the general noise benefit for the expectation-maximization algorithm because many clustering algorithms are special cases of the expectation-maximization algorithm. Simulations show that noise also speeds up convergence in stochastic unsupervised competitive learning, supervised competitive learning, and differential competitive learning.

Noise can speed convergence in Markov chains

A new theorem shows that noise can speed convergence to equilibrium in discrete finite-state Markov chains. The noise applies to the state density and helps the Markov chain explore improbable regions of the state space. The theorem ensures that a stochastic-resonance noise benefit exists for states that obey a vector-norm inequality. Such noise leads to faster convergence because the noise reduces the norm components. A corollary shows that a noise benefit still occurs if the system states obey an alternate norm inequality. This leads to a noise-benefit algorithm that requires knowledge of the steady state. An alternative blind algorithm uses only past state information to achieve a weaker noise benefit. Simulations illustrate the predicted noise benefits in three well-known Markov models. The first model is a two-parameter Ehrenfest diffusion model that shows how noise benefits can occur in the class of birth-death processes. The second model is a Wright-Fisher model of genotype drift in population genetics. The third model is a chemical reaction network of zeolite crystallization. A fourth simulation shows a convergence rate increase of 64% for states that satisfy the theorem and an increase of 53% for states that satisfy the corollary. A final simulation shows that even suboptimal noise can speed convergence if the noise applies over successive time cycles. Noise benefits tend to be sharpest in Markov models that do not converge quickly and that do not have strong absorbing states.

Synaptic signal transduction aided by noise in a dynamical saturating model

A generic dynamical model with saturation for neural signal transduction at the synaptic stage is presented. Analysis of this model of a synaptic pathway demonstrates its ability to give rise to stochastic resonance or improvement by noise, at this stage of signal transmission. Beyond the case of the intrinsic threshold nonlinearity of the neuron response, the results extend the feasibility of stochastic resonance to neural saturating dynamics at the synaptic stage. The present results also constitute the exposition of a new type of nonlinear (saturating) dynamics capable of stochastic resonance.