The signal-to-noise analysis of the Little–Hopfield model revisited

ADAPTIVE THRESHOLDS FOR NEURAL NETWORKS WITH SYNAPTIC NOISE

The inclusion of a macroscopic adaptive threshold is studied for the retrieval dynamics of both layered feedforward and fully connected neural network models with synaptic noise. These two types of architectures require a different method to be solved numerically. In both cases it is shown that, if the threshold is chosen appropriately as a function of the cross-talk noise and of the activity of the stored patterns, adapting itself automatically in the course of the recall process, an autonomous functioning of the network is guaranteed. This self-control mechanism considerably improves the quality of retrieval, in particular the storage capacity, the basins of attraction and the mutual information content.

Transient dynamics for sequence-processing neural networks: effect of degree distributions

We derive an analytic evolution equation for overlap parameters, including the effect of degree distribution on the transient dynamics of sequence processing neural networks. In the special case of globally coupled networks, the precisely retrieved critical loading ratio alpha_{c}=N;{-12} is obtained, where N is the network size. In the presence of random networks, our theoretical predictions agree quantitatively with the numerical experiments for delta, binomial, and power-law degree distributions.

Localized activity profiles and storage capacity of rate-based autoassociative networks

We study analytically the effect of metrically structured connectivity on the behavior of autoassociative networks. The steady state equations are derived for a generic input-output function, and then we focus on their solutions in the case of networks composed of three alternative simple rate-based model neurons: threshold-linear, binary or smoothly saturating units. For a connectivity which is short range enough the threshold-linear network shows localized retrieval states. The saturating and binary models also exhibit spatially modulated retrieval states if the highest activity level that they can achieve is above the maximum activity of the units in the stored patterns. We show that this saturation level together with the linear gain of the transfer function are important parameters that determine the possibility of localized retrieval. If the ratio of the number of stored patterns to the number of connections per unit goes to zero, while the latter goes to infinity, it is possible to derive an analytical formula for the critical value of the connectivity width, below which one observes spatially nonuniform retrieval states. The formula is also shown to offer a good first approximation for higher storage loads. We show that even in the case of localized retrieval the storage capacity remains proportional to the number of connections per neurons, with the proportionality constant lower by a factor of 3-4 compared to uniform retrieval. The approach that we present here is generic in the sense that there are no specific assumptions on the single unit input-output function nor on the exact connectivity structure.

Retrieval-time properties of the Little-Hopfield model and their physiological relevance

We perform an extensive numerical investigation on the retrieval dynamics of the synchronous Hopfield model, also known as Little-Hopfield model, up to sizes of 2(18) neurons. Our results correct and extend much of the early simulations on the model. We find that the average convergence time has a power law behavior for a wide range of system sizes, whose exponent depends both on the network loading and the initial overlap with the memory to be retrieved. Surprisingly, we also find that the variance of the convergence time grows as fast as its average, making it a non-self-averaging quantity. Based on the simulation data we differentiate between two definitions for memory retrieval time, one that is mathematically strict, tau(c), the number of updates needed to reach the attractor whose properties we just described, and a second definition correspondent to the time tau(eta) when the network stabilizes within a tolerance threshold eta such that the difference of two consecutive overlaps with a stored memory is smaller that eta. We show that the scaling relationships between tau(c) and tau(eta) and the typical network parameters as the memory load alpha or the size of the network N vary greatly, being tau(eta) relatively insensitive to system sizes and loading. We propose tau(eta) as the physiological realistic measure for the typical attractor network response.