Glauber dynamics of the Little-Hopfield model

Learning of spatiotemporal patterns in Ising-spin neural networks: analysis of storage capacity by path integral methods

We encode periodic spatiotemporal patterns in Ising-spin neural networks, using the simple learning rule inspired by the spike-timing-dependent synaptic plasticity. It is then found that periodically oscillating spin neurons successfully reproduce phase differences of the encoded periodic patterns. The storage capacity of this associative memory neural network is enhanced with an adequate level of asymmetry in synapse connections. To understand the properties of these nonequilibrium retrieval states of the neural network, we carry out an analysis based on a path integral method. The relation of a dynamic crosstalk term to time-persistent oscillation of a correlation function well explains the enhancement of the storage capacity in spite of our approximation on nonpersistent terms. We investigate the accuracy of this approximation further by detailed comparison with numerical simulations.

Pattern reconstruction and sequence processing in feed-forward layered neural networks near saturation

The dynamics and the stationary states for the competition between pattern reconstruction and asymmetric sequence processing are studied here in an exactly solvable feed-forward layered neural network model of binary units and patterns near saturation. Earlier work by Coolen and Sherrington on a parallel dynamics far from saturation is extended here to account for finite stochastic noise due to a Hebbian and a sequential learning rule. Phase diagrams are obtained with stationary states and quasiperiodic nonstationary solutions. The relevant dependence of these diagrams and of the quasiperiodic solutions on the stochastic noise and on initial inputs for the overlaps is explicitly discussed.

Notions of Associative Memory and Sparse Coding

This paper summarizes associative memory models and sparse representation of memory in these models. Important properties of the associative memory models are their storage capacity, basin of attraction, and the existence of spurious memories. Sparse coding and nonmonotonic output functions are proposed to improve them. Sparsely coded associative memory model has an extremely large storage capacity which diverges as the mean firing rate of memory patterns approaches 0. The storage capacity strongly depends on the shape of the output function as well as the mean firing rate, even in the case of monotonic output functions. Dynamical properties of the model are analyzed by means of a statistical neurodynamical method. We emphasize the necessity of a feedback mechanism to control the mean firing rate in the recall process. Recently, there have been some experimental results suggesting its existence in the brain. On the other hand, it has been shown that the storage capacity can be markedly improved by replacing the usual monotonic output function with a nonmonotonic one. Another remarkable property of the model with the nonmonotonic neurons is that it seems to have no, or almost no, spurious memory. An associative memory model using nonmonotonic modules with a feedforward inhibition is discussed. The modules consist of two types of threshold units, each of which has a different threshold and can be considered as a biologically plausible representation of the nonmonotonic output function. The above model is compared with the monotonic one. The difference in the storage capacity between the two models becomes small when the sparse patterns are stored. Finally, we discuss the biological plausibility of the discussed associative memory models and sparse coding. Copyright 1996 Elsevier Science Ltd.