Phase diagram and storage capacity of sequence processing neural networks

Characteristics of sequential activity in networks with temporally asymmetric Hebbian learning

Sequential activity is a prominent feature of many neural systems, in multiple behavioral contexts. Here, we investigate how Hebbian rules lead to storage and recall of random sequences of inputs in both rate and spiking recurrent networks. In the case of the simplest (bilinear) rule, we characterize extensively the regions in parameter space that allow sequence retrieval and compute analytically the storage capacity of the network. We show that nonlinearities in the learning rule can lead to sparse sequences and find that sequences maintain robust decoding but display highly labile dynamics to continuous changes in the connectivity matrix, similar to recent observations in hippocampus and parietal cortex.

Sequential activity has been observed in multiple neuronal circuits across species, neural structures, and behaviors. It has been hypothesized that sequences could arise from learning processes. However, it is still unclear whether biologically plausible synaptic plasticity rules can organize neuronal activity to form sequences whose statistics match experimental observations. Here, we investigate temporally asymmetric Hebbian rules in sparsely connected recurrent rate networks and develop a theory of the transient sequential activity observed after learning. These rules transform a sequence of random input patterns into synaptic weight updates. After learning, recalled sequential activity is reflected in the transient correlation of network activity with each of the stored input patterns. Using mean-field theory, we derive a low-dimensional description of the network dynamics and compute the storage capacity of these networks. Multiple temporal characteristics of the recalled sequential activity are consistent with experimental observations. We find that the degree of sparseness of the recalled sequences can be controlled by nonlinearities in the learning rule. Furthermore, sequences maintain robust decoding, but display highly labile dynamics, when synaptic connectivity is continuously modified due to noise or storage of other patterns, similar to recent observations in hippocampus and parietal cortex. Finally, we demonstrate that our results also hold in recurrent networks of spiking neurons with separate excitatory and inhibitory populations.

Pseudo-orthogonalization of memory patterns for associative memory

A new method for improving the storage capacity of associative memory models on a neural network is proposed. The storage capacity of the network increases in proportion to the network size in the case of random patterns, but, in general, the capacity suffers from correlation among memory patterns. Numerous solutions to this problem have been proposed so far, but their high computational cost limits their scalability. In this paper, we propose a novel and simple solution that is locally computable without any iteration. Our method involves XNOR masking of the original memory patterns with random patterns, and the masked patterns and masks are concatenated. The resulting decorrelated patterns allow higher storage capacity at the cost of the pattern length. Furthermore, the increase in the pattern length can be reduced through blockwise masking, which results in a small amount of capacity loss. Movie replay and image recognition are presented as examples to demonstrate the scalability of the proposed method.

Learning of spatiotemporal patterns in Ising-spin neural networks: analysis of storage capacity by path integral methods

We encode periodic spatiotemporal patterns in Ising-spin neural networks, using the simple learning rule inspired by the spike-timing-dependent synaptic plasticity. It is then found that periodically oscillating spin neurons successfully reproduce phase differences of the encoded periodic patterns. The storage capacity of this associative memory neural network is enhanced with an adequate level of asymmetry in synapse connections. To understand the properties of these nonequilibrium retrieval states of the neural network, we carry out an analysis based on a path integral method. The relation of a dynamic crosstalk term to time-persistent oscillation of a correlation function well explains the enhancement of the storage capacity in spite of our approximation on nonpersistent terms. We investigate the accuracy of this approximation further by detailed comparison with numerical simulations.

Rank abundance relations in evolutionary dynamics of random replicators

We present a nonequilibrium statistical mechanics description of rank abundance relations (RAR) in random community models of ecology. Specifically, we study a multispecies replicator system with quenched random interaction matrices. We here consider symmetric interactions as well as asymmetric and antisymmetric cases. RARs are obtained analytically via a generating functional analysis, describing fixed-point states of the system in terms of a small set of order parameters, and in dependence on the symmetry or otherwise of interactions and on the productivity of the community. Our work is an extension of Tokita [Phys. Rev. Lett. 93, 178102 (2004)], where the case of symmetric interactions was considered within an equilibrium setup. The species abundance distribution in our model come out as truncated normal distributions or transformations thereof and, in some case, are similar to left-skewed distributions observed in ecology. We also discuss the interaction structure of the resulting food-web of stable species at stationarity, cases of heterogeneous cooperation pressures as well as effects of finite system size and of higher-order interactions.

Transient dynamics for sequence-processing neural networks: effect of degree distributions

We derive an analytic evolution equation for overlap parameters, including the effect of degree distribution on the transient dynamics of sequence processing neural networks. In the special case of globally coupled networks, the precisely retrieved critical loading ratio alpha_{c}=N;{-12} is obtained, where N is the network size. In the presence of random networks, our theoretical predictions agree quantitatively with the numerical experiments for delta, binomial, and power-law degree distributions.

Period-two cycles in a feedforward layered neural network model with symmetric sequence processing

The effects of dominant sequential interactions are investigated in an exactly solvable feedforward layered neural network model of binary units and patterns near saturation in which the interaction consists of a Hebbian part and a symmetric sequential term. Phase diagrams of stationary states are obtained and a phase of cyclic correlated states of period two is found for a weak Hebbian term, independently of the number of condensed patterns c.

Memory Capacity for Sequences in a Recurrent Network with Biological Constraints

The CA3 region of the hippocampus is a recurrent neural network that is essential for the storage and replay of sequences of patterns that represent behavioral events. Here we present a theoretical framework to calculate a sparsely connected network's capacity to store such sequences. As in CA3, only a limited subset of neurons in the network is active at any one time, pattern retrieval is subject to error, and the resources for plasticity are limited. Our analysis combines an analytical mean field approach, stochastic dynamics, and cellular simulations of a time-discrete McCulloch-Pitts network with binary synapses. To maximize the number of sequences that can be stored in the network, we concurrently optimize the number of active neurons, that is, pattern size, and the firing threshold. We find that for one-step associations (i.e., minimal sequences), the optimal pattern size is inversely proportional to the mean connectivity c, whereas the optimal firing threshold is independent of the connectivity. If the number of synapses per neuron is fixed, the maximum number P of stored sequences in a sufficiently large, nonmodular network is independent of its number N of cells. On the other hand, if the number of synapses scales as the network size to the power of 3/2, the number of sequences P is proportional to N. In other words, sequential memory is scalable. Further-more, we find that there is an optimal ratio r between silent and nonsilent synapses at which the storage capacity α = P/[c (1 +r)N] assumes a maximum. For long sequences, the capacity of sequential memory is about one order of magnitude below the capacity for minimal sequences, but otherwise behaves similar to the case of minimal sequences. In a biologically inspired scenario, the information content per synapse is far below theoretical optimality, suggesting that the brain trades off error tolerance against information content in encoding sequential memories.

Pattern reconstruction and sequence processing in feed-forward layered neural networks near saturation

The dynamics and the stationary states for the competition between pattern reconstruction and asymmetric sequence processing are studied here in an exactly solvable feed-forward layered neural network model of binary units and patterns near saturation. Earlier work by Coolen and Sherrington on a parallel dynamics far from saturation is extended here to account for finite stochastic noise due to a Hebbian and a sequential learning rule. Phase diagrams are obtained with stationary states and quasiperiodic nonstationary solutions. The relevant dependence of these diagrams and of the quasiperiodic solutions on the stochastic noise and on initial inputs for the overlaps is explicitly discussed.

Bifurcation analysis in an associative memory model

We previously reported the chaos induced by the frustration of interaction in a nonmonotonic sequential associative memory model, and showed the chaotic behaviors at absolute zero. We have now analyzed bifurcation in a stochastic system, namely, a finite-temperature model of the nonmonotonic sequential associative memory model. We derived order-parameter equations from the stochastic microscopic equations. Two-parameter bifurcation diagrams obtained from those equations show the coexistence of attractors, which do not appear at absolute zero, and the disappearance of chaos due to the temperature effect.

Storage Capacity Diverges With Synaptic Efficiency in an Associative Memory Model With Synaptic Delay and Pruning

It is known that storage capacity per synapse increases by synaptic pruning in the case of a correlation-type associative memory model. However, the storage capacity of the entire network then decreases. To overcome this difficulty, we propose decreasing the connectivity while keeping the total number of synapses constant by introducing delayed synapses. In this paper, a discrete synchronous-type model with both delayed synapses and their prunings is discussed as a concrete example of the proposal. First, we explain the Yanai-Kim theory by employing statistical neurodynamics. This theory involves macrodynamical equations for the dynamics of a network with serial delay elements. Next, considering the translational symmetry of the explained equations, we rederive macroscopic steady-state equations of the model by using the discrete Fourier transformation. The storage capacities are analyzed quantitatively. Furthermore, two types of synaptic prunings are treated analytically: random pruning and systematic pruning. As a result, it becomes clear that in both prunings, the storage capacity increases as the length of delay increases and the connectivity of the synapses decreases when the total number of synapses is constant. Moreover, an interesting fact becomes clear: the storage capacity asymptotically approaches 2/pi due to random pruning. In contrast, the storage capacity diverges in proportion to the logarithm of the length of delay by systematic pruning and the proportion constant is 4/pi. These results theoretically support the significance of pruning following an overgrowth of synapses in the brain and may suggest that the brain prefers to store dynamic attractors such as sequences and limit cycles rather than equilibrium states.

Associative memory by recurrent neural networks with delay elements

The synapses of real neural systems seem to have delays. Therefore, it is worthwhile to analyze associative memory models with delayed synapses. Thus, a sequential associative memory model with delayed synapses is discussed, where a discrete synchronous updating rule and a correlation learning rule are employed. Its dynamic properties are analyzed by the statistical neurodynamics. In this paper, we first re-derive the Yanai-Kim theory, which involves macrodynamical equations for the dynamics of the network with serial delay elements. Since their theory needs a computational complexity of O(L4t) to obtain the macroscopic state at time step t where L is the length of delay, it is intractable to discuss the macroscopic properties for a large L limit. Thus, we derive steady state equations using the discrete Fourier transformation, where the computational complexity does not formally depend on L. We show that the storage capacity alphaC is in proportion to the delay length L with a large L limit, and the proportion constant is 0.195, i.e. alphaC=0.195L. These results are supported by computer simulations.