Attractors in fully asymmetric neural networks

Multiperiodic orbits from interacting soft spots in cyclically sheared amorphous solids

When an amorphous solid is deformed cyclically, it may reach a steady state in which the paths of constituent particles trace out closed loops that repeat in each driving cycle. A remarkable variant has been noticed in simulations where the period of particle motions is a multiple of the period of driving, but the reasons for this behavior have remained unclear. Motivated by mesoscopic features of displacement fields in experiments on jammed solids, we propose and analyze a simple model of interacting soft spots-locations where particles rearrange under stress and that resemble two-level systems with hysteresis. We show that multiperiodic behavior can arise among just three or more soft spots that interact with each other, but in all cases it requires frustrated interactions, illuminating this otherwise elusive type of interaction. We suggest directions for seeking this signature of frustration in experiments and for achieving it in designed systems.

On the Maximum Storage Capacity of the Hopfield Model

Recurrent neural networks (RNN) have traditionally been of great interest for their capacity to store memories. In past years, several works have been devoted to determine the maximum storage capacity of RNN, especially for the case of the Hopfield network, the most popular kind of RNN. Analyzing the thermodynamic limit of the statistical properties of the Hamiltonian corresponding to the Hopfield neural network, it has been shown in the literature that the retrieval errors diverge when the number of stored memory patterns (P) exceeds a fraction (≈ 14%) of the network size N. In this paper, we study the storage performance of a generalized Hopfield model, where the diagonal elements of the connection matrix are allowed to be different from zero. We investigate this model at finite N. We give an analytical expression for the number of retrieval errors and show that, by increasing the number of stored patterns over a certain threshold, the errors start to decrease and reach values below unit for P ≫ N. We demonstrate that the strongest trade-off between efficiency and effectiveness relies on the number of patterns (P) that are stored in the network by appropriately fixing the connection weights. When P≫N and the diagonal elements of the adjacency matrix are not forced to be zero, the optimal storage capacity is obtained with a number of stored memories much larger than previously reported. This theory paves the way to the design of RNN with high storage capacity and able to retrieve the desired pattern without distortions.

Structure of attractors in randomly connected networks

The deterministic dynamics of randomly connected neural networks are studied, where a state of binary neurons evolves according to a discrete-time synchronous update rule. We give theoretical support that the overlap of systems' states between the current and a previous time develops in time according to a Markovian stochastic process in large networks. This Markovian process predicts how often a network revisits one of the previously visited states, depending on the system size. The state concentration probability, i.e., the probability that two distinct states coevolve to the same state, is utilized to analytically derive various characteristics that quantify attractors' structure. The analytical predictions about the total number of attractors, the typical cycle length, and the number of states belonging to all attractive cycles match well with numerical simulations for relatively large system sizes.

Very long transients, irregular firing, and chaotic dynamics in networks of randomly connected inhibitory integrate-and-fire neurons

We present results of an extensive numerical study of the dynamics of networks of integrate-and-fire neurons connected randomly through inhibitory interactions. We first consider delayed interactions with infinitely fast rise and decay. Depending on the parameters, the network displays transients which are short or exponentially long in the network size. At the end of these transients, the dynamics settle on a periodic attractor. If the number of connections per neuron is large ( approximately 1000) , this attractor is a cluster state with a short period. In contrast, if the number of connections per neuron is small ( approximately 100) , the attractor has complex dynamics and very long period. During the long transients the neurons fire in a highly irregular manner. They can be viewed as quasistationary states in which, depending on the coupling strength, the pattern of activity is asynchronous or displays population oscillations. In the first case, the average firing rates and the variability of the single-neuron activity are well described by a mean-field theory valid in the thermodynamic limit. Bifurcations of the long transient dynamics from asynchronous to synchronous activity are also well predicted by this theory. The transient dynamics display features reminiscent of stable chaos. In particular, despite being linearly stable, the trajectories of the transient dynamics are destabilized by finite perturbations as small as O(1/N) . We further show that stable chaos is also observed for postsynaptic currents with finite decay time. However, we report in this type of network that chaotic dynamics characterized by positive Lyapunov exponents can also be observed. We show in fact that chaos occurs when the decay time of the synaptic currents is long compared to the synaptic delay, provided that the network is sufficiently large.

Threshold disorder as a source of diverse and complex behavior in random nets

We study the diversity of complex spatio-temporal patterns in the behavior of random synchronous asymmetric neural networks (RSANNs). Special attention is given to the impact of disordered threshold values on limit-cycle diversity and limit-cycle complexity in RSANNs which have 'normal' thresholds by default. Surprisingly, RSANNs exhibit only a small repertoire of rather complex limit-cycle patterns when all parameters are fixed. This repertoire of complex patterns is also rather stable with respect to small parameter changes. These two unexpected results may generalize to the study of other complex systems. In order to reach beyond this seemingly disabling 'stable and small' aspect of the limit-cycle repertoire of RSANNs, we have found that if an RSANN has threshold disorder above a critical level, then there is a rapid increase of the size of the repertoire of patterns. The repertoire size initially follows a power-law function of the magnitude of the threshold disorder. As the disorder increases further, the limit-cycle patterns themselves become simpler until at a second critical level most of the limit cycles become simple fixed points. Nonetheless, for moderate changes in the threshold parameters, RSANNs are found to display specific features of behavior desired for rapidly responding processing systems: accessibility to a large set of complex patterns.