Topology and computational performance of attractor neural networks

Equilibrium Propagation for Memristor-Based Recurrent Neural Networks

Among the recent innovative technologies, memristor (memory-resistor) has attracted researchers attention as a fundamental computation element. It has been experimentally shown that memristive elements can emulate synaptic dynamics and are even capable of supporting spike timing dependent plasticity (STDP), an important adaptation rule that is gaining particular interest because of its simplicity and biological plausibility. The overall goal of this work is to provide a novel (theoretical) analog computing platform based on memristor devices and recurrent neural networks that exploits the memristor device physics to implement two variations of the backpropagation algorithm: recurrent backpropagation and equilibrium propagation. In the first learning technique, the use of memristor–based synaptic weights permits to propagate the error signals in the network by means of the nonlinear dynamics via an analog side network. This makes the processing non-digital and different from the current procedures. However, the necessity of a side analog network for the propagation of error derivatives makes this technique still highly biologically implausible. In order to solve this limitation, it is therefore proposed an alternative solution to the use of a side network by introducing a learning technique used for energy-based models: equilibrium propagation. Experimental results show that both approaches significantly outperform conventional architectures used for pattern reconstruction. Furthermore, due to the high suitability for VLSI implementation of the equilibrium propagation learning rule, additional results on the classification of the MNIST dataset are here reported.

Spatially Arranged Sparse Recurrent Neural Networks for Energy Efficient Associative Memory

The development of hardware neural networks, including neuromorphic hardware, has been accelerated over the past few years. However, it is challenging to operate very large-scale neural networks with low-power hardware devices, partly due to signal transmissions through a massive number of interconnections. Our aim is to deal with the issue of communication cost from an algorithmic viewpoint and study learning algorithms for energy-efficient information processing. Here, we consider two approaches to finding spatially arranged sparse recurrent neural networks with the high cost-performance ratio for associative memory. In the first approach following classical methods, we focus on sparse modular network structures inspired by biological brain networks and examine their storage capacity under an iterative learning rule. We show that incorporating long-range intermodule connections into purely modular networks can enhance the cost-performance ratio. In the second approach, we formulate for the first time an optimization problem where the network sparsity is maximized under the constraints imposed by a pattern embedding condition. We show that there is a tradeoff between the interconnection cost and the computational performance in the optimized networks. We demonstrate that the optimized networks can achieve a better cost-performance ratio compared with those considered in the first approach. We show the effectiveness of the optimization approach mainly using binary patterns and apply it also to gray-scale image restoration. Our results suggest that the presented approaches are useful in seeking more sparse and less costly connectivity of neural networks for the enhancement of energy efficiency in hardware neural networks.

Effect of similarity between patterns in associative memory

We study the stability of patterns in Hopfield networks in which a part of memorized patterns are similar. The similarity between patterns impacts the stability of these patterns, but the stability of other independent patterns is only changed slightly. We show that the stability of patterns is affected in different ways by similarity. For networks storing a number of patterns, the similarity between patterns enhances the pattern stability. However, the stability of patterns can be weakened by the similarity when networks store fewer patterns, and the relation between the stability of patterns and similarity is nonmonotonic. We present a theoretical explanation of the effect of similarity on stability using signal-to-noise-ratio analysis.

Nonbacktracking operator for the Ising model and its applications in systems with multiple states

The nonbacktracking operator for a graph is the adjacency matrix defined on directed edges of the graph. The operator was recently shown to perform optimally in spectral clustering in sparse synthetic graphs and have a deep connection to belief propagation algorithm. In this paper we consider nonbacktracking operator for Ising model on a general graph with a general coupling distribution and study the spectrum of this operator analytically. We show that spectral algorithms based on this operator is equivalent to belief propagation algorithm linearized at the paramagnetic fixed point and recovers replica-symmetry results on phase boundaries obtained by replica methods. This operator can be applied directly to systems with multiple states like Hopfield model. We show that spectrum of the operator can be used to determine number of patterns that stored successfully in the network, and the associated eigenvectors can be used to retrieve all the patterns simultaneously. We also give an example on how to control the Hopfield model, i.e., making network more sparse while keeping patterns stable, using the nonbacktracking operator and matrix perturbation theory.

Graph degree sequence solely determines the expected hopfield network pattern stability

We analyze the effect of network topology on the pattern stability of the Hopfield neural network in the case of general graphs. The patterns are randomly selected from a uniform distribution. We start the Hopfield procedure from some pattern v. An error in an entry e of v is the situation where, if the procedure is started at e, the value of e flips. Such an entry is an instability point. Note that we disregard the value at e by the end of the procedure, as well as what happens if we start the procedure from another pattern v' or another entry e' of v. We measure the instability of the system by the expected total number of instability points of all the patterns. Our main result is that the instability of the system does not depend on the exact topology of the underlying graph, but rather only on its degree sequence. Moreover, for a large number of nodes, the instability can be approximated by mΣni=1(1 − Φ(√δi/m−1)), where is the standard normal distribution function and δ1, . . . , δn are the degrees of the nodes.

Sequence memory based on coherent spin-interaction neural networks

Sequence information processing, for instance, the sequence memory, plays an important role on many functions of brain. In the workings of the human brain, the steady-state period is alterable. However, in the existing sequence memory models using heteroassociations, the steady-state period cannot be changed in the sequence recall. In this work, a novel neural network model for sequence memory with controllable steady-state period based on coherent spininteraction is proposed. In the proposed model, neurons fire collectively in a phase-coherent manner, which lets a neuron group respond differently to different patterns and also lets different neuron groups respond differently to one pattern. The simulation results demonstrating the performance of the sequence memory are presented. By introducing a new coherent spin-interaction sequence memory model, the steady-state period can be controlled by dimension parameters and the overlap between the input pattern and the stored patterns. The sequence storage capacity is enlarged by coherent spin interaction compared with the existing sequence memory models. Furthermore, the sequence storage capacity has an exponential relationship to the dimension of the neural network.

Hierarchical excitatory synaptic connectivity in mouse olfactory cortex

Topological motifs in synaptic connectivity-such as the cortical column-are fundamental to processing of information in cortical structures. However, the mesoscale topology of cortical networks beyond columns remains largely unknown. In the olfactory cortex, which lacks an obvious columnar structure, sensory-evoked patterns of activity have failed to reveal organizational principles of the network and its structure has been considered to be random. We probed the excitatory network in the mouse olfactory cortex using variance analysis of paired whole-cell recording in olfactory cortex slices. On a given trial, triggered network-wide bursts in disinhibited slices had remarkably similar time courses in widely separated and randomly selected cell pairs of pyramidal neurons despite significant trial-to-trial variability within each neuron. Simulated excitatory network models with random topologies only partially reproduced the experimental burst-variance patterns. Network models with local (columnar) or distributed subnetworks, which have been predicted as the basis of encoding odor objects, were also inconsistent with the experimental data, showing greater variability between cells than across trials. Rather, network models with power-law and especially hierarchical connectivity showed the best fit. Our results suggest that distributed subnetworks are weak or absent in the olfactory cortex, whereas a hierarchical excitatory topology may predominate. A hierarchical excitatory network organization likely underlies burst generation in this epileptogenic region, and may also shape processing of sensory information in the olfactory cortex.

Theoretical model for mesoscopic-level scale-free self-organization of functional brain networks

In this paper, we provide theoretical and numerical analysis of a geometric activity flow network model which is aimed at explaining mathematically the scale-free functional graph self-organization phenomena emerging in complex nervous systems at a mesoscale level. In our model, each unit corresponds to a large number of neurons and may be roughly seen as abstracting the functional behavior exhibited by a single voxel under functional magnetic resonance imaging (fMRI). In the course of the dynamics, the units exchange portions of formal charge, which correspond to waves of activity in the underlying microscale neuronal circuit. The geometric model abstracts away the neuronal complexity and is mathematically tractable, which allows us to establish explicit results on its ground states and the resulting charge transfer graph modeling functional graph of the network. We show that, for a wide choice of parameters and geometrical setups, our model yields a scale-free functional connectivity with the exponent approaching 2, which is in agreement with previous empirical studies based on fMRI. The level of universality of the presented theory allows us to claim that the model does shed light on mesoscale functional self-organization phenomena of the nervous system, even without resorting to closer details of brain connectivity geometry which often remain unknown. The material presented here significantly extends our previous work where a simplified mean-field model in a similar spirit was constructed, ignoring the underlying network geometry.

Maximum Memory Capacity on Neural Networks with Short-Term Synaptic Depression and Facilitation

In this work, we study, analytically and employing Monte Carlo simulations, the influence of the competition between several activity-dependent synaptic processes, such as short-term synaptic facilitation and depression, on the maximum memory storage capacity in a neural network. In contrast to the case of synaptic depression, which drastically reduces the capacity of the network to store and retrieve “static” activity patterns, synaptic facilitation enhances the storage capacity in different contexts. In particular, we found optimal values of the relevant synaptic parameters (such as the neurotransmitter release probability or the characteristic facilitation time constant) for which the storage capacity can be maximal and similar to the one obtained with static synapses, that is, without activity-dependent processes. We conclude that depressing synapses with a certain level of facilitation allow recovering the good retrieval properties of networks with static synapses while maintaining the nonlinear characteristics of dynamic synapses, convenient for information processing and coding.

Structured information in small-world neural networks

The retrieval abilities of spatially uniform attractor networks can be measured by the global overlap between patterns and neural states. However, we found that nonuniform networks, for instance, small-world networks, can retrieve fragments of patterns (blocks) without performing global retrieval. We propose a way to measure the local retrieval using a parameter that is related to the fluctuation of the block overlaps. Simulation of neural dynamics shows a competition between local and global retrieval. The phase diagram shows a transition from local retrieval to global retrieval when the storage ratio increases and the topology becomes more random. A theoretical approach confirms the simulation results and predicts that the stability of blocks can be improved by dilution.

Efficient connection strategies in 1D and 2D associative memory models with and without displaced connectivity

This study examines the performance of sparsely connected associative memory models built using a number of different connection strategies, applied to one- and two-dimensional topologies. Efficient patterns of connectivity are identified which yield high performance at relatively low wiring costs in both topologies. Networks with displaced connectivity are seen to perform particularly well. It is found that two-dimensional models are more tolerant of variations in connection strategy than their one-dimensional counterparts; though networks built with both topologies become less so as their connection density is decreased.

Transient dynamics for sequence-processing neural networks: effect of degree distributions

We derive an analytic evolution equation for overlap parameters, including the effect of degree distribution on the transient dynamics of sequence processing neural networks. In the special case of globally coupled networks, the precisely retrieved critical loading ratio alpha_{c}=N;{-12} is obtained, where N is the network size. In the presence of random networks, our theoretical predictions agree quantitatively with the numerical experiments for delta, binomial, and power-law degree distributions.

Storage capacity and retrieval time of small-world neural networks

To understand the influence of structure on the function of neural networks, we study the storage capacity and the retrieval time of Hopfield-type neural networks for four network structures: regular, small world, random networks generated by the Watts-Strogatz (WS) model, and the same network as the neural network of the nematode Caenorhabditis elegans. Using computer simulations, we find that (1) as the randomness of network is increased, its storage capacity is enhanced; (2) the retrieval time of WS networks does not depend on the network structure, but the retrieval time of C. elegans's neural network is longer than that of WS networks; (3) the storage capacity of the C. elegans network is smaller than that of networks generated by the WS model, though the neural network of C. elegans is considered to be a small-world network.

Imposing biological constraints onto an abstract neocortical attractor network model

In this letter, we study an abstract model of neocortex based on its modularization into mini- and hypercolumns. We discuss a full-scale instance of this model and connect its network properties to the underlying biological properties of neurons in cortex. In particular, we discuss how the biological constraints put on the network determine the network's performance in terms of storage capacity. We show that a network instantiating the model scales well given the biologically constrained parameters on activity and connectivity, which makes this network interesting also as an engineered system. In this model, the minicolumns are grouped into hypercolumns that can be active or quiescent, and the model predicts that only a few percent of the hypercolumns should be active at any one time. With this model, we show that at least 20 to 30 pyramidal neurons should be aggregated into a minicolumn and at least 50 to 60 minicolumns should be grouped into a hypercolumn in order to achieve high storage capacity.

Information and topology in attractor neural networks

A wide range of networks, including those with small-world topology, can be modeled by the connectivity ratio and randomness of the links. Both learning and attractor abilities of a neural network can be measured by the mutual information (MI) as a function of the load and the overlap between patterns and retrieval states. In this letter, we use MI to search for the optimal topology with regard to the storage and attractor properties of the network in an Amari-Hopfield model. We find that while an optimal storage implies an extremely diluted topology, a large basin of attraction leads to moderate levels of connectivity. This optimal topology is related to the clustering and path length of the network. We also build a diagram for the dynamical phases with random or local initial overlap and show that very diluted networks lose their attractor ability.

Autoassociative memory retrieval and spontaneous activity bumps in small-world networks of integrate-and-fire neurons

The metric structure of synaptic connections is obviously an important factor in shaping the properties of neural networks, in particular the capacity to retrieve memories, with which are endowed autoassociative nets operating via attractor dynamics. Qualitatively, some real networks in the brain could be characterized as 'small worlds', in the sense that the structure of their connections is intermediate between the extremes of an orderly geometric arrangement and of a geometry-independent random mesh. Small worlds can be defined more precisely in terms of their mean path length and clustering coefficient; but is such a precise description useful for a better understanding of how the type of connectivity affects memory retrieval? We have simulated an autoassociative memory network of integrate-and-fire units, positioned on a ring, with the network connectivity varied parametrically between ordered and random. We find that the network retrieves previously stored memory patterns when the connectivity is close to random, and displays the characteristic behavior of ordered nets (localized 'bumps' of activity) when the connectivity is close to ordered. Recent analytical work shows that these two behaviors can coexist in a network of simple threshold-linear units, leading to localized retrieval states. We find that they tend to be mutually exclusive behaviors, however, with our integrate-and-fire units. Moreover, the transition between the two occurs for values of the connectivity parameter which are not simply related to the notion of small worlds.

Statistical-mechanical iterative algorithms on complex networks

The Ising models have been applied for various problems on information sciences, social sciences, and so on. In many cases, solving these problems corresponds to minimizing the Bethe free energy. To minimize the Bethe free energy, a statistical-mechanical iterative algorithm is often used. We study the statistical-mechanical iterative algorithm on complex networks. To investigate effects of heterogeneous structures on the iterative algorithm, we introduce an iterative algorithm based on information of heterogeneity of complex networks, in which higher-degree nodes are likely to be updated more frequently than lower-degree ones. Numerical experiments clarified that the usage of the information of heterogeneity affects the algorithm in Barabási and Albert networks, but does not influence that in Erdös and Rényi networks. It is revealed that information of the whole system propagates rapidly through such high-degree nodes in the case of Barabási-Albert's scale-free networks.

Effects of static and dynamic disorder on the performance of neural automata

We report on both analytical and numerical results concerning stochastic Hopfield-like neural automata exhibiting the following (biologically inspired) features: (1) Neurons and synapses evolve in time as in contact with respective baths at different temperatures; (2) the connectivity between neurons may be tuned from full connection to high random dilution, or to the case of networks with the small-world property and/or scale-free architecture; and (3) there is synaptic kinetics simulating repeated scanning of the stored patterns. Although these features may apparently result in additional disorder, the model exhibits, for a wide range of parameter values, an extraordinary computational performance, and some of the qualitative behaviors observed in natural systems. In particular, we illustrate here very efficient and robust associative memory, and jumping between pattern attractors.

Slow relaxation in the Ising model on a small-world network with strong long-range interactions

We consider the Ising model on a small-world network, where the long-range interaction strength J2 is in general different from the local interaction strength J1, and examine its relaxation behaviors as well as phase transitions. As J(2)/J(1) is raised from zero, the critical temperature also increases, manifesting contributions of long-range interactions to ordering. However, it becomes saturated eventually at large values of J(2)/J(1) and the system is found to display very slow relaxation, revealing that ordering dynamics is inhibited rather than facilitated by strong long-range interactions. To circumvent this problem, we propose a modified updating algorithm in Monte Carlo simulations, assisting the system to reach equilibrium quickly.

Performance of networks of artificial neurons: the role of clustering

The performance of the Hopfield neural network model is numerically studied on various complex networks, such as the Watts-Strogatz network, the Barabási-Albert network, and the neuronal network of Caenorhabditis elegans. Through the use of a systematic way of controlling the clustering coefficient, with the degree of each neuron kept unchanged, we find that the networks with the lower clustering exhibit much better performance. The results are discussed in the practical viewpoint of application, and the biological implications are also suggested.