Fixed-point attractors in analog neural computation

Spintronic virtual neural network by a voltage controlled ferromagnet for associative memory

Recently, an associative memory operation by a virtual oscillator network, consisting of a single spintronic oscillator, was examined to solve issues in conventional, real oscillators-based neural networks such as inhomogeneities between the oscillators. However, the spintronic oscillator still carries issues dissipating large amount of energy because it is driven by electric current. Here, we propose to use a single ferromagnet manipulated by voltage-controlled magnetic anisotropy (VCMA) effect as a fundamental element in a virtual neural network, which will contribute to significantly reducing the Joule heating caused by electric current. Instead of the oscillation in oscillator networks, magnetization relaxation dynamics were used for the associative memory operation. The associative memory operation for alphabet patterns is successfully demonstrated by giving correspondences between the colors in a pattern recognition task and the sign of a perpendicular magnetic anisotropy coefficient, which could be either positive or negative via the VCMA effect.

Associative memory by virtual oscillator network based on single spin-torque oscillator

A coupled oscillator network may be able to perform an energy-efficient associative memory operation. However, its realization has been difficult because inhomogeneities unavoidably arise among the oscillators during fabrication and lead to an unreliable operation. This issue could be resolved if the oscillator network were able to be formed from a single oscillator. Here, we performed numerical simulations and theoretical analyses on an associative memory operation that uses a virtual oscillator network based on a spin-torque oscillator. The virtual network combines the concept of coupled oscillators with that of feedforward neural networks. Numerical experiments demonstrate successful associations of 60-pixel patterns with various memorized patterns. Moreover, the origin of the associative memory is shown to be forced synchronization driven by feedforward input, where phase differences among oscillators are fixed and correspond to the colors of the pixels in the pattern.

Metastable dynamics of neural circuits and networks

Cortical neurons emit seemingly erratic trains of action potentials or "spikes," and neural network dynamics emerge from the coordinated spiking activity within neural circuits. These rich dynamics manifest themselves in a variety of patterns, which emerge spontaneously or in response to incoming activity produced by sensory inputs. In this Review, we focus on neural dynamics that is best understood as a sequence of repeated activations of a number of discrete hidden states. These transiently occupied states are termed "metastable" and have been linked to important sensory and cognitive functions. In the rodent gustatory cortex, for instance, metastable dynamics have been associated with stimulus coding, with states of expectation, and with decision making. In frontal, parietal, and motor areas of macaques, metastable activity has been related to behavioral performance, choice behavior, task difficulty, and attention. In this article, we review the experimental evidence for neural metastable dynamics together with theoretical approaches to the study of metastable activity in neural circuits. These approaches include (i) a theoretical framework based on non-equilibrium statistical physics for network dynamics; (ii) statistical approaches to extract information about metastable states from a variety of neural signals; and (iii) recent neural network approaches, informed by experimental results, to model the emergence of metastable dynamics. By discussing these topics, we aim to provide a cohesive view of how transitions between different states of activity may provide the neural underpinnings for essential functions such as perception, memory, expectation, or decision making, and more generally, how the study of metastable neural activity may advance our understanding of neural circuit function in health and disease.

Destabilization of Local Minima in Analog Spin Systems by Correction of Amplitude Heterogeneity

The relaxation of binary spins to analog values has been the subject of much debate in the field of statistical physics, neural networks, and more recently quantum computing, notably because the benefits of using an analog state for finding lower energy spin configurations are usually offset by the negative impact of the improper mapping of the energy function that results from the relaxation. We show that it is possible to destabilize trapping sets of analog states that correspond to local minima of the binary spin Hamiltonian by extending the phase space to include error signals that correct amplitude inhomogeneity of the analog spin states and controlling the divergence of their velocity. Performance of the proposed analog spin system in finding lower energy states is competitive against state-of-the-art heuristics.

Structural model for fluctuations in financial markets

In this paper we provide a comprehensive analysis of a structural model for the dynamics of prices of assets traded in a market which takes the form of an interacting generalization of the geometric Brownian motion model. It is formally equivalent to a model describing the stochastic dynamics of a system of analog neurons, which is expected to exhibit glassy properties and thus many metastable states in a large portion of its parameter space. We perform a generating functional analysis, introducing a slow driving of the dynamics to mimic the effect of slowly varying macroeconomic conditions. Distributions of asset returns over various time separations are evaluated analytically and are found to be fat-tailed in a manner broadly in line with empirical observations. Our model also allows us to identify collective, interaction-mediated properties of pricing distributions and it predicts pricing distributions which are significantly broader than their noninteracting counterparts, if interactions between prices in the model contain a ferromagnetic bias. Using simulations, we are able to substantiate one of the main hypotheses underlying the original modeling, viz., that the phenomenon of volatility clustering can be rationalized in terms of an interplay between the dynamics within metastable states and the dynamics of occasional transitions between them.

Memory Dynamics in Attractor Networks with Saliency Weights

Memory is a fundamental part of computational systems like the human brain. Theoretical models identify memories as attractors of neural network activity patterns based on the theory that attractor (recurrent) neural networks are able to capture some crucial characteristics of memory, such as encoding, storage, retrieval, and long-term and working memory. In such networks, long-term storage of the memory patterns is enabled by synaptic strengths that are adjusted according to some activity-dependent plasticity mechanisms (of which the most widely recognized is the Hebbian rule) such that the attractors of the network dynamics represent the stored memories. Most of previous studies on associative memory are focused on Hopfield-like binary networks, and the learned patterns are often assumed to be uncorrelated in a way that minimal interactions between memories are facilitated. In this letter, we restrict our attention to a more biological plausible attractor network model and study the neuronal representations of correlated patterns. We have examined the role of saliency weights in memory dynamics. Our results demonstrate that the retrieval process of the memorized patterns is characterized by the saliency distribution, which affects the landscape of the attractors. We have established the conditions that the network state converges to unique memory and multiple memories. The analytical result also holds for other cases for variable coding levels and nonbinary levels, indicating a general property emerging from correlated memories. Our results confirmed the advantage of computing with graded-response neurons over binary neurons (i.e., reducing of spurious states). It was also found that the nonuniform saliency distribution can contribute to disappearance of spurious states when they exit.

Correlation between eigenvalue spectra and dynamics of neural networks

This letter presents a study of the correlation between the eigenvalue spectra of synaptic matrices and the dynamical properties of asymmetric neural networks with associative memories. For this type of neural network, it was found that there are essentially two different dynamical phases: the chaos phase, with almost all trajectories converging to a single chaotic attractor, and the memory phase, with almost all trajectories being attracted toward fixed-point attractors acting as memories. We found that if a neural network is designed in the chaos phase, the eigenvalue spectrum of its synaptic matrix behaves like that of a random matrix (i.e., all eigenvalues lie uniformly distributed within a circle in the complex plan), and if it is designed in the memory phase, the eigenvalue spectrum will split into two parts: one part corresponds to a random background, the other part equal in number to the memory attractors. The mechanism for these phenomena is discussed in this letter.

Analysis and Design of Associative Memories Based on Recurrent Neural Networks with Linear Saturation Activation Functions and Time-Varying Delays

In this letter, some sufficient conditions are obtained to guarantee recurrent neural networks with linear saturation activation functions, and time-varying delays have multiequilibria located in the saturation region and the boundaries of the saturation region. These results on pattern characterization are used to analyze and design autoassociative memories, which are directly based on the parameters of the neural networks. Moreover, a formula for the numbers of spurious equilibria is also derived. Four design procedures for recurrent neural networks with linear saturation activation functions and time-varying delays are developed based on stability results. Two of these procedures allow the neural network to be capable of learning and forgetting. Finally, simulation results demonstrate the validity and characteristics of the proposed approach.

Generation and reshaping of sequences in neural systems

The generation of informational sequences and their reorganization or reshaping is one of the most intriguing subjects for both neuroscience and the theory of autonomous intelligent systems. In spite of the diversity of sequential activities of sensory, motor, and cognitive neural systems, they have many similarities from the dynamical point of view. In this review we discus the ideas, models, and mathematical image of sequence generation and reshaping on different levels of the neural hierarchy, i.e., the role of a sensory network dynamics in the generation of a motor program (hunting swimming of marine mollusk Clione), olfactory dynamical coding, and sequential learning and decision making. Analysis of these phenomena is based on the winnerless competition principle. The considered models can be a basis for the design of biologically inspired autonomous intelligent systems.

Pattern recognition using asymmetric attractor neural networks

The asymmetric attractor neural networks designed by the Monte Carlo- (MC-) adaptation rule are shown to be promising candidates for pattern recognition. In such a neural network with relatively low symmetry, when the members of a set of template patterns are stored as fixed-point attractors, their attraction basins are shown to be isolated islands embedded in a "chaotic sea." The sizes of these islands can be controlled by a single parameter. We show that these properties can be used for effective pattern recognition and rejection. In our method, the pattern to be identified is attracted to a template pattern or a chaotic attractor. If the difference between the pattern to be identified and the template pattern is smaller than a predescribed threshold, the pattern is attracted to the template pattern automatically and thus is identified as belonging to this template pattern. Otherwise, it wanders in a chaotic attractor for ever and thus is rejected as an unknown pattern. The maximum sizes of these islands allowed by this kind of neural networks are determined by a modified MC-adaptation rule which are shown to be able to dramatically enlarge the sizes of the islands. We illustrate the use of our method for pattern recognition and rejection with an example of recognizing a set of Chinese characters.

Designing asymmetric neural networks with associative memory

A strategy for designing asymmetric neural networks of associative memory with controllable degree of symmetry and controllable basins of attraction is presented. It is shown that the performance of the networks depends on the degree of the symmetry, and by adjusting the degree of the symmetry the spurious memories or unwanted attractors can be suppressed completely.

Disappearance of spurious states in analog associative memories

We show that symmetric n-mixture states, when they exist, are almost never stable in autoassociative networks with threshold-linear units. Only with a binary coding scheme, we could find a limited region of the parameter space in which either 2-mixture or 3-mixture states are stable attractors of the dynamics.

Spike-timing-dependent learning rule to encode spatiotemporal patterns in a network of spiking neurons

We study associative memory neural networks based on the Hodgkin-Huxley type of spiking neurons. We introduce the spike-timing-dependent learning rule, in which the time window with the negative part as well as the positive part is used to describe the biologically plausible synaptic plasticity. The learning rule is applied to encode a number of periodical spatiotemporal patterns, which are successfully reproduced in the periodical firing pattern of spiking neurons in the process of memory retrieval. The global inhibition is incorporated into the model so as to induce the gamma oscillation. The occurrence of gamma oscillation turns out to give appropriate spike timings for memory retrieval of discrete type of spatiotemporal pattern. The theoretical analysis to elucidate the stationary properties of perfect retrieval state is conducted in the limit of an infinite number of neurons and shows the good agreement with the result of numerical simulations. The result of this analysis indicates that the presence of the negative and positive parts in the form of the time window contributes to reduce the size of crosstalk term, implying that the time window with the negative and positive parts is suitable to encode a number of spatiotemporal patterns. We draw some phase diagrams, in which we find various types of phase transitions with change of the intensity of global inhibition.

Associative memory storing an extensive number of patterns based on a network of oscillators with distributed natural frequencies in the presence of external white noise

We study associative memory based on temporal coding in which successful retrieval is realized as an entrainment in a network of simple phase oscillators with distributed natural frequencies under the influence of white noise. The memory patterns are assumed to be given by uniformly distributed random numbers on [0, 2 pi) so that the patterns encode the phase differences of the oscillators. To derive the macroscopic order parameter equations for the network with an extensive number of stored patterns, we introduce an effective transfer function by assuming a fixed-point equation of the form of the Thouless-Anderson-Palmer equation, which describes the time-averaged output as a function of the effective time-averaged local field. Properties of the networks associated with synchronization phenomena for a discrete symmetric natural frequency distribution with three frequency components are studied based on the order parameter equations, and are shown to be in good agreement with the results of numerical simulations. Two types of retrieval states are found to occur with respect to the degree of synchronization, when the size of the width of the natural frequency distribution is changed.

Retrieval properties of a Hopfield model with random asymmetric interactions

The process of pattern retrieval in a Hopfield model in which a random antisymmetric component is added to the otherwise symmetric synaptic matrix is studied by computer simulations. The introduction of the anti-symmetric component is found to increase the fraction of random inputs that converge to the memory states. However, the size of the basin of attraction of a memory state does not show any significant change when asymmetry is introduced in the synaptic matrix. We show that this is due to the fact that the spurious fixed points, which are destabilized by the introduction of asymmetry, have very small basins of attraction. The convergence time to spurious fixed-point attractors increases faster than that for the memory states as the asymmetry parameter is increased. The possibility of convergence to spurious fixed points is greatly reduced if a suitable upper limit is set for the convergence time. This prescription works better if the synaptic matrix has an antisymmetric component.

Notions of Associative Memory and Sparse Coding

This paper summarizes associative memory models and sparse representation of memory in these models. Important properties of the associative memory models are their storage capacity, basin of attraction, and the existence of spurious memories. Sparse coding and nonmonotonic output functions are proposed to improve them. Sparsely coded associative memory model has an extremely large storage capacity which diverges as the mean firing rate of memory patterns approaches 0. The storage capacity strongly depends on the shape of the output function as well as the mean firing rate, even in the case of monotonic output functions. Dynamical properties of the model are analyzed by means of a statistical neurodynamical method. We emphasize the necessity of a feedback mechanism to control the mean firing rate in the recall process. Recently, there have been some experimental results suggesting its existence in the brain. On the other hand, it has been shown that the storage capacity can be markedly improved by replacing the usual monotonic output function with a nonmonotonic one. Another remarkable property of the model with the nonmonotonic neurons is that it seems to have no, or almost no, spurious memory. An associative memory model using nonmonotonic modules with a feedforward inhibition is discussed. The modules consist of two types of threshold units, each of which has a different threshold and can be considered as a biologically plausible representation of the nonmonotonic output function. The above model is compared with the monotonic one. The difference in the storage capacity between the two models becomes small when the sparse patterns are stored. Finally, we discuss the biological plausibility of the discussed associative memory models and sparse coding. Copyright 1996 Elsevier Science Ltd.