Nonlinear neural networks: Principles, mechanisms, and architectures

Neural Mechanisms of Cortical Motion Computation Based on a Neuromorphic Sensory System

The visual cortex analyzes motion information along hierarchically arranged visual areas that interact through bidirectional interconnections. This work suggests a bio-inspired visual model focusing on the interactions of the cortical areas in which a new mechanism of feedforward and feedback processing are introduced. The model uses a neuromorphic vision sensor (silicon retina) that simulates the spike-generation functionality of the biological retina. Our model takes into account two main model visual areas, namely V1 and MT, with different feature selectivities. The initial motion is estimated in model area V1 using spatiotemporal filters to locally detect the direction of motion. Here, we adapt the filtering scheme originally suggested by Adelson and Bergen to make it consistent with the spike representation of the DVS. The responses of area V1 are weighted and pooled by area MT cells which are selective to different velocities, i.e. direction and speed. Such feature selectivity is here derived from compositions of activities in the spatio-temporal domain and integrating over larger space-time regions (receptive fields). In order to account for the bidirectional coupling of cortical areas we match properties of the feature selectivity in both areas for feedback processing. For such linkage we integrate the responses over different speeds along a particular preferred direction. Normalization of activities is carried out over the spatial as well as the feature domains to balance the activities of individual neurons in model areas V1 and MT. Our model was tested using different stimuli that moved in different directions. The results reveal that the error margin between the estimated motion and synthetic ground truth is decreased in area MT comparing with the initial estimation of area V1. In addition, the modulated V1 cell activations shows an enhancement of the initial motion estimation that is steered by feedback signals from MT cells.

The Use of Hebbian Cell Assemblies for Nonlinear Computation

When learning a complex task our nervous system self-organizes large groups of neurons into coherent dynamic activity patterns. During this, a network with multiple, simultaneously active, and computationally powerful cell assemblies is created. How such ordered structures are formed while preserving a rich diversity of neural dynamics needed for computation is still unknown. Here we show that the combination of synaptic plasticity with the slower process of synaptic scaling achieves (i) the formation of cell assemblies and (ii) enhances the diversity of neural dynamics facilitating the learning of complex calculations. Due to synaptic scaling the dynamics of different cell assemblies do not interfere with each other. As a consequence, this type of self-organization allows executing a difficult, six degrees of freedom, manipulation task with a robot where assemblies need to learn computing complex non-linear transforms and – for execution – must cooperate with each other without interference. This mechanism, thus, permits the self-organization of computationally powerful sub-structures in dynamic networks for behavior control.

A Neurocomputational account of the role of contour facilitation in brightness perception

A new filling-in model is proposed in order to account for challenging brightness illusions, where inducing background elements are spatially separated from the gray target such as dungeon, cube and grating illusions, bullseye display and ring patterns. This model implements the simple idea that neural response to low-contrast contour is enhanced (facilitated) by the presence of collinear or parallel high-contrast contours in its wider neighborhood. Contour facilitation is achieved via dendritic inhibition, which enables the computation of maximum function among inputs to the node. Recurrent application of maximum function leads to the propagation of the neural signal along collinear or parallel contour segments. When a strong global-contour signal is accompanied with a weak local-contour signal at the same location, conditions are met to produce brightness assimilation within the Filling-in Layer. Computer simulations showed that the model correctly predicts brightness appearance in all of the aforementioned illusions as well as in White's effect, Benary's cross, Todorović's illusion, checkerboard contrast, contrast-contrast illusion and various variations of the White's effect. The proposed model offers new insights on how geometric factors (contour colinearity or parallelism), together with contrast magnitude contribute to the brightness perception.

Short-term plasticity as cause-effect hypothesis testing in distal reward learning

Asynchrony, overlaps, and delays in sensory-motor signals introduce ambiguity as to which stimuli, actions, and rewards are causally related. Only the repetition of reward episodes helps distinguish true cause-effect relationships from coincidental occurrences. In the model proposed here, a novel plasticity rule employs short- and long-term changes to evaluate hypotheses on cause-effect relationships. Transient weights represent hypotheses that are consolidated in long-term memory only when they consistently predict or cause future rewards. The main objective of the model is to preserve existing network topologies when learning with ambiguous information flows. Learning is also improved by biasing the exploration of the stimulus-response space toward actions that in the past occurred before rewards. The model indicates under which conditions beliefs can be consolidated in long-term memory, it suggests a solution to the plasticity-stability dilemma, and proposes an interpretation of the role of short-term plasticity.

Multi-AUV Hunting Algorithm Based on Bio-inspired Neural Network in Unknown Environments

The multi-AUV hunting problem is one of the key issues in multi-robot system research. In order to hunt the target efficiently a new hunting algorithm based on a bio-inspired neural network has been proposed in this paper. Firstly, the AUV's working environment can be represented, based on the biological-inspired neural network model. There is one-to-one correspondence between each neuron in the neural network and the position of the grid map in the underwater environment. The activity values of biological neurons then guide the AUV's sailing path and finally the target is surrounded by AUVs. In addition, a method called negotiation is used to solve the AUV's allocation of hunting points. The simulation results show that the algorithm used in the paper can provide rapid and highly efficient path planning in the unknown environment with obstacles and non-obstacles.

Adaptive learning in a compartmental model of visual cortex-how feedback enables stable category learning and refinement

The categorization of real world objects is often reflected in the similarity of their visual appearances. Such categories of objects do not necessarily form disjunct sets of objects, neither semantically nor visually. The relationship between categories can often be described in terms of a hierarchical structure. For instance, tigers and leopards build two separate mammalian categories, both of which are subcategories of the category Felidae. In the last decades, the unsupervised learning of categories of visual input stimuli has been addressed by numerous approaches in machine learning as well as in computational neuroscience. However, the question of what kind of mechanisms might be involved in the process of subcategory learning, or category refinement, remains a topic of active investigation. We propose a recurrent computational network architecture for the unsupervised learning of categorial and subcategorial visual input representations. During learning, the connection strengths of bottom-up weights from input to higher-level category representations are adapted according to the input activity distribution. In a similar manner, top-down weights learn to encode the characteristics of a specific stimulus category. Feedforward and feedback learning in combination realize an associative memory mechanism, enabling the selective top-down propagation of a category's feedback weight distribution. We suggest that the difference between the expected input encoded in the projective field of a category node and the current input pattern controls the amplification of feedforward-driven representations. Large enough differences trigger the recruitment of new representational resources and the establishment of additional (sub-) category representations. We demonstrate the temporal evolution of such learning and show how the proposed combination of an associative memory with a modulatory feedback integration successfully establishes category and subcategory representations.

Computing with a canonical neural circuits model with pool normalization and modulating feedback

Evidence suggests that the brain uses an operational set of canonical computations like normalization, input filtering, and response gain enhancement via reentrant feedback. Here, we propose a three-stage columnar architecture of cascaded model neurons to describe a core circuit combining signal pathways of feedforward and feedback processing and the inhibitory pooling of neurons to normalize the activity. We present an analytical investigation of such a circuit by first reducing its detail through the lumping of initial feedforward response filtering and reentrant modulating signal amplification. The resulting excitatory-inhibitory pair of neurons is analyzed in a 2D phase-space. The inhibitory pool activation is treated as a separate mechanism exhibiting different effects. We analyze subtractive as well as divisive (shunting) interaction to implement center-surround mechanisms that include normalization effects in the characteristics of real neurons. Different variants of a core model architecture are derived and analyzed--in particular, individual excitatory neurons (without pool inhibition), the interaction with an inhibitory subtractive or divisive (i.e., shunting) pool, and the dynamics of recurrent self-excitation combined with divisive inhibition. The stability and existence properties of these model instances are characterized, which serve as guidelines to adjust these properties through proper model parameterization. The significance of the derived results is demonstrated by theoretical predictions of response behaviors in the case of multiple interacting hypercolumns in a single and in multiple feature dimensions. In numerical simulations, we confirm these predictions and provide some explanations for different neural computational properties. Among those, we consider orientation contrast-dependent response behavior, different forms of attentional modulation, contrast element grouping, and the dynamic adaptation of the silent surround in extraclassical receptive field configurations, using only slight variations of the same core reference model.

Where's Waldo? How perceptual, cognitive, and emotional brain processes cooperate during learning to categorize and find desired objects in a cluttered scene

The Where's Waldo problem concerns how individuals can rapidly learn to search a scene to detect, attend, recognize, and look at a valued target object in it. This article develops the ARTSCAN Search neural model to clarify how brain mechanisms across the What and Where cortical streams are coordinated to solve the Where's Waldo problem. The What stream learns positionally-invariant object representations, whereas the Where stream controls positionally-selective spatial and action representations. The model overcomes deficiencies of these computationally complementary properties through What and Where stream interactions. Where stream processes of spatial attention and predictive eye movement control modulate What stream processes whereby multiple view- and positionally-specific object categories are learned and associatively linked to view- and positionally-invariant object categories through bottom-up and attentive top-down interactions. Gain fields control the coordinate transformations that enable spatial attention and predictive eye movements to carry out this role. What stream cognitive-emotional learning processes enable the focusing of motivated attention upon the invariant object categories of desired objects. What stream cognitive names or motivational drives can prime a view- and positionally-invariant object category of a desired target object. A volitional signal can convert these primes into top-down activations that can, in turn, prime What stream view- and positionally-specific categories. When it also receives bottom-up activation from a target, such a positionally-specific category can cause an attentional shift in the Where stream to the positional representation of the target, and an eye movement can then be elicited to foveate it. These processes describe interactions among brain regions that include visual cortex, parietal cortex, inferotemporal cortex, prefrontal cortex (PFC), amygdala, basal ganglia (BG), and superior colliculus (SC).

Optimality and some of its discontents: successes and shortcomings of existing models for binary decisions

We review how leaky competing accumulators (LCAs) can be used to model decision making in two-alternative, forced-choice tasks, and we show how they reduce to drift diffusion (DD) processes in special cases. As continuum limits of the sequential probability ratio test, DD processes are optimal in producing decisions of specified accuracy in the shortest possible time. Furthermore, the DD model can be used to derive a speed-accuracy trade-off that optimizes reward rate for a restricted class of two alternative forced-choice decision tasks. We review findings that compare human performance with this benchmark, and we reveal both approximations to and deviations from optimality. We then discuss three potential sources of deviations from optimality at the psychological level--avoidance of errors, poor time estimation, and minimization of the cost of control--and review recent theoretical and empirical findings that address these possibilities. We also discuss the role of cognitive control in changing environments and in modulating exploitation and exploration. Finally, we consider physiological factors in which nonlinear dynamics may also contribute to deviations from optimality.

The mismeasure of machine: Synthetic biology and the trouble with engineering metaphors

The scientific study of living organisms is permeated by machine and design metaphors. Genes are thought of as the "blueprint" of an organism, organisms are "reverse engineered" to discover their functionality, and living cells are compared to biochemical factories, complete with assembly lines, transport systems, messenger circuits, etc. Although the notion of design is indispensable to think about adaptations, and engineering analogies have considerable heuristic value (e.g., optimality assumptions), we argue they are limited in several important respects. In particular, the analogy with human-made machines falters when we move down to the level of molecular biology and genetics. Living organisms are far more messy and less transparent than human-made machines. Notoriously, evolution is an opportunistic tinkerer, blindly stumbling on "designs" that no sensible engineer would come up with. Despite impressive technological innovation, the prospect of artificially designing new life forms from scratch has proven more difficult than the superficial analogy with "programming" the right "software" would suggest. The idea of applying straightforward engineering approaches to living systems and their genomes-isolating functional components, designing new parts from scratch, recombining and assembling them into novel life forms-pushes the analogy with human artifacts beyond its limits. In the absence of a one-to-one correspondence between genotype and phenotype, there is no straightforward way to implement novel biological functions and design new life forms. Both the developmental complexity of gene expression and the multifarious interactions of genes and environments are serious obstacles for "engineering" a particular phenotype. The problem of reverse-engineering a desired phenotype to its genetic "instructions" is probably intractable for any but the most simple phenotypes. Recent developments in the field of bio-engineering and synthetic biology reflect these limitations. Instead of genetically engineering a desired trait from scratch, as the machine/engineering metaphor promises, researchers are making greater strides by co-opting natural selection to "search" for a suitable genotype, or by borrowing and recombining genetic material from extant life forms.

Early warning of illegal development for protected areas by integrating cellular automata with neural networks

Ecological security has become a major issue under fast urbanization in China. As the first two cities in this country, Shenzhen and Dongguan issued the ordinance of Eco-designated Line of Control (ELC) to "wire" ecologically important areas for strict protection in 2005 and 2009 respectively. Early warning systems (EWS) are a useful tool for assisting the implementation ELC. In this study, a multi-model approach is proposed for the early warning of illegal development by integrating cellular automata (CA) and artificial neural networks (ANN). The objective is to prevent the ecological risks or catastrophe caused by such development at an early stage. The integrated model is calibrated by using the empirical information from both remote sensing and handheld GPS (global positioning systems). The MAR indicator which is the ratio of missing alarms to all the warnings is proposed for better assessment of the model performance. It is found that the fast urban development has caused significant threats to natural-area protection in the study area. The integration of CA, ANN and GPS provides a powerful tool for describing and predicting illegal development which is in highly non-linear and fragmented forms. The comparison shows that this multi-model approach has much better performances than the single-model approach for the early warning. Compared with the single models of CA and ANN, this integrated multi-model can improve the value of MAR by 65.48% and 5.17% respectively.

Biologically inspired face recognition: toward pose-invariance

A small change in image will cause a dramatic change in signals. Visual system is required to be able to ignore these changes, yet specific enough to perform recognition. This work intends to provide biological-backed insights into 2D translation and scaling invariance and 3D pose-invariance without imposing strain on memory and with biological justification. The model can be divided into lower and higher visual stages. Lower visual stage models the visual pathway from retina to the striate cortex (V1), whereas the modeling of higher visual stage is mainly based on current psychophysical evidences.

Decirculation process in neural network dynamics

We describe a decirculation process which marks perturbations of network structure and neural updating that are necessary for evolutionary neural networks to proceed from one circulating state to another. Two aspects of control parameters, screen updating and flow diagrams, are developed to quantify such perturbations, and hence to manage the dynamics of evolutionary neural networks. A dynamic state-shifting algorithm is derived from the decirculation process. This algorithm is used to build models of evolutionary content-addressable memory (ECAM) networks endowed with many dynamic relaxation processes. By the training of ECAM networks based on the dynamic state-shifting algorithm, we obtain the classification of training samples and the construction of recognition mappings, both of which perform adaptive computations essential to CAM.

Neogenomic events challenge current models of heritability, neuronal plasticity dynamics, and machine learning

We address current needs for neogenomics-based theoretical and computational approaches for several neuroscience research fields, from investigations of heritability properties, passing by investigations of spatiotemporal dynamics in the neuromodulatory microcircuits involved in perceptual learning and attentional shifts, to the application of genetic algorithms to create robots exhibiting ongoing emergence.

An analysis of the gamma memory in dynamic neural networks

Presents a vector space framework to study short-term memory filters in dynamic neural networks. The authors define parameters to quantify the function of feedforward and recursive linear memory filters. They show, using vector spaces, what is the optimization problem solved by the PEs of the first hidden layer of the single input focused network architecture. Due to the special properties of the gamma bases, recursion brings an extra parameter lambda (the time constant of the leaky integrator) that displaces the memory manifold towards the desired signal when the mean square error is minimized. In contrast, for the feedforward memory filter the angle between the desired signal and the memory manifold is fixed for a given memory order. The adaptation of the feedback parameter can be done using gradient descent, but the optimization is nonconvex.

Neural network approaches to dynamic collision-free trajectory generation

In this paper, dynamic collision-free trajectory generation in a nonstationary environment is studied using biologically inspired neural network approaches. The proposed neural network is topologically organized, where the dynamics of each neuron is characterized by a shunting equation or an additive equation. The state space of the neural network can be either the Cartesian workspace or the joint space of multi-joint robot manipulators. There are only local lateral connections among neurons. The real-time optimal trajectory is generated through the dynamic activity landscape of the neural network without explicitly searching over the free space nor the collision paths, without explicitly optimizing any global cost functions, without any prior knowledge of the dynamic environment, and without any learning procedures. Therefore the model algorithm is computationally efficient. The stability of the neural network system is guaranteed by the existence of a Lyapunov function candidate. In addition, this model is not very sensitive to the model parameters. Several model variations are presented and the differences are discussed. As examples, the proposed models are applied to generate collision-free trajectories for a mobile robot to solve a maze-type of problem, to avoid concave U-shaped obstacles, to track a moving target and at the same to avoid varying obstacles, and to generate a trajectory for a two-link planar robot with two targets. The effectiveness and efficiency of the proposed approaches are demonstrated through simulation and comparison studies.

Recurrent neural network training with feedforward complexity

This paper presents a training method that is of no more than feedforward complexity for fully recurrent networks. The method is not approximate, but rather depends on an exact transformation that reveals an embedded feedforward structure in every recurrent network. It turns out that given any unambiguous training data set, such as samples of the state variables and their derivatives, we need only to train this embedded feedforward structure. The necessary recurrent network parameters are then obtained by an inverse transformation that consists only of linear operators. As an example of modeling a representative nonlinear dynamical system, the method is applied to learn Bessel's differential equation, thereby generating Bessel functions within, as well as outside the training set.

VIRTUAL INSTRUMENTATION BASED SYSTEMS FOR REAL-TIME PATH PLANNING OF MOBILE ROBOTS USING BIO-INSPIRED NEURAL NETWORKS

In this paper, novel virtual instrumentation based systems for real-time collision-free path planning and tracking control of mobile robots are proposed. The developed virtual instruments are computationally simple and efficient in comparison to other approaches, which act as a new soft-computing platform to implement a biologically-inspired neural network. This neural network is topologically arranged with only local lateral connections among neurons. The dynamics of each neuron is described by a shunting equation with both excitatory and inhibitory connections. The neural network requires no off-line training or on-line learning, which is capable of planning a comfortable trajectory to the target without suffering from neither the too close nor the too far problems. LabVIEW is chosen as the software platform to build the proposed virtual instrumentation systems, as it is one of the most important industrial platforms. We take the initiative to develop the first neuro-dynamic application in LabVIEW. The developed virtual instruments could be easily used as educational and research tools for studying various robot path planning and tracking situations that could be easily understood and analyzed step by step. The effectiveness and efficiency of the developed virtual instruments are demonstrated through simulation and comparison studies.

A computational model of fMRI activity in the intraparietal sulcus that supports visual working memory

A computational model was developed to explain a pattern of results of fMRI activation in the intraparietal sulcus (IPS) supporting visual working memory for multiobject scenes. The model is based on the hypothesis that dendrites of excitatory neurons are major computational elements in the cortical circuit. Dendrites enable formation of a competitive queue that exhibits a gradient of activity values for nodes encoding different objects, and this pattern is stored in working memory. In the model, brain imaging data are interpreted as a consequence of blood flow arising from dendritic processing. Computer simulations showed that the model successfully simulates data showing the involvement of inferior IPS in object individuation and spatial grouping through representation of objects' locations in space, along with the involvement of superior IPS in object identification through representation of a set of objects' features. The model exhibits a capacity limit due to the limited dynamic range for nodes and the operation of lateral inhibition among them. The capacity limit is fixed in the inferior IPS regardless of the objects' complexity, due to the normalization of lateral inhibition, and variable in the superior IPS, due to the different encoding demands for simple and complex shapes. Systematic variation in the strength of self-excitation enables an understanding of the individual differences in working memory capacity. The model offers several testable predictions regarding the neural basis of visual working memory.

Exponential synchronization of stochastic Cohen-Grossberg neural networks with mixed time-varying delays and reaction-diffusion via periodically intermittent control

The issue of exponential synchronization for Cohen-Grossberg neural networks with mixed time-varying delays, stochastic noise disturbance and reaction-diffusion effects is investigated. An approach combining Lyapunov stability theory with stochastic analysis approaches and periodically intermittent control is taken to investigate this problem. The proposed criterion for exponential synchronization generalizes and improves those reported recently in the literature. This paper also presents an illustrative example and uses simulated results of this example to show the feasibility and effectiveness of the proposed scheme.

Contrast normalization contributes to a biologically-plausible model of receptive-field development in primary visual cortex (V1)

Neuronal populations in the primary visual cortex (V1) of mammals exhibit contrast normalization. Neurons that respond strongly to simple visual stimuli – such as sinusoidal gratings – respond less well to the same stimuli when they are presented as part of a more complex stimulus which also excites other, neighboring neurons. This phenomenon is generally attributed to generalized patterns of inhibitory connections between nearby V1 neurons. The Bienenstock, Cooper and Munro (BCM) rule is a neural network learning rule that, when trained on natural images, produces model neurons which, individually, have many tuning properties in common with real V1 neurons. However, when viewed as a population, a BCM network is very different from V1 – each member of the BCM population tends to respond to the same dominant features of visual input, producing an incomplete, highly redundant code for visual information. Here, we demonstrate that, by adding contrast normalization into the BCM rule, we arrive at a neurally-plausible Hebbian learning rule that can learn an efficient sparse, overcomplete representation that is a better model for stimulus selectivity in V1. This suggests that one role of contrast normalization in V1 is to guide the neonatal development of receptive fields, so that neurons respond to different features of visual input.

Type-K Exponential Ordering with Application to Delayed Hopfield-Type Neural Networks

Order-preserving and convergent results of delay functional differential equations without quasimonotone condition are established under type-K exponential ordering. As an application, the model of delayed Hopfield-type neural networks with a type-K monotone interconnection matrix is considered, and the attractor result is obtained.

Normalization as a canonical neural computation

There is increasing evidence that the brain relies on a set of canonical neural computations, repeating them across brain regions and modalities to apply similar operations to different problems. A promising candidate for such a computation is normalization, in which the responses of neurons are divided by a common factor that typically includes the summed activity of a pool of neurons. Normalization was developed to explain responses in the primary visual cortex and is now thought to operate throughout the visual system, and in many other sensory modalities and brain regions. Normalization may underlie operations such as the representation of odours, the modulatory effects of visual attention, the encoding of value and the integration of multisensory information. Its presence in such a diversity of neural systems in multiple species, from invertebrates to mammals, suggests that it serves as a canonical neural computation.

Bioinspired neural network for real-time cooperative hunting by multirobots in unknown environments

Multiple robot cooperation is a challenging and critical issue in robotics. To conduct the cooperative hunting by multirobots in unknown and dynamic environments, the robots not only need to take into account basic problems (such as searching, path planning, and collision avoidance), but also need to cooperate in order to pursue and catch the evaders efficiently. In this paper, a novel approach based on a bioinspired neural network is proposed for the real-time cooperative hunting by multirobots, where the locations of evaders and the environment are unknown and changing. The bioinspired neural network is used for cooperative pursuing by the multirobot team. Some other algorithms are used to enable the robots to catch the evaders efficiently, such as the dynamic alliance and formation construction algorithm. In the proposed approach, the pursuing alliances can dynamically change and the robot motion can be adjusted in real-time to pursue the evader cooperatively, to guarantee that all the evaders can be caught efficiently. The proposed approach can deal with various situations such as when some robots break down, the environment has different boundary shapes, or the obstacles are linked with different shapes. The simulation results show that the proposed approach is capable of guiding the robots to achieve the hunting of multiple evaders in real-time efficiently.

A unified approach to the stability of generalized static neural networks with linear fractional uncertainties and delays

In this paper, the robust global asymptotic stability (RGAS) of generalized static neural networks (SNNs) with linear fractional uncertainties and a constant or time-varying delay is concerned within a novel input-output framework. The activation functions in the model are assumed to satisfy a more general condition than the usually used Lipschitz-type ones. First, by four steps of technical transformations, the original generalized SNN model is equivalently converted into the interconnection of two subsystems, where the forward one is a linear time-invariant system with a constant delay while the feedback one bears the norm-bounded property. Then, based on the scaled small gain theorem, delay-dependent sufficient conditions for the RGAS of generalized SNNs are derived via combining a complete Lyapunov functional and the celebrated discretization scheme. All the results are given in terms of linear matrix inequalities so that the RGAS problem of generalized SNNs is projected into the feasibility of convex optimization problems that can be readily solved by effective numerical algorithms. The effectiveness and superiority of our results over the existing ones are demonstrated by two numerical examples.