Modular neural networks: a survey

Artificial Evolution Network: A Computational Perspective on the Expansibility of the Nervous System

Neurobiologists recently found the brain can use sudden emerged channels to process information. Based on this finding, we put forward a question whether we can build a computation model that is able to integrate a sudden emerged new type of perceptual channel into itself in an online way. If such a computation model can be established, it will introduce a channel-free property to the computation model and meanwhile deepen our understanding about the extendibility of the brain. In this article, a biologically inspired neural network named artificial evolution (AE) network is proposed to handle the problem. When a new perceptual channel emerges, the neurons in the network can grow new connections to connect the emerged channel according to the Hebb rule. In this article, we design a sensory channel expansion experiment to test the AE network. The experimental results demonstrate that the AE network can handle the sudden emerged perceptual channels effectively.

Spontaneous sparse learning for PCM-based memristor neural networks

Neural networks trained by backpropagation have achieved tremendous successes on numerous intelligent tasks. However, naïve gradient-based training and updating methods on memristors impede applications due to intrinsic material properties. Here, we built a 39 nm 1 Gb phase change memory (PCM) memristor array and quantified the unique resistance drift effect. On this basis, spontaneous sparse learning (SSL) scheme that leverages the resistance drift to improve PCM-based memristor network training is developed. During training, SSL regards the drift effect as spontaneous consistency-based distillation process that reinforces the array weights at the high-resistance state continuously unless the gradient-based method switches them to low resistance. Experiments show that the SSL not only helps the convergence of network with better performance and sparsity controllability without additional computation in handwritten digit classification. This work promotes the learning algorithms with the intrinsic properties of memristor devices, opening a new direction for development of neuromorphic computing chips.

Distributed dual vigilance fuzzy adaptive resonance theory learns online, retrieves arbitrarily-shaped clusters, and mitigates order dependence

This paper presents a novel adaptive resonance theory (ART)-based modular architecture for unsupervised learning, namely the distributed dual vigilance fuzzy ART (DDVFA). DDVFA consists of a global ART system whose nodes are local fuzzy ART modules. It is equipped with distributed higher-order activation and match functions and a dual vigilance mechanism. Together, these allow DDVFA to perform unsupervised modularization, create multi-prototype cluster representations, retrieve arbitrarily-shaped clusters, and reduce category proliferation. Another important contribution is the reduction of order-dependence, an issue that affects any agglomerative clustering method. This paper demonstrates two approaches for mitigating order-dependence: pre-processing using visual assessment of cluster tendency (VAT) or post-processing using a novel Merge ART module. The former is suitable for batch processing, whereas the latter also works for online learning. Experimental results in online mode carried out on 30 benchmark data sets show that DDVFA cascaded with Merge ART statistically outperformed the best other ART-based systems when samples were randomly presented. Conversely, they were found to be statistically equivalent in offline mode when samples were pre-processed using VAT. Remarkably, performance comparisons to non-ART-based clustering algorithms show that DDVFA (which learns incrementally) was also statistically equivalent to the non-incremental (offline) methods of density-based spatial clustering of applications with noise (DBSCAN), single linkage hierarchical agglomerative clustering (SL-HAC), and k-means, while retaining the appealing properties of ART. Links to the source code and data are provided. Considering the algorithm's simplicity, online learning capability, and performance, it is an ideal choice for many agglomerative clustering applications.

Perception Coordination Network: A Neuro Framework for Multimodal Concept Acquisition and Binding

To simulate the concept acquisition and binding of different senses in the brain, a biologically inspired neural network model named perception coordination network (PCN) is proposed. It is a hierarchical structure, which is functionally divided into the primary sensory area (PSA), the primary sensory association area (SAA), and the higher order association area (HAA). The PSA contains feature neurons which respond to many elementary features, e.g., colors, shapes, syllables, and basic flavors. The SAA contains primary concept neurons which combine the elementary features in the PSA to represent unimodal concept of objects, e.g., the image of an apple, the Chinese word "[píng guǒ]" which names the apple, and the taste of the apple. The HAA contains associated neurons which connect the primary concept neurons of several PSA, e.g., connects the image, the taste, and the name of an apple. It means that the associated neurons have a multimodal response mode. Therefore, this area executes multisensory integration. PCN is an online incremental learning system, it is able to continuously acquire and bind multimodality concepts in an online way. The experimental results suggest that PCN is able to handle the multimodal concept acquisition and binding effectively.

Formulation and Implementation of Nonlinear Integral Equations to Model Neural Dynamics Within the Vertebrate Retina

Existing computational models of the retina often compromise between the biophysical accuracy and a hardware-adaptable methodology of implementation. When compared to the current modes of vision restoration, algorithmic models often contain a greater correlation between stimuli and the affected neural network, but lack physical hardware practicality. Thus, if the present processing methods are adapted to complement very-large-scale circuit design techniques, it is anticipated that it will engender a more feasible approach to the physical construction of the artificial retina. The computational model presented in this research serves to provide a fast and accurate predictive model of the retina, a deeper understanding of neural responses to visual stimulation, and an architecture that can realistically be transformed into a hardware device. Traditionally, implicit (or semi-implicit) ordinary differential equations (OES) have been used for optimal speed and accuracy. We present a novel approach that requires the effective integration of different dynamical time scales within a unified framework of neural responses, where the rod, cone, amacrine, bipolar, and ganglion cells correspond to the implemented pathways. Furthermore, we show that adopting numerical integration can both accelerate retinal pathway simulations by more than 50% when compared with traditional ODE solvers in some cases, and prove to be a more realizable solution for the hardware implementation of predictive retinal models.

Artificial Epigenetic Networks: Automatic Decomposition of Dynamical Control Tasks Using Topological Self-Modification

This paper describes the artificial epigenetic network, a recurrent connectionist architecture that is able to dynamically modify its topology in order to automatically decompose and solve dynamical problems. The approach is motivated by the behavior of gene regulatory networks, particularly the epigenetic process of chromatin remodeling that leads to topological change and which underlies the differentiation of cells within complex biological organisms. We expected this approach to be useful in situations where there is a need to switch between different dynamical behaviors, and do so in a sensitive and robust manner in the absence of a priori information about problem structure. This hypothesis was tested using a series of dynamical control tasks, each requiring solutions that could express different dynamical behaviors at different stages within the task. In each case, the addition of topological self-modification was shown to improve the performance and robustness of controllers. We believe this is due to the ability of topological changes to stabilize attractors, promoting stability within a dynamical regime while allowing rapid switching between different regimes. Post hoc analysis of the controllers also demonstrated how the partitioning of the networks could provide new insights into problem structure.

Modeling Approach Based on Modular Fuzzy Model

Fuzzy modeling is one of the most important techniques for nonlinear modeling. SIRMs (Single Input Rule Modules) has been studied as a useful modeling method for real-life applications such as control and pattern recognition. Although the SIRMs is a practical modeling approach based on fuzzy reasoning, its performance is adversely affected by high-dimensional or complicated characteristics of the problems. The modular fuzzy model is an extension of the SIRMs for overcoming such a performance problem. In this paper, we study a modeling approach based on the modular fuzzy model by extending the SIRMs architecture. We show that the construction of error objective functions for modeling the modular fuzzy model and the SIRMs affects the prediction performance of the model. Through numerical experiments on modeling problems and reinforcement learning problems, we study the model construction based on the error objective functions. We find that the error objective function should be selected according to the number of dimensions of projection in the modular fuzzy model.

MODULAR NEURAL NETWORK DESIGN FOR THE PROBLEM OF ALPHABETIC CHARACTER RECOGNITION

This paper reports on an experimental approach to find a modularized artificial neural network solution for the UCI letters recognition problem. Our experiments have been carried out in two parts. We investigate directed task decomposition using expert knowledge and clustering approaches to find the subtasks for the modules of the network. We next investigate processes to combine the modules effectively in a single decision process. After having found suitable modules through task decomposition we have found through further experimentation that when the modules are combined with decision tree supervision, their functional error is reduced significantly to improve their combination through the decision process that has been implemented as a small multilayered perceptron. The experiments conclude with a modularized neural network design for this classification problem that has increased learning and generalization characteristics. The test results for this network are markedly better than a single or stand alone network that has a fully connected topology.

ANCHOR — A CONNECTIONIST ARCHITECTURE FOR PARTITIONING FEATURE SPACES AND HIERARCHICAL NESTING OF NEURAL NETS

We present a novel connectionist architecture for handling arbitrarily complex neural computations. The architecture — which we call an Artificial Neural Network Compiler for Hierarchical ORganisation (ANCHOR) — facilitates network hierarchy and the partitioning of the input space into a number of small, simpler sub-mappings. This strategy is a mix of both a modular approach for reduction of the problem complexity, and an ensemble-based approach for increasing prediction accuracy. ANCHOR is implemented around the concept of a Superneuron which is a generalised view of a neuron-processing element; a superneuron is a single (or higher-order) perceptron which can be trained individually with the support of multiple learning algorithms. The indistinguishability between a superneuron, and a neuron is employed in hierarchical nesting of superneurons, up to (theoretically) infinite depth, within other superneurons as well as linear or tree-structured cascading. Hierarchical decomposition of simple boolean functions has been demonstrated as proof-of-concept. It is argued that this strategy of data-partitioning for subsequent mapping onto a hierarchical classifier reduces the learning complexity and offers better generalisation.

Divide-and-conquer learning and modular perceptron networks

A novel modular perceptron network (MPN) and divide-and-conquer learning (DCL) schemes for the design of modular neural networks are proposed. When a training process in a multilayer perceptron falls into a local minimum or stalls in a flat region, the proposed DCL scheme is applied to divide the current training data region into two easier to be learned regions. The learning process continues when a self-growing perceptron network and its initial weight estimation are constructed for one of the newly partitioned regions. Another partitioned region will resume the training process on the original perceptron network. Data region partitioning, weight estimating and learning are iteratively repeated until all the training data are completely learned by the MPN. We evaluated and compared the proposed MPN with several representative neural networks on the two-spirals problem and real-world dataset. The MPN achieved better weight learning performance by requiring much less data presentations during the network training phases, and better generalization performance, and less processing time during the retrieving phase.

Cooperative recurrent modular neural networks for constrained optimization: a survey of models and applications

Constrained optimization problems arise in a wide variety of scientific and engineering applications. Since several single recurrent neural networks when applied to solve constrained optimization problems for real-time engineering applications have shown some limitations, cooperative recurrent neural network approaches have been developed to overcome drawbacks of these single recurrent neural networks. This paper surveys in details work on cooperative recurrent neural networks for solving constrained optimization problems and their engineering applications, and points out their standing models from viewpoint of both convergence to the optimal solution and model complexity. We provide examples and comparisons to shown advantages of these models in the given applications.

On global-local artificial neural networks for function approximation

We present a hybrid radial basis function (RBF) sigmoid neural network with a three-step training algorithm that utilizes both global search and gradient descent training. The algorithm used is intended to identify global features of an input-output relationship before adding local detail to the approximating function. It aims to achieve efficient function approximation through the separate identification of aspects of a relationship that are expressed universally from those that vary only within particular regions of the input space. We test the effectiveness of our method using five regression tasks; four use synthetic datasets while the last problem uses real-world data on the wave overtopping of seawalls. It is shown that the hybrid architecture is often superior to architectures containing neurons of a single type in several ways: lower mean square errors are often achievable using fewer hidden neurons and with less need for regularization. Our global-local artificial neural network (GL-ANN) is also seen to compare favorably with both perceptron radial basis net and regression tree derived RBFs. A number of issues concerning the training of GL-ANNs are discussed: the use of regularization, the inclusion of a gradient descent optimization step, the choice of RBF spreads, model selection, and the development of appropriate stopping criteria.

Incremental learning in biological and machine learning systems

Incremental learning concepts are reviewed in machine learning and neurobiology. They are identified in evolution, neurodevelopment and learning. A timeline of qualitative axon, neuron and synapse development summarizes the review on neurodevelopment. A discussion of experimental results on data incremental learning with recurrent artificial neural networks reveals that incremental learning often seems to be more efficient or powerful than standard learning but can produce unexpected side effects. A characterization of incremental learning is proposed which takes the elaborated biological and machine learning concepts into account.