The target switch algorithm: a constructive learning procedure for feed-forward neural networks

The constraint based decomposition (CBD) training architecture

The Constraint Based Decomposition (CBD) is a constructive neural network technique that builds a three or four layer network, has guaranteed convergence and can deal with binary, n-ary, class labeled and real-value problems. CBD is shown to be able to solve complicated problems in a simple, fast and reliable manner. The technique is further enhanced by two modifications (locking detection and redundancy elimination) which address the training speed and the efficiency of the internal representation built by the network. The redundancy elimination aims at building more compact architectures while the locking detection aims at improving the training speed. The computational cost of the redundancy elimination is negligible and this enhancement can be used for any problem. However, the computational cost of the locking detection is exponential in the number of dimensions and should only be used in low dimensional spaces. The experimental results show the performance of the algorithm presented in a series of classical benchmark problems including the 2-spiral problem and the Iris, Wine, Glass, Lenses, Ionosphere, Lung cancer, Pima Indians, Bupa, TicTacToe, Balance and Zoo data sets from the UCI machine learning repository. CBD's generalization accuracy is compared with that of C4.5, C4.5 with rules, incremental decision trees, oblique classifiers, linear machine decision trees, CN2, learning vector quantization (LVQ), backpropagation, nearest neighbor, Q* and radial basis functions (RBFs). CBD provides the second best average accuracy on the problems tested as well as the best reliability (the lowest standard deviation).

The recursive deterministic perceptron neural network

We introduce a feedforward multilayer neural network which is a generalization of the single layer perceptron topology (SLPT), called recursive deterministic perceptron (RDP). This new model is capable of solving any two-class classification problem, as opposed to the single layer perceptron which can only solve classification problems dealing with linearly separable sets (two subsets X and Y of R(d) are said to be linearly separable if there exists a hyperplane such that the elements of X and Y lie on the two opposite sides of R(d) delimited by this hyperplane). We propose several growing methods for constructing a RDP. These growing methods build a RDP by successively adding intermediate neurons (IN) to the topology (an IN corresponds to a SLPT). Thus, as a result, we obtain a multilayer perceptron topology, which together with the weights, are determined automatically by the constructing algorithms. Each IN augments the affine dimension of the set of input vectors. This augmentation is done by adding the output of each of these INs, as a new component, to every input vector. The construction of a new IN is made by selecting a subset from the set of augmented input vectors which is LS from the rest of this set. This process ends with LS classes in almost n-1 steps where n is the number of input vectors. For this construction, if we assume that the selected LS subsets are of maximum cardinality, the problem is proven to be NP-complete. We also introduce a generalization of the RDP model for classification of m classes (m>2) allowing to always separate m classes. This generalization is based on a new notion of linear separability for m classes, and it follows naturally from the RDP. This new model can be used to compute functions with a finite domain, and thus, to approximate continuous functions. We have also compared - over several classification problems - the percentage of test data correctly classified, or the topology of the 2 and m classes RDPs with that of the backpropagation (BP), cascade correlation (CC), and two other growing methods.

Two constructive methods for designing compact feedforward networks of threshold units

We propose two algorithms for constructing and training compact feedforward networks of linear threshold units. The SHIFT procedure constructs networks with a single hidden layer while the PTI constructs multilayered networks. The resulting networks are guaranteed to perform any given task with binary or real-valued inputs. The various experimental results reported for tasks with binary and real-valued inputs indicate that our methods compare favorably with alternative procedures deriving from similar strategies, both in terms of size of the resulting networks and of their generalization properties.