Universal Approximation Using Radial-Basis-Function Networks

Neural networks with local receptive fields and superlinear VC dimension

Local receptive field neurons comprise such well-known and widely used unit types as radial basis function (RBF) neurons and neurons with center-surround receptive field. We study the Vapnik-Chervonenkis (VC) dimension of feedforward neural networks with one hidden layer of these units. For several variants of local receptive field neurons, we show that the VC dimension of these networks is superlinear. In particular, we establish the bound Omega(W log k) for any reasonably sized network with W parameters and k hidden nodes. This bound is shown to hold for discrete center-surround receptive field neurons, which are physiologically relevant models of cells in the mammalian visual system, for neurons computing a difference of gaussians, which are popular in computational vision, and for standard RBF neurons, a major alternative to sigmoidal neurons in artificial neural networks. The result for RBF neural networks is of particular interest since it answers a question that has been open for several years. The results also give rise to lower bounds for networks with fixed input dimension. Regarding constants, all bounds are larger than those known thus far for similar architectures with sigmoidal neurons. The superlinear lower bounds contrast with linear upper bounds for single local receptive field neurons also derived here.

Use of artificial neural networks in prostate cancer

Artificial neural networks (ANNs) are a type of artificial intelligence software inspired by biological neuronal systems that can be used for nonlinear statistical modeling. In recent years, these applications have played an increasing role in predictive and classification modeling in medical research. We review the basic concepts behind ANNs and examine the role of this technology in selected applications in prostate cancer research.

Enhanced 3D shape recovery using the neural-based hybrid reflectance model

It is known that most real surfaces usually are neither perfectly Lambertian model nor ideally specular model; rather, they are formed by the hybrid structure of these two models. This hybrid reflectance model still suffers from the noise, strong specular, and unknown reflectivity conditions. In this article, these limitations are addressed, and a new neural-based hybrid reflectance model is proposed. The goal of this method is to optimize a proper reflectance model by learning the weight and parameters of the hybrid structure of feedforward neural networks and radial basis function networks and to recover the 3D object shape by the shape from shading technique with this resulting model. Experimental results, including synthetic and real images, were performed to demonstrate the performance of the proposed reflectance model in the case of different specular effects and noise environments.

A NEURAL NETWORK-BASED IMAGE RETRIEVAL USING NONLINEAR COMBINATION OF HETEROGENEOUS FEATURES

In content-based image retrieval (CBIR), content of an image can be expressed in terms of different features such as color, texture, shape, or text annotations. Retrieval based on these features can be various by the way how to combine the feature values. Most of the existing approaches assume a linear relationship between different features, and the usefulness of such systems was limited due to the difficulty in representing high-level concepts using low-level features.

In this paper, we introduce Neural Network-based Flexible Image Retrieval (NNFIR) system, a human-computer interaction approach to CBIR using Radial Basis Function (RBF) network to combine the values of the heterogeneous features. By using the RBF network, this approach determines nonlinear relationship between features so that more accurate similarity comparison between images can be supported. The experimental results show that the proposed approach has the superior retrieval performance than the existing linear combining method, the rank-based method and the BackPropagation-based method. Although the proposed retrieval model is for CBIR, it can be easily expanded to handle other media types, such as video and audio.

Artificial neural networks: opening the black box

Artificial neural networks now are used in many fields. They have become well established as viable, multipurpose, robust computational methodologies with solid theoretic support and with strong potential to be effective in any discipline, especially medicine. For example, neural networks can extract new medical information from raw data, build computer models that are useful for medical decision-making, and aid in the distribution of medical expertise. Because many important neural network applications currently are emerging, the authors have prepared this article to bring a clearer understanding of these biologically inspired computing paradigms to anyone interested in exploring their use in medicine. They discuss the historical development of neural networks and provide the basic operational mathematics for the popular multilayered perceptron. The authors also describe good training, validation, and testing techniques, and discuss measurements of performance and reliability, including the use of bootstrap methods to obtain confidence intervals. Because it is possible to predict outcomes for individual patients with a neural network, the authors discuss the paradigm shift that is taking place from previous "bin-model" approaches, in which patient outcome and management is assumed from the statistical groups in which the patient fits. The authors explain that with neural networks it is possible to mediate predictions for individual patients with prevalence and misclassification cost considerations using receiver operating characteristic methodology. The authors illustrate their findings with examples that include prostate carcinoma detection, coronary heart disease risk prediction, and medication dosing. The authors identify and discuss obstacles to success, including the need for expanded databases and the need to establish multidisciplinary teams. The authors believe that these obstacles can be overcome and that neural networks have a very important role in future medical decision support and the patient management systems employed in routine medical practice.

A review of analytical techniques for gait data. Part 2: neural network and wavelet methods

Multivariate gait data have traditionally been challenging to analyze. Part 1 of this review explored applications of fuzzy, multivariate statistical and fractal methods to gait data analysis. Part 2 extends this critical review to the applications of artificial neural networks and wavelets to gait data analysis. The review concludes with a practical guide to the selection of alternative gait data analysis methods. Neural networks are found to be the most prevalent non-traditional methodology for gait data analysis in the last 10 years. Interpretation of multiple gait signal interactions and quantitative comparisons of gait waveforms are identified as important data analysis topics in need of further research.

THE APPLICATION OF FEEDFORWARD NEURAL NETWORKS IN VLSI FABRICATION PROCESS OPTIMIZATION

In this paper, we present a neural-based manufacturing process control system for semiconductor factories to improve the die yield. A model based on neural networks is proposed to simulate Very Large-Scale Integrated (VLSI) manufacturing process. Learning from the historical processing lists with Radial Basis Function (RBF), we simulate the functional relationship between the wafer probing parameters and the die yield. Then we use a gradient-descent method to search a set of 'optimal' parameters that lead to the maximum yield of the model. At last, we adjust the specification in the practical semiconductor manufacturing process. The average die yield increased from 51.7% to 57.5% after the system had been applied in Huajing Corporation.

Modular neural networks for non-linearity recovering by the Haar approximation

The paper deals with the design of a composite neural system for recovering non-linear characteristics from random input-output measurement data. It is assumed that non-linearity output measurements are corrupted by an additive zero-mean white random noise and that the input excitation is an i.i.d. random sequence with an arbitrary (and unknown) probability density function. A class of modular networks is developed. The class is based on the Haar approximation of functions with piecewise constant functions on a refinable grid and consists of the networks composed of perceptron-like modules connected in parallel. The networks provide a local mean value estimators of functions. The relationship between complexity and accuracy of modular networks is analysed. It is shown that under mild conditions on the non-linearities and input probability density functions the networks yield pointwise consistent estimates of non-linear characteristics, provided that complexity of the networks grows appropriately with the number of training data. Efficiency of the networks is examined and the asymptotic rate of convergence of the network estimates is established. Specifically, local ability of the networks to recover non-linear characteristics in dependence on local smoothness of the underlying non-linear function and the input probability density is discussed. Optimum complexity selection rules, guaranteeing the best performance of the networks, are given. Illustrative simulation examples are provided.

Approximation of continuous and discontinuous mappings by a growing neural RBF-based algorithm

In this paper a neural network for approximating continuous and discontinuous mappings is described. The activation functions of the hidden nodes are the Radial Basis Functions (RBF) whose variances are learnt by means of an evolutionary optimization strategy. A new incremental learning strategy is used in order to improve the net performances. The learning strategy is able to save computational time because of the selective growing of the net structure and the capability of the learning algorithm to keep the effects of the activation functions local. Further, it does not require high order derivatives. An analysis of the learning capabilities and a comparison of the net performances with other approaches reported in literature have been performed. It is shown that the resulting network improves the approximation results reported for continuous mappings and for those exhibiting a finite number of discontinuities.

Radical pruning: a method to construct skeleton radial basis function networks

Trained radial basis function networks are well-suited for use in extracting rules and explanations because they contain a set of locally tuned units. However, for rule extraction to be useful, these networks must first be pruned to eliminate unnecessary weights. The pruning algorithm cannot search the network exhaustively because of the computational effort involved. It is shown that using multiple pruning methods with smart ordering of the pruning candidates, the number of weights in a radial basis function network can be reduced to a small fraction of the original number. The complexity of the pruning algorithm is quadratic (instead of exponential) in the number of network weights. Pruning performance is shown using a variety of benchmark problems from the University of California, Irvine machine learning database.

Progressive cross-section display of 3D medical images

The paper presents a hierarchical coding algorithm for 3D medical images based upon hierarchical interpolation with radial basis function networks. By using the properties of the Kronecker product, the computation of the network parameters and the 3D image reconstruction are efficiently done in (L4) computation time and O(L3) storage space, when applied to 3D images of size (L x L x L). A further reduction in processing time is accomplished by using sparse matrix techniques. The salient features of the proposed coding method are that arbitrary cross-section images can be progressively displayed without reconstruction of the whole 3D image; the first image reconstruction starts as soon as the first data transmission has been completed; no expanding procedure is required in 3D image reconstruction, and the blocking effects are not apparent even in the lowest-resolution image. Experimental results using two 3D MRI images, of size (128 x 18 x 64) and with 8-bit grey levels, show that the coding performance is better than that of the 3D DCT coding by about 0.25 bits pixel-1 at higher bit rates, and that the new cross-section display method synthesises the coarsest (finest) section image about six (three) times faster than the standard method that requires the whole 3D image reconstruction.

Digital color restoration of old paintings

Physical and chemical changes can degrade the visual color appearance of old paintings. Five digital color restoration techniques, which can be used to simulate the original appearance of paintings, are presented. Although a small number of color samples is employed in the restoration procedure, simulation results indicate that good restoration quality can be attained.

Inferential estimation of polymer quality using bootstrap aggregated neural networks

Inferential estimation of polymer quality in a batch polymerisation reactor using bootstrap aggregated neural networks is studied in this paper. Number average molecular weight and weight average molecular weight are estimated from the on-line measurements of reactor temperature, jacket inlet and outlet temperatures, coolant flow rate through the jacket, monomer conversion, and the initial batch conditions. Bootstrap aggregated neural networks are used to enhance the accuracy and robustness of neural network models built from a limited amount of training data. The training data set is re-sampled using bootstrap re-sampling with replacement to form several sets of training data. For each set of training data, a neural network model is developed. The individual neural networks are then combined together to form a bootstrap aggregated neural network. Determination of appropriate weights for combining individual networks using principal component regression is proposed in this paper. Confidence bounds for neural network predictions can also be obtained using the bootstrapping technique. The techniques have been successfully applied to the simulation of a batch methyl methacrylate polymerisation reactor.

Artificial neural networks as approximators of stochastic processes

Artificial Neural Networks (ANNs) must be able to learn by experience from environment. This property can be considered as being closely related to the approximating capabilities of the networks. Unfortunately at present only the ability of ANNs in approximating deterministic input-output mappings has been exploited. In this article it has been shown that some classes of neural networks, named Stochastic Neural Networks, which are capable of using approximating stochastic processes are defined. As stochastic processes may also be viewed as random functions, they include deterministic (non-random) functions as a particular case. Thus the class of Stochastic Neural Networks can be considered as a generalisation of the usually defined neural networks. From an application point of view such a class of networks is more adherent to real world in which neural networks must work in an environment which is essentially stochastic. The theory presented in the article has been carried out starting from the so-called "canonical representation" for non-stationary stochastic processes. Finally, an application example showing in detail the validity of the proposed approach has been reported.

A tracing evoked potential estimator

The paper presents an adaptive Gaussian radial basis function neural network (RBFNN) for rapid estimation of evoked potential (EP). Usually, a recorded EP is severely contaminated by background ongoing activities of the brain. Many approaches have been reported to enhance the signal-to-noise ratio (SNR) of the recorded signal. However, non-linear methods are seldom explored due to their complexity and the fact that the non-linear characteristics of the signal are generally hard to determine. An RBFNN possesses built-in non-linear activation functions that enable the neural network to learn any function mapping. An RBFNN was carefully designed to model the EP signal. It has the advantage of being linear-in-parameter, thus a conventional adaptive method can efficiently estimate its parameters. The proposed algorithm is simple so that its convergence behaviour and performance in signal-to-noise ratio (SNR) improvement can be mathematically derived. A series of experiments carried out on simulated and human test responses confirmed the superior performance of the method. In a simulation experiment, an RBFNN having 15 hidden nodes was trained to approximate human visual EP (VEP). For detecting human brain stem auditory EP (BAEP), the approach (40 hidden nodes and convergence rate = 0.005) speeded up the estimation remarkably by using only 80 ensembles to achieve a result comparable to that obtained by averaging 1000 ensembles.

Object classification in 3-D images using alpha-trimmed mean radial basis function network

We propose a pattern classification based approach for simultaneous three-dimensional (3-D) object modeling and segmentation in image volumes. The 3-D objects are described as a set of overlapping ellipsoids. The segmentation relies on the geometrical model and graylevel statistics. The characteristic parameters of the ellipsoids and of the graylevel statistics are embedded in a radial basis function (RBF) network and they are found by means of unsupervised training. A new robust training algorithm for RBF networks based on alpha-trimmed mean statistics is employed in this study. The extension of the Hough transform algorithm in the 3-D space by employing a spherical coordinate system is used for ellipsoidal center estimation. We study the performance of the proposed algorithm and we present results when segmenting a stack of microscopy images.

Hybrid interior point training of modular neural networks

Modular neural networks use a single gating neuron to select the outputs of a collection of agent neurons. Expectation-maximization (EM) algorithms provide one way of training modular neural networks to approximate non-linear functionals. This paper introduces a hybrid interior-point (HIP) algorithm for training modular networks. The HIP algorithm combines an interior-point linear programming (LP) algorithm with a Newton-Raphson iteration in such a way that the computational efficiency of the interior point LP methods is preserved. The algorithm is formally proven to converge asymptotically to locally optimal networks with a total computational cost that scales in a polynomial manner with problem size. Simulation experiments show that the HIP algorithm produces networks whose average approximation error is better than that of EM-trained networks. These results also demonstrate that the computational cost of the HIP algorithm scales at a slower rate than the EM-procedure and that, for small-size networks, the total computational costs of both methods are comparable.

Cubic approximation neural network for multivariate functions

This paper introduces a novel neural network architecture-cubic approximation neural network (CANN), capable of local approximation of multivariate functions. It is particularly simple in concept and in structure. Its simplicity enables a quantitative evaluation of its approximation capabilities, namely, for a desired error bound the size of the needed network can be calculated. In addition, if a training session is used, a thorough analysis of the learning process performance is performed. The trade-off between the rate of learning and the steady-state performance is clearly demonstrated. On the other hand, this approach suffers from the problem common to all local approximation networks-the number of neurons grows exponentially with the dimension of the input vector.

A Sequential Learning Approach for Single Hidden Layer Neural Networks

A sequential orthogonal approach to the building and training of single hidden layer neural networks is presented in this paper. In the proposed method, hidden neurons are added one at a time. The procedure starts with a single hidden neuron and sequentially increases the number of hidden neurons until the model error is sufficiently small. When adding a neuron, the new information introduced by this neuron is caused by that part of its output vector which is orthogonal to the space spanned by the output vectors of previously added hidden neurons. The classical Gram-Schmidt orthogonalization method is used at each step to form a set of orthogonal bases for the space spanned by the output vectors of hidden neurons. Hidden layer weights are found through optimization while output layer weights are obtained from the least-squares regression. Using the proposed technique, it is possible to determine the necessary number of hidden neurons required. A regularization factor is also incorporated into the sequential orthogonal training algorithm to improve the network generalization capability. An additional advantage of this method is that it can be used to build and train neural networks with mixed types of hidden neurons and thus to develop hybrid models. By using mixed types of neurons, it is found that more accurate neural network models, with a smaller number of hidden neurons than seen in conventional networks, can be developed. The proposed sequential orthogonal training method was successfully applied to three non-linear modelling examples.

Optical flow estimation and moving object segmentation based on median radial basis function network

Various approaches have been proposed for simultaneous optical flow estimation and segmentation in image sequences. In this study, the moving scene is decomposed into different regions with respect to their motion, by means of a pattern recognition scheme. The inputs of the proposed scheme are the feature vectors representing still image and motion information. Each class corresponds to a moving object. The classifier employed is the median radial basis function (MRBF) neural network. An error criterion function derived from the probability estimation theory and expressed as a function of the moving scene model is used as the cost function. Each basis function is activated by a certain image region. Marginal median and median of the absolute deviations from the median (MAD) estimators are employed for estimating the basis function parameters. The image regions associated with the basis functions are merged by the output units in order to identify moving objects.