Orthogonal least squares learning algorithm for radial basis function networks

Near infrared spectroscopic (NIRS) analysis of drug-loading rate and particle size of risperidone microspheres by improved chemometric model

Microspheres have been developed as drug carriers in controlled drug delivery systems for years. In our present study, near infrared spectroscopy (NIRS) is applied to analyze the particle size and drug loading rate in risperidone poly(d,l-lactide-co-glycolide) (PLGA) microspheres. Various batches of risperidone PLGA microspheres were designed and prepared successfully. The particle size and drug-loading rate of all the samples were determined by a laser diffraction particle size analyzer and high performance liquid chromatography (HPLC) system. Monte Carlo algorithm combined with partial least squares (MCPLS) method was applied to identify the outliers and choose the numbers of calibration set. Furthermore, a series of preprocessing methods were performed to remove signal noise in NIR spectra. Moving window PLS and radical basis function neural network (RBFNN) methods were employed to establish calibration model. Our data demonstrated that PLS-developed model was only suitable for drug loading analysis in risperidone PLGA microspheres. Comparatively, RBFNN-based predictive models possess better fitting quality, predictive effect, and stability for both drug loading rate and particle size analysis. The correlation coefficients of calibration set (Rc(2)) were 0.935 and 0.880, respectively. The performance of optimum RBFNN models was confirmed by independent verification test with 15 samples. Collectively, our method is successfully performed to monitor drug-loading rate and particle size during risperidone PLGA microspheres preparation.

Exploration of a capability-focused aerospace system of systems architecture alternative with bilayer design space, based on RST-SOM algorithmic methods

In defense related programs, the use of capability-based analysis, design, and acquisition has been significant. In order to confront one of the most challenging features of a huge design space in capability based analysis (CBA), a literature review of design space exploration was first examined. Then, in the process of an aerospace system of systems design space exploration, a bilayer mapping method was put forward, based on the existing experimental and operating data. Finally, the feasibility of the foregoing approach was demonstrated with an illustrative example. With the data mining RST (rough sets theory) and SOM (self-organized mapping) techniques, the alternative to the aerospace system of systems architecture was mapping from P-space (performance space) to C-space (configuration space), and then from C-space to D-space (design space), respectively. Ultimately, the performance space was mapped to the design space, which completed the exploration and preliminary reduction of the entire design space. This method provides a computational analysis and implementation scheme for large-scale simulation.

Incipient interturn fault diagnosis in induction machines using an analytic wavelet-based optimized Bayesian inference

Interturn fault diagnosis of induction machines has been discussed using various neural network-based techniques. The main challenge in such methods is the computational complexity due to the huge size of the network, and in pruning a large number of parameters. In this paper, a nearly shift insensitive complex wavelet-based probabilistic neural network (PNN) model, which has only a single parameter to be optimized, is proposed for interturn fault detection. The algorithm constitutes two parts and runs in an iterative way. In the first part, the PNN structure determination has been discussed, which finds out the optimum size of the network using an orthogonal least squares regression algorithm, thereby reducing its size. In the second part, a Bayesian classifier fusion has been recommended as an effective solution for deciding the machine condition. The testing accuracy, sensitivity, and specificity values are highest for the product rule-based fusion scheme, which is obtained under load, supply, and frequency variations. The point of overfitting of PNN is determined, which reduces the size, without compromising the performance. Moreover, a comparative evaluation with traditional discrete wavelet transform-based method is demonstrated for performance evaluation and to appreciate the obtained results.

ERNN: a biologically inspired feedforward neural network to discriminate emotion from EEG signal

Emotions play an important role in human cognition, perception, decision making, and interaction. This paper presents a six-layer biologically inspired feedforward neural network to discriminate human emotions from EEG. The neural network comprises a shift register memory after spectral filtering for the input layer, and the estimation of coherence between each pair of input signals for the hidden layer. EEG data are collected from 57 healthy participants from eight locations while subjected to audio-visual stimuli. Discrimination of emotions from EEG is investigated based on valence and arousal levels. The accuracy of the proposed neural network is compared with various feature extraction methods and feedforward learning algorithms. The results showed that the highest accuracy is achieved when using the proposed neural network with a type of radial basis function.

Finite time convergent learning law for continuous neural networks

This paper addresses the design of a discontinuous finite time convergent learning law for neural networks with continuous dynamics. The neural network was used here to obtain a non-parametric model for uncertain systems described by a set of ordinary differential equations. The source of uncertainties was the presence of some external perturbations and poor knowledge of the nonlinear function describing the system dynamics. A new adaptive algorithm based on discontinuous algorithms was used to adjust the weights of the neural network. The adaptive algorithm was derived by means of a non-standard Lyapunov function that is lower semi-continuous and differentiable in almost the whole space. A compensator term was included in the identifier to reject some specific perturbations using a nonlinear robust algorithm. Two numerical examples demonstrated the improvements achieved by the learning algorithm introduced in this paper compared to classical schemes with continuous learning methods. The first one dealt with a benchmark problem used in the paper to explain how the discontinuous learning law works. The second one used the methane production model to show the benefits in engineering applications of the learning law proposed in this paper.

A new discrete-continuous algorithm for radial basis function networks construction

The construction of a radial basis function (RBF) network involves the determination of the model size, hidden nodes, and output weights. Least squares-based subset selection methods can determine a RBF model size and its parameters simultaneously. Although these methods are robust, they may not achieve optimal results. Alternatively, gradient methods are widely used to optimize all the parameters. The drawback is that most algorithms may converge slowly as they treat hidden nodes and output weights separately and ignore their correlations. In this paper, a new discrete-continuous algorithm is proposed for the construction of a RBF model. First, the orthogonal least squares (OLS)-based forward stepwise selection constructs an initial model by selecting model terms one by one from a candidate term pool. Then a new Levenberg-Marquardt (LM)-based parameter optimization is proposed to further optimize the hidden nodes and output weights in the continuous space. To speed up the convergence, the proposed parameter optimization method considers the correlation between the hidden nodes and output weights, which is achieved by translating the output weights to dependent parameters using the OLS method. The correlation is also used by the previously proposed continuous forward algorithm (CFA). However, unlike the CFA, the new method optimizes all the parameters simultaneously. In addition, an equivalent recursive sum of squared error is derived to reduce the computation demanding for the first derivatives used in the LM method. Computational complexity is given to confirm the new method is much more computationally efficient than the CFA. Different numerical examples are presented to illustrate the effectiveness of the proposed method. Further, Friedman statistical tests on 13 classification problems are performed, and the results demonstrate that RBF networks built by the new method are very competitive in comparison with some popular classifiers.

Comparison of logistic regression and neural network classifiers in the detection of hard exudates in retinal images

Diabetic Retinopathy (DR) is a common cause of visual impairment in industrialized countries. Automatic recognition of DR lesions in retinal images can contribute to the diagnosis and screening of this disease. The aim of this study is to automatically detect one of these lesions: hard exudates (EXs). Based on their properties, we extracted a set of features from image regions and selected the subset that best discriminated between EXs and the retinal background using logistic regression (LR). The LR model obtained, a multilayer perceptron (MLP) classifier and a radial basis function (RBF) classifier were subsequently used to obtain the final segmentation of EXs. Our database contained 130 images with variable color, brightness, and quality. Fifty of them were used to obtain the training examples. The remaining 80 images were used to test the performance of the method. The highest statistics were achieved for MLP or RBF. Using a lesion based criterion, our results reached a mean sensitivity of 95.9% (MLP) and a mean positive predictive value of 85.7% (RBF). With an image-based criterion, we achieved a 100% mean sensitivity, 87.5% mean specificity and 93.8% mean accuracy (MLP and RBF).

Modeling of visuospatial perspectives processing and modulation of the fronto-parietal network activity during action imitation

It has been suggested that the human mirror neuron system (MNS) plays a critical role in action observation and imitation. However, the transformation of perspective between the observed (allocentric) and the imitated (egocentric) actions has received little attention. We expand a previously proposed biologically plausible MNS model by incorporating general spatial transformation capabilities that are assumed to be encoded by the intraparietal sulcus (IPS) and the superior parietal lobule (SPL) as well as investigating their interactions with the inferior frontal gyrus and the inferior parietal lobule. The results reveal that the IPS/SPL could process the frame of reference and the viewpoint transformations, and provide invariant visual representations for the temporo-parieto-frontal circuit. This allows the imitator to imitate the action performed by a demonstrator under various perspectives while replicating results from the literatures. Our results confirm and extend the importance of perspective transformation processing during action observation and imitation.

Classification of schizophrenia patients based on resting-state functional network connectivity

There is a growing interest in automatic classification of mental disorders based on neuroimaging data. Small training data sets (subjects) and very large amount of high dimensional data make it a challenging task to design robust and accurate classifiers for heterogeneous disorders such as schizophrenia. Most previous studies considered structural MRI, diffusion tensor imaging and task-based fMRI for this purpose. However, resting-state data has been rarely used in discrimination of schizophrenia patients from healthy controls. Resting data are of great interest, since they are relatively easy to collect, and not confounded by behavioral performance on a task. Several linear and non-linear classification methods were trained using a training dataset and evaluate with a separate testing dataset. Results show that classification with high accuracy is achievable using simple non-linear discriminative methods such as k-nearest neighbors (KNNs) which is very promising. We compare and report detailed results of each classifier as well as statistical analysis and evaluation of each single feature. To our knowledge our effects represent the first use of resting-state functional network connectivity (FNC) features to classify schizophrenia.

Genome-enabled methods for predicting litter size in pigs: a comparison

Predictive ability of models for litter size in swine on the basis of different sources of genetic information was investigated. Data represented average litter size on 2598, 1604 and 1897 60K genotyped sows from two purebred and one crossbred line, respectively. The average correlation (r) between observed and predicted phenotypes in a 10-fold cross-validation was used to assess predictive ability. Models were: pedigree-based mixed-effects model (PED), Bayesian ridge regression (BRR), Bayesian LASSO (BL), genomic BLUP (GBLUP), reproducing kernel Hilbert spaces regression (RKHS), Bayesian regularized neural networks (BRNN) and radial basis function neural networks (RBFNN). BRR and BL used the marker matrix or its principal component scores matrix (UD) as covariates; RKHS employed a Gaussian kernel with additive codes for markers whereas neural networks employed the additive genomic relationship matrix (G) or UD as inputs. The non-parametric models (RKHS, BRNN, RNFNN) gave similar predictions to the parametric counterparts (average r ranged from 0.15 to 0.23); most of the genome-based models outperformed PED (r = 0.16). Predictive abilities of linear models and RKHS were similar over lines, but BRNN varied markedly, giving the best prediction (r = 0.31) when G was used in crossbreds, but the worst (r = 0.02) when the G matrix was used in one of the purebred lines. The r values for RBFNN ranged from 0.16 to 0.23. Predictive ability was better in crossbreds (0.26) than in purebreds (0.15 to 0.22). This may be related to family structure in the purebred lines.

A New Surrogate-Assisted Interactive Genetic Algorithm With Weighted Semisupervised Learning

Surrogate-assisted interactive genetic algorithms (IGAs) are found to be very effective in reducing human fatigue. Different from models used in most surrogate-assisted evolutionary algorithms, surrogates in IGA must be able to handle the inherent uncertainties in fitness assignment by human users, where, e.g., interval-based fitness values are assigned to individuals. This poses another challenge to using surrogates for fitness approximation in evolutionary optimization, in addition to the lack of training data. In this paper, a new surrogate-assisted IGA has been proposed, where the uncertainty in subjective fitness evaluations is exploited both in training the surrogates and in managing surrogates. To enhance the approximation accuracy of the surrogates, an improved cotraining algorithm for semisupervised learning has been suggested, where the uncertainty in interval-based fitness values is taken into account in training and weighting the two cotrained models. Moreover, uncertainty in the interval-based fitness values is also considered in model management so that not only the best individuals but also the most uncertain individuals will be chosen to be re-evaluated by the human user. The effectiveness of the proposed algorithm is verified on two test problems as well as in fashion design, a typical application of IGA. Our results indicate that the new surrogate-assisted IGA can effectively alleviate user fatigue and is more likely to find acceptable solutions in solving complex design problems.

Radial basis function network training using a nonsymmetric partition of the input space and particle swarm optimization

This paper presents a novel algorithm for training radial basis function (RBF) networks, in order to produce models with increased accuracy and parsimony. The proposed methodology is based on a nonsymmetric variant of the fuzzy means (FM) algorithm, which has the ability to determine the number and locations of the hidden-node RBF centers, whereas the synaptic weights are calculated using linear regression. Taking advantage of the short computational times required by the FM algorithm, we wrap a particle swarm optimization (PSO) based engine around it, designed to optimize the fuzzy partition. The result is an integrated framework for fully determining all the parameters of an RBF network. The proposed approach is evaluated through its application on 12 real-world and synthetic benchmark datasets and is also compared with other neural network training techniques. The results show that the RBF network models produced by the PSO-based nonsymmetric FM algorithm outperform the models produced by the other techniques, exhibiting higher prediction accuracies in shorter computational times, accompanied by simpler network structures.

Feature combiners with gate-generated weights for classification

Using functional weights in a conventional linear combination architecture is a way of obtaining expressive power and represents an alternative to classical trainable and implicit nonlinear transformations. In this brief, we explore this way of constructing binary classifiers, taking advantage of the possibility of generating functional weights by means of a gate with fixed radial basis functions. This particular form of the gate permits training the machine directly with maximal margin algorithms. We call the resulting scheme "feature combiners with gate generated weights for classification." Experimental results show that these architectures outperform support vector machines (SVMs) and Real AdaBoost ensembles in most considered benchmark examples. An increase in the computational design effort due to cross-validation demands is the price to be paid to obtain this advantage. Nevertheless, the operational effort is usually lower than that needed by SVMs.

Development and comparison of neural network based soft sensors for online estimation of cement clinker quality

The online estimation of process outputs mostly related to quality, as opposed to their belated measurement by means of hardware measuring devices and laboratory analysis, represents the most valuable feature of soft sensors. As of now there have been very few attempts for soft sensing of cement clinker quality which is mostly done by offline laboratory analysis resulting at times in low quality clinker. In the present work three different neural network based soft sensors have been developed for online estimation of cement clinker properties. Different input and output data for a rotary cement kiln were collected from a cement plant producing 10,000 tons of clinker per day. The raw data were pre-processed to remove the outliers and the resulting missing data were imputed. The processed data were then used to develop a back propagation neural network model, a radial basis network model and a regression network model to estimate the clinker quality online. A comparison of the estimation capabilities of the three models has been done by simulation of the developed models. It was observed that radial basis network model produced better estimation capabilities than the back propagation and regression network models.

Comparison between linear and non-parametric regression models for genome-enabled prediction in wheat

In genome-enabled prediction, parametric, semi-parametric, and non-parametric regression models have been used. This study assessed the predictive ability of linear and non-linear models using dense molecular markers. The linear models were linear on marker effects and included the Bayesian LASSO, Bayesian ridge regression, Bayes A, and Bayes B. The non-linear models (this refers to non-linearity on markers) were reproducing kernel Hilbert space (RKHS) regression, Bayesian regularized neural networks (BRNN), and radial basis function neural networks (RBFNN). These statistical models were compared using 306 elite wheat lines from CIMMYT genotyped with 1717 diversity array technology (DArT) markers and two traits, days to heading (DTH) and grain yield (GY), measured in each of 12 environments. It was found that the three non-linear models had better overall prediction accuracy than the linear regression specification. Results showed a consistent superiority of RKHS and RBFNN over the Bayesian LASSO, Bayesian ridge regression, Bayes A, and Bayes B models.

On the efficiency of the orthogonal least squares training method for radial basis function networks

The efficiency of the orthogonal least squares (OLS) method for training approximation networks is examined using the criterion of energy compaction. We show that the selection of basis vectors produced by the procedure is not the most compact when the approximation is performed using a nonorthogonal basis. Hence, the algorithm does not produce the smallest possible networks for a given approximation error. Specific examples are given using the Gaussian radial basis functions type of approximation networks.

Effects of moving the center's in an RBF network

In radial basis function (RBF) networks, placement of centers is said to have a significant effect on the performance of the network. Supervised learning of center locations in some applications show that they are superior to the networks whose centers are located using unsupervised methods. But such networks can take the same training time as that of sigmoid networks. The increased time needed for supervised learning offsets the training time of regular RBF networks. One way to overcome this may be to train the network with a set of centers selected by unsupervised methods and then to fine tune the locations of centers. This can be done by first evaluating whether moving the centers would decrease the error and then, depending on the required level of accuracy, changing the center locations. This paper provides new results on bounds for the gradient and Hessian of the error considered first as a function of the independent set of parameters, namely the centers, widths, and weights; and then as a function of centers and widths where the linear weights are now functions of the basis function parameters for networks of fixed size. Moreover, bounds for the Hessian are also provided along a line beginning at the initial set of parameters. Using these bounds, it is possible to estimate how much one can reduce the error by changing the centers. Further to that, a step size can be specified to achieve a guaranteed, amount of reduction in error.

Nonlinear model structure design and construction using orthogonal least squares and D-optimality design

A very efficient learning algorithm for model subset selection is introduced based on a new composite cost function that simultaneously optimizes the model approximation ability and model robustness and adequacy. The derived model parameters are estimated via forward orthogonal least squares, but the model subset selection cost function includes a D-optimality design criterion that maximizes the determinant of the design matrix of the subset to ensure the model robustness, adequacy, and parsimony of the final model. The proposed approach is based on the forward orthogonal least square (OLS) algorithm, such that new D-optimality-based cost function is constructed based on the orthogonalization process to gain computational advantages and hence to maintain the inherent advantage of computational efficiency associated with the conventional forward OLS approach. Illustrative examples are included to demonstrate the effectiveness of the new approach.

Simplifying fuzzy rule-based models using orthogonal transformation methods

An important issue in fuzzy-rule-based modeling is how to select a set of important fuzzy rules from a given rule base. Even though it is conceivable that removal of redundant or less important fuzzy rules from the rule base can result in a compact fuzzy model with better generalizing ability, the decision as to which rules are redundant or less important is not an easy exercise. In this paper, we introduce several orthogonal transformation-based methods that provide new or alternative tools for rule selection. These methods include an orthogonal least squares (OLS) method, an eigenvalue decomposition (ED) method, a singular value decomposition and QR with column pivoting (SVD-QR) method, a total least squares (TLS) method, and a direct singular value decomposition (D-SVD) method. A common attribute of these methods is that they all work on a firing strength matrix and employ some measure index to detect the rules that should be retained and eliminated. We show the performance of these methods by applying them to solving a nonlinear plant modeling problem. Our conclusions based on analysis and simulation can be used as a guideline for choosing a proper rule selection method for a specific application.

A clustering technique for digital communications channel equalization using radial basis function networks

The application of a radial basis function network to digital communications channel equalization is examined. It is shown that the radial basis function network has an identical structure to the optimal Bayesian symbol-decision equalizer solution and, therefore, can be employed to implement the Bayesian equalizer. The training of a radial basis function network to realize the Bayesian equalization solution can be achieved efficiently using a simple and robust supervised clustering algorithm. During data transmission a decision-directed version of the clustering algorithm enables the radial basis function network to track a slowly time-varying environment. Moreover, the clustering scheme provides an automatic compensation for nonlinear channel and equipment distortion. Computer simulations are included to illustrate the analytical results.

Integrating local and global error statistics for multi-scale RBF network training: an assessment on remote sensing data

Background

This study discusses the theoretical underpinnings of a novel multi-scale radial basis function (MSRBF) neural network along with its application to classification and regression tasks in remote sensing. The novelty of the proposed MSRBF network relies on the integration of both local and global error statistics in the node selection process.

Methodology and Principal Findings

The method was tested on a binary classification task, detection of impervious surfaces using a Landsat satellite image, and a regression problem, simulation of waveform LiDAR data. In the classification scenario, results indicate that the MSRBF is superior to existing radial basis function and back propagation neural networks in terms of obtained classification accuracy and training-testing consistency, especially for smaller datasets. The latter is especially important as reference data acquisition is always an issue in remote sensing applications. In the regression case, MSRBF provided improved accuracy and consistency when contrasted with a multi kernel RBF network.

Conclusion and Significance

Results highlight the potential of a novel training methodology that is not restricted to a specific algorithmic type, therefore significantly advancing machine learning algorithms for classification and regression tasks. The MSRBF is expected to find numerous applications within and outside the remote sensing field.

Genome-enabled prediction of genetic values using radial basis function neural networks

The availability of high density panels of molecular markers has prompted the adoption of genomic selection (GS) methods in animal and plant breeding. In GS, parametric, semi-parametric and non-parametric regressions models are used for predicting quantitative traits. This article shows how to use neural networks with radial basis functions (RBFs) for prediction with dense molecular markers. We illustrate the use of the linear Bayesian LASSO regression model and of two non-linear regression models, reproducing kernel Hilbert spaces (RKHS) regression and radial basis function neural networks (RBFNN) on simulated data and real maize lines genotyped with 55,000 markers and evaluated for several trait-environment combinations. The empirical results of this study indicated that the three models showed similar overall prediction accuracy, with a slight and consistent superiority of RKHS and RBFNN over the additive Bayesian LASSO model. Results from the simulated data indicate that RKHS and RBFNN models captured epistatic effects; however, adding non-signal (redundant) predictors (interaction between markers) can adversely affect the predictive accuracy of the non-linear regression models.

Autonomous learning derived from experimental modeling of physical laws

This article deals with experimental description of physical laws by probability density function of measured data. The Gaussian mixture model specified by representative data and related probabilities is utilized for this purpose. The information cost function of the model is described in terms of information entropy by the sum of the estimation error and redundancy. A new method is proposed for searching the minimum of the cost function. The number of the resulting prototype data depends on the accuracy of measurement. Their adaptation resembles a self-organized, highly non-linear cooperation between neurons in an artificial NN. A prototype datum corresponds to the memorized content, while the related probability corresponds to the excitability of the neuron. The method does not include any free parameters except objectively determined accuracy of the measurement system and is therefore convenient for autonomous execution. Since representative data are generally less numerous than the measured ones, the method is applicable for a rather general and objective compression of overwhelming experimental data in automatic data-acquisition systems. Such compression is demonstrated on analytically determined random noise and measured traffic flow data. The flow over a day is described by a vector of 24 components. The set of 365 vectors measured over one year is compressed by autonomous learning to just 4 representative vectors and related probabilities. These vectors represent the flow in normal working days and weekends or holidays, while the related probabilities correspond to relative frequencies of these days. This example reveals that autonomous learning yields a new basis for interpretation of representative data and the optimal model structure.

A NOVEL EFFICIENT LEARNING ALGORITHM FOR SELF-GENERATING FUZZY NEURAL NETWORK WITH APPLICATIONS

In this paper, a novel efficient learning algorithm towards self-generating fuzzy neural network (SGFNN) is proposed based on ellipsoidal basis function (EBF) and is functionally equivalent to a Takagi-Sugeno-Kang (TSK) fuzzy system. The proposed algorithm is simple and efficient and is able to generate a fuzzy neural network with high accuracy and compact structure. The structure learning algorithm of the proposed SGFNN combines criteria of fuzzy-rule generation with a pruning technology. The Kalman filter (KF) algorithm is used to adjust the consequent parameters of the SGFNN. The SGFNN is employed in a wide range of applications ranging from function approximation and nonlinear system identification to chaotic time-series prediction problem and real-world fuel consumption prediction problem. Simulation results and comparative studies with other algorithms demonstrate that a more compact architecture with high performance can be obtained by the proposed algorithm. In particular, this paper presents an adaptive modeling and control scheme for drug delivery system based on the proposed SGFNN. Simulation study demonstrates the ability of the proposed approach for estimating the drug's effect and regulating blood pressure at a prescribed level.

Fast and efficient second-order method for training radial basis function networks

This paper proposes an improved second order (ISO) algorithm for training radial basis function (RBF) networks. Besides the traditional parameters, including centers, widths and output weights, the input weights on the connections between input layer and hidden layer are also adjusted during the training process. More accurate results can be obtained by increasing variable dimensions. Initial centers are chosen from training patterns and other parameters are generated randomly in limited range. Taking the advantages of fast convergence and powerful search ability of second order algorithms, the proposed ISO algorithm can normally reach smaller training/testing error with much less number of RBF units. During the computation process, quasi Hessian matrix and gradient vector are accumulated as the sum of related sub matrices and vectors, respectively. Only one Jacobian row is stored and used for multiplication, instead of the entire Jacobian matrix storage and multiplication. Memory reduction benefits the computation speed and allows the training of problems with basically unlimited number of patterns. Several practical discrete and continuous classification problems are applied to test the properties of the proposed ISO training algorithm.