ADMM-Based Algorithm for Training Fault Tolerant RBF Networks and Selecting Centers

Geodesic Basis Function Neural Network

In the learning of existing radial basis function neural networks-based methods, it is difficult to propagate errors back. This leads to an inconsistency between the learning and recognition task. This article proposes a geodesic basis function neural network with subclass extension learning (GBFNN-ScE). The geodesic basis function (GBF), which is defined here for the first time, uses the geodetic distance in the manifold as a measure to obtain the response of the sample with respect to the local center. To learn network parameters by back-propagating errors for the purpose of correct classification, a specific GBF based on a pruned gamma encoding cosine function is constructed. This function has a concise and explicit expression on the hyperspherical manifold, which is conducive to the realization of error back propagation. In the preprocessing layer, a sample unitization method with no loss of information, nonnegative unit hyperspherical crown (NUHC) mapping, is proposed. The sample can be mapped to the support set of the GBF. To alleviate the problem that one-hot encoding is not effective enough in the differential expression of data labels within a class, a subclass extension (ScE) learning strategy is proposed. The ScE learning strategy can help the learned network be more robust. For the working of GBFNN-ScE, the original sample is projected onto the support set of GBF through the NUHC mapping. Then the mapped samples are sent to the nonlinear computation units composed of GBFs in the hidden layer. Finally, the response obtained in the hidden layer is weighted by the learned weight to obtain the network output. This article theoretically proves that the separability of the data with ScE learning is stronger. The experimental results show that the proposed GBFNN-ScE has a better performance in recognition tasks than existing methods. The ablation experiments show that the ideas of the GBFNN-ScE contribute to the algorithm performance.

Cardinality Constrained Portfolio Optimization via Alternating Direction Method of Multipliers

Inspired by sparse learning, the Markowitz mean-variance model with a sparse regularization term is popularly used in sparse portfolio optimization. However, in penalty-based portfolio optimization algorithms, the cardinality level of the resultant portfolio relies on the choice of the regularization parameter. This brief formulates the mean-variance model as a cardinality ( l0 -norm) constrained nonconvex optimization problem, in which we can explicitly specify the number of assets in the portfolio. We then use the alternating direction method of multipliers (ADMMs) concept to develop an algorithm to solve the constrained nonconvex problem. Unlike some existing algorithms, the proposed algorithm can explicitly control the portfolio cardinality. In addition, the dynamic behavior of the proposed algorithm is derived. Numerical results on four real-world datasets demonstrate the superiority of our approach over several state-of-the-art algorithms.

An Accelerated Maximally Split ADMM for a Class of Generalized Ridge Regression

Ridge regression (RR) has been commonly used in machine learning, but is facing computational challenges in big data applications. To meet the challenges, this article develops a highly parallel new algorithm, i.e., an accelerated maximally split alternating direction method of multipliers (A-MS-ADMM), for a class of generalized RR (GRR) that allows different regularization factors for different regression coefficients. Linear convergence of the new algorithm along with its convergence ratio is established. Optimal parameters of the algorithm for the GRR with a particular set of regularization factors are derived, and a selection scheme of the algorithm parameters for the GRR with general regularization factors is also discussed. The new algorithm is then applied in the training of single-layer feedforward neural networks. Experimental results on performance validation on real-world benchmark datasets for regression and classification and comparisons with existing methods demonstrate the fast convergence, low computational complexity, and high parallelism of the new algorithm.

Integrated neural network model with pre-RBF kernels

To improve the network performance of radial basis function (RBF) and back-propagation (BP) networks on complex nonlinear problems, an integrated neural network model with pre-RBF kernels is proposed. The proposed method is based on the framework of a single optimized BP network and an RBF network. By integrating and connecting the RBF kernel mapping layer and BP neural network, the local features of a sample set can be effectively extracted to improve separability; subsequently, the connected BP network can be used to perform learning and classification in the kernel space. Experiments on an artificial dataset and three benchmark datasets show that the proposed model combines the advantages of RBF and BP networks, as well as improves the performances of the two networks. Finally, the effectiveness of the proposed method is verified.

A Maximally Split and Relaxed ADMM for Regularized Extreme Learning Machines

One of the salient features of the extreme learning machine (ELM) is its fast learning speed. However, in a big data environment, the ELM still suffers from an overly heavy computational load due to the high dimensionality and the large amount of data. Using the alternating direction method of multipliers (ADMM), a convex model fitting problem can be split into a set of concurrently executable subproblems, each with just a subset of model coefficients. By maximally splitting across the coefficients and incorporating a novel relaxation technique, a maximally split and relaxed ADMM (MS-RADMM), along with a scalarwise implementation, is developed for the regularized ELM (RELM). The convergence conditions and the convergence rate of the MS-RADMM are established, which exhibits linear convergence with a smaller convergence ratio than the unrelaxed maximally split ADMM. The optimal parameter values of the MS-RADMM are obtained and a fast parameter selection scheme is provided. Experiments on ten benchmark classification data sets are conducted, the results of which demonstrate the fast convergence and parallelism of the MS-RADMM. Complexity comparisons with the matrix-inversion-based method in terms of the numbers of multiplication and addition operations, the computation time and the number of memory cells are provided for performance evaluation of the MS-RADMM.