Convergence and objective functions of noise-injected multilayer perceptrons with hidden multipliers

Analysis on the inherent noise tolerance of feedforward network and one noise-resilient structure

In the past few decades, feedforward neural networks have gained much attraction in their hardware implementations. However, when we realize a neural network in analog circuits, the circuit-based model is sensitive to hardware nonidealities. The nonidealities, such as random offset voltage drifts and thermal noise, may lead to variation in hidden neurons and further affect neural behaviors. This paper considers that time-varying noise exists at the input of hidden neurons, with zero-mean Gaussian distribution. First, we derive lower and upper bounds on the mean square error loss to estimate the inherent noise tolerance of a noise-free trained feedforward network. Then, the lower bound is extended for any non-Gaussian noise cases based on the Gaussian mixture model concept. The upper bound is generalized for any non-zero-mean noise case. As the noise could degrade the neural performance, a new network architecture is designed to suppress the noise effect. This noise-resilient design does not require any training process. We also discuss its limitation and give a closed-form expression to describe the noise tolerance when the limitation is exceeded.

A fractional gradient descent algorithm robust to the initial weights of multilayer perceptron

For multilayer perceptron (MLP), the initial weights will significantly influence its performance. Based on the enhanced fractional derivative extend from convex optimization, this paper proposes a fractional gradient descent (RFGD) algorithm robust to the initial weights of MLP. We analyze the effectiveness of the RFGD algorithm. The convergence of the RFGD algorithm is also analyzed. The computational complexity of the RFGD algorithm is generally larger than that of the gradient descent (GD) algorithm but smaller than that of the Adam, Padam, AdaBelief, and AdaDiff algorithms. Numerical experiments show that the RFGD algorithm has strong robustness to the order of fractional calculus which is the only added parameter compared to the GD algorithm. More importantly, compared to the GD, Adam, Padam, AdaBelief, and AdaDiff algorithms, the experimental results show that the RFGD algorithm has the best robust performance for the initial weights of MLP. Meanwhile, the correctness of the theoretical analysis is verified.

Two-stage improved Grey Wolf optimization algorithm for feature selection on high-dimensional classification

In recent years, evolutionary algorithms have shown great advantages in the field of feature selection because of their simplicity and potential global search capability. However, most of the existing feature selection algorithms based on evolutionary computation are wrapper methods, which are computationally expensive, especially for high-dimensional biomedical data. To significantly reduce the computational cost, it is essential to study an effective evaluation method. In this paper, a two-stage improved gray wolf optimization (IGWO) algorithm for feature selection on high-dimensional data is proposed. In the first stage, a multilayer perceptron (MLP) network with group lasso regularization terms is first trained to construct an integer optimization problem using the proposed algorithm for pre-selection of features and optimization of the hidden layer structure. The dataset is compressed using the feature subset obtained in the first stage. In the second stage, a multilayer perceptron network with group lasso regularization terms is retrained using the compressed dataset, and the proposed algorithm is employed to construct the discrete optimization problem for feature selection. Meanwhile, a rapid evaluation strategy is constructed to mitigate the evaluation cost and improve the evaluation efficiency in the feature selection process. The effectiveness of the algorithm was analyzed on ten gene expression datasets. The experimental results show that the proposed algorithm not only removes almost more than 95.7% of the features in all datasets, but also has better classification accuracy on the test set. In addition, the advantages of the proposed algorithm in terms of time consumption, classification accuracy and feature subset size become more and more prominent as the dimensionality of the feature selection problem increases. This indicates that the proposed algorithm is particularly suitable for solving high-dimensional feature selection problems.

Comparison of Two Automated Targeted Metabolomics Programs to Manual Profiling by an Experienced Spectroscopist for 1H-NMR Spectra

Automated programs that carry out targeted metabolite identification and quantification using proton nuclear magnetic resonance spectra can overcome time and cost barriers that limit metabolomics use. However, their performance needs to be comparable to that of an experienced spectroscopist. A previously analyzed pediatric sepsis data set of serum samples was used to compare results generated by the automated programs rDolphin and BATMAN with the results obtained by manual profiling for 58 identified metabolites. Metabolites were selected using Student’s t-tests and evaluated with several performance metrics. The manual profiling results had the highest performance metrics values, especially for sensitivity (76.9%), area under the receiver operating characteristic curve (0.90), precision (62.5%), and testing accuracy based on a neural net (88.6%). All three approaches had high specificity values (77.7–86.7%). Manual profiling by an expert spectroscopist outperformed two open-source automated programs, indicating that further development is needed to achieve acceptable performance levels.

Two-stage improved Grey Wolf optimization algorithm for feature selection on high-dimensional classification

In recent years, evolutionary algorithms have shown great advantages in the field of feature selection because of their simplicity and potential global search capability. However, most of the existing feature selection algorithms based on evolutionary computation are wrapper methods, which are computationally expensive, especially for high-dimensional biomedical data. To significantly reduce the computational cost, it is essential to study an effective evaluation method. In this paper, a two-stage improved gray wolf optimization (IGWO) algorithm for feature selection on high-dimensional data is proposed. In the first stage, a multilayer perceptron (MLP) network with group lasso regularization terms is first trained to construct an integer optimization problem using the proposed algorithm for pre-selection of features and optimization of the hidden layer structure. The dataset is compressed using the feature subset obtained in the first stage. In the second stage, a multilayer perceptron network with group lasso regularization terms is retrained using the compressed dataset, and the proposed algorithm is employed to construct the discrete optimization problem for feature selection. Meanwhile, a rapid evaluation strategy is constructed to mitigate the evaluation cost and improve the evaluation efficiency in the feature selection process. The effectiveness of the algorithm was analyzed on ten gene expression datasets. The experimental results show that the proposed algorithm not only removes almost more than 95.7% of the features in all datasets, but also has better classification accuracy on the test set. In addition, the advantages of the proposed algorithm in terms of time consumption, classification accuracy and feature subset size become more and more prominent as the dimensionality of the feature selection problem increases. This indicates that the proposed algorithm is particularly suitable for solving high-dimensional feature selection problems.