Boundedness and convergence analysis of weight elimination for cyclic training of neural networks

Learning Optimized Structure of Neural Networks by Hidden Node Pruning With L1 Regularization

We propose three different methods to determine the optimal number of hidden nodes based on L₁ regularization for a multilayer perceptron network. The first two methods, respectively, use a set of multiplier functions and multipliers for the hidden-layer nodes and implement the L₁ regularization on those, while the third method equipped with the same multipliers uses a smoothing approximation of the L₁ regularization. Each of these methods begins with a given number of hidden nodes, then the network is trained to obtain an optimal architecture discarding redundant hidden nodes using the multiplier functions or multipliers. A simple and generic method, namely, the matrix-based convergence proving method (MCPM), is introduced to prove the weak and strong convergence of the presented smoothing algorithms. The performance of the three pruning methods has been tested on 11 different classification datasets. The results demonstrate the efficient pruning abilities and competitive generalization by the proposed methods. The theoretical results are also validated by the results.