An information theoretic approach to quantify the stability of feature selection and ranking algorithms

Forward propagation dropout in deep neural networks using Jensen-Shannon and random forest feature importance ranking

Dropout is a mechanism to prevent deep neural networks from overfitting and improving their generalization. Random dropout is the simplest method, where nodes are randomly terminated at each step of the training phase, which may lead to network accuracy reduction. In dynamic dropout, the importance of each node and its impact on the network performance is calculated, and the important nodes do not participate in the dropout. But the problem is that the importance of the nodes is not calculated consistently. A node may be considered less important and be dropped in one training epoch and on a batch of data before entering the next epoch, in which it may be an important node. On the other hand, calculating the importance of each unit in every training step is costly. In the proposed method, using random forest and Jensen-Shannon divergence, the importance of each node is calculated once. Then, in the forward propagation steps, the importance of the nodes is propagated and used in the dropout mechanism. This method is evaluated and compared with some previously proposed dropout approaches using two different deep neural network architectures on the MNIST, NorB, CIFAR10, CIFAR100, SVHN, and ImageNet datasets. The results suggest that the proposed method has better accuracy with fewer nodes and better generalizability. Also, the evaluations show that the approach has comparable complexity with other approaches and its convergence time is low as compared with state-of-the-art methods.

Feature Selection Based on Adaptive Particle Swarm Optimization with Leadership Learning

With the rapid development of the Internet of Things (IoT), the curse of dimensionality becomes increasingly common. Feature selection (FS) is to eliminate irrelevant and redundant features in the datasets. Particle swarm optimization (PSO) is an efficient metaheuristic algorithm that has been successfully applied to obtain the optimal feature subset with essential information in an acceptable time. However, it is easy to fall into the local optima when dealing with high-dimensional datasets due to constant parameter values and insufficient population diversity. In the paper, an FS method is proposed by utilizing adaptive PSO with leadership learning (APSOLL). An adaptive updating strategy for parameters is used to replace the constant parameters, and the leadership learning strategy is utilized to provide valid population diversity. Experimental results on 10 UCI datasets show that APSOLL has better exploration and exploitation capabilities through comparison with PSO, grey wolf optimizer (GWO), Harris hawks optimization (HHO), flower pollination algorithm (FPA), salp swarm algorithm (SSA), linear PSO (LPSO), and hybrid PSO and differential evolution (HPSO-DE). Moreover, less than 8% of features in the original datasets are selected on average, and the feature subsets are more effective in most cases compared to those generated by 6 traditional FS methods (analysis of variance (ANOVA), Chi-Squared (CHI2), Pearson, Spearman, Kendall, and Mutual Information (MI)).

An Effective Ensemble Automatic Feature Selection Method for Network Intrusion Detection

The mass of redundant and irrelevant data in network traffic brings serious challenges to intrusion detection, and feature selection can effectively remove meaningless information from the data. Most current filtered and embedded feature selection methods use a fixed threshold or ratio to determine the number of features in a subset, which requires a priori knowledge. In contrast, wrapped feature selection methods are computationally complex and time-consuming; meanwhile, individual feature selection methods have a bias in evaluating features. This work designs an ensemble-based automatic feature selection method called EAFS. Firstly, we calculate the feature importance or ranks based on individual methods, then add features to subsets sequentially by importance and evaluate subset performance comprehensively by designing an NSOM to obtain the subset with the largest NSOM value. When searching for a subset, the subset with higher accuracy is retained to lower the computational complexity by calculating the accuracy when the full set of features is used. Finally, the obtained subsets are ensembled, and by comparing the experimental results on three large-scale public datasets, the method described in this study can help in the classification, and also compared with other methods, we discover that our method outperforms other recent methods in terms of performance.