Supervised feature selection through Deep Neural Networks with pairwise connected structure

Importance estimate of features via analysis of their weight and gradient profile

Understanding what is important and redundant within data can improve the modelling process of neural networks by reducing unnecessary model complexity, training time and memory storage. This information is however not always priorly available nor trivial to obtain from neural networks. There are existing feature selection methods which utilise the internal working of a neural network for selection, however further analysis and interpretation of the input features' significance is often limiting. We propose an approach that offers an extension that estimates the significance of features by analysing the gradient descent of a pairwise layer within a model. The changes that occur with the weights and gradients throughout training provide a profile that can be used to better understand the importance hierarchy between the features for ranking and feature selection. Additionally, this method is transferable to existing fully or partially trained models, which is beneficial for understanding existing or active models. The proposed approach is demonstrated empirically with a study which uses benchmark datasets from libraries such as MNIST and scikit-feat, as well as a simulated dataset and an applied real world dataset. This is verified with the ground truth where available, and if not, via a comparison of fundamental feature selection methods, which includes existing statistical based and embedded neural network based feature selection methods through the methodology of Reduce and Retrain.

Surrogate-Assisted and Filter-Based Multiobjective Evolutionary Feature Selection for Deep Learning

Feature selection (FS) for deep learning prediction models is a difficult topic for researchers to tackle. Most of the approaches proposed in the literature consist of embedded methods through the use of hidden layers added to the neural network architecture that modify the weights of the units associated with each input attribute so that the worst attributes have less weight in the learning process. Other approaches used for deep learning are filter methods, which are independent of the learning algorithm, which can limit the precision of the prediction model. Wrapper methods are impractical with deep learning due to their high computational cost. In this article, we propose new attribute subset evaluation FS methods for deep learning of the wrapper, filter and wrapper-filter hybrid types, where multiobjective and many-objective evolutionary algorithms are used as search strategies. A novel surrogate-assisted approach is used to reduce the high computational cost of the wrapper-type objective function, while the filter-type objective functions are based on correlation and an adaptation of the reliefF algorithm. The proposed techniques have been applied in a time series forecasting problem of air quality in the Spanish south-east and an indoor temperature forecasting problem in a domotic house, with promising results compared to other FS techniques used in the literature.

Adaptive soft sensor using stacking approximate kernel based BLS for batch processes

To deal with the highly nonlinear and time-varying characteristics of Batch Process, a model named adaptive stacking approximate kernel based broad learning system is proposed in this paper. This model innovatively introduces the approximate kernel based broad learning system (AKBLS) algorithm and the Adaptive Stacking framework, giving it strong nonlinear fitting ability, excellent generalization ability, and adaptive ability. The Broad Learning System (BLS) is known for its shorter training time for effective nonlinear processing, but the uncertainty brought by its double random mapping results in poor resistance to noisy data and unpredictable impact on performance. To address this issue, this paper proposes an AKBLS algorithm that reduces uncertainty, eliminates redundant features, and improves prediction accuracy by projecting feature nodes into the kernel space. It also significantly reduces the computation time of the kernel matrix by searching for approximate kernels to enhance its ability in industrial online applications. Extensive comparative experiments on various public datasets of different sizes validate this. The Adaptive Stacking framework utilizes the Stacking ensemble learning method, which integrates predictions from multiple AKBLS models using a meta-learner to improve generalization. Additionally, by employing the moving window method-where a fixed-length window slides through the database over time-the model gains adaptive ability, allowing it to better respond to gradual changes in industrial Batch Process. Experiments on a substantial dataset of penicillin simulations demonstrate that the proposed model significantly improves predictive accuracy compared to other common algorithms.

Optimization of biocementation responses by artificial neural network and random forest in comparison to response surface methodology

In this article, the optimization of the specific urease activity (SUA) and the calcium carbonate (CaCO₃) using microbially induced calcite precipitation (MICP) was compared to optimization using three algorithms based on machine learning: random forest regressor, artificial neural networks (ANNs), and multivariate linear regression. This study applied the techniques in two existing response surface method (RSM) experiments involving MICP technique. Random forest-based models and artificial neural network-based models were submitted through the optimization of hyperparameters via cross-validation technique and grid search, to select the best-optimized model. For this study, the random forest-based algorithm is aimed at having the best performance of 0.9381 and 0.9463 in comparison to the original r² of 0.9021 and 0.8530, respectively. This study is aimed at exploring the capability of using machine learning-based models in small datasets for the purpose of optimization of experimental variables in MICP technique and the meaningfulness of the models by their specificities in the small experimental datasets applied to experimental designs. This study is aimed at exploring the capability of using machine learning-based models in small datasets for experimental variable optimization in MICP technique. The use of these techniques can create prerogatives to scale and mitigate costs in future experiments associated to the field.

Characterization of signature trends across the spectrum of non-alcoholic fatty liver disease using deep learning method

AIMS

The timely diagnosis of different stages in NAFLD is crucial for disease treatment and reversal. We used hepatocellular ballooning to determine different NAFLD stages.

MAIN METHODS

We analyzed differentially expressed genes (DEGs) in 78 patients with NAFLD and in healthy controls from previously published RNA-seq data. We identified two expression types in NAFLD progression, calculated the predictive power of candidate genes, and validated them in an independent cohort. We also performed cancer studies with these candidates retrieved from the Cancer Genome Atlas.

KEY FINDINGS

We identified 103 DEGs in NAFLD patients compared to healthy controls: 75 genes gradually increased or decreased in the NAFLD stage, whereas 28 genes showed differences only in NASH. The former were enriched in negative regulation and binding-related genes; the latter were involved in positive regulation and cell proliferation. Feature selection showed the gradual up- or down-regulation of 21 genes in NASH compared to controls; 18 were highly expressed only in NASH. Using deep-learning method with subset of features from lasso regression, we obtained reliable determination performance in NAFL and NASH (accuracy: 0.857) and validated these genes using an independent cohort (accuracy: 0.805). From cancer studies, we identified significant differential expression of several candidate genes in LIHC; 5 genes were gradually up-regulated and 6 showing high expression only in NASH were influential to patient survival.

SIGNIFICANCE

The identified biomolecular signatures may determine the spectrum of NAFLD and its relationship with HCC, improving clinical diagnosis and prognosis and enabling a therapeutic intervention for NAFLD.

Efficient Intelligent Intrusion Detection System for Heterogeneous Internet of Things (HetIoT)

Moving towards a more digital and intelligent world equipped with internet-of-thing (IoT) devices creates many security issues. A distributed denial of service (DDoS) attack is one of the most formidable and challenging security threats that has taken hold with the emergence of the heterogeneous IoT (HetIoT). The massive DDoS attacks have exhibited their impact by continuously destroying a variety of infrastructures, resulting in huge losses, and endangering the overall availability of the digital world. The emphasis of this research is to identify and mitigate various DDoS attacks for HetIoT. The research proposes an intelligent intrusion detection system (IDS) using a convolutional neural network (CNN), i.e., HetIoT-CNN IDS, a novel deep learning-based convolutional neural network for the HetIoT environment. The proposed intelligent IDS successfully identifies and mitigates various DDoS attacks in the HetIoT infrastructure. The feasibility of the new proposed HetIoT-CNN IDS is assessed by considering binary and multi-class (8- and 13-classes) classification. The performance of the proposed intelligent IDS is compared with two state-of-the-art deep learning approaches for HetIoT, and the results reveal that the proposed HetIoT-CNN IDS outperforms it. The proposed HetIoT-CNN IDS successfully identifies various DDoS attacks with an accuracy rate of 99.75% for binary classes, 99.95% for 8-classes, and 99.99% for 13-classes. The work also compares the individual accuracy of binary classes, 8-classes, and 13-classes with state-of-the-art work.

A Filter-APOSD approach for feature selection and linguistic knowledge discovery

The development of natural language processing promotes the progress of general linguistic studies. Based on the selected features and the extracted rules for word sense disambiguation (WSD), some valuable knowledge of the relations between linguistic features and word sense classes may be discovered, which may provide theoretical and practical evidence and references for lexical semantic study and natural language processing. However, many available approaches of feature selection for WSD are in the end to end operation, they can only select the optimal features for WSD, but not provide the rules for WSD, which makes knowledge discovery impossible. Therefore, a new Filter-Attribute partial ordered structure diagram (Filter-APOSD) approach is proposed in this article to fulfill both feature selection and knowledge discovery. The new approach is a combination of a Filter approach and an Attribute Partial Ordered Structure Diagram (APOSD) approach. The Filter approach is designed and used for filtering the simplest rules for WSD, and the APOSD approach is used to provide the complementary rules for WSD and visualize the structure of the datasets for knowledge discovery. The features occurring in the final rule set are selected as the optimal features. The proposed approach is verified by the benchmark data set from the SemEval-2007 preposition sense disambiguation corpus with around as the target word for WSD. The test result shows that the accuracy of WSD of around is greatly improved comparing with the one by the state of the art, and 17 out of 22 features are finally selected and ranked according to their contribution to the WSD, and some knowledge on the relations between the word senses and the selected features is discovered.

An Efficient Heap Based Optimizer Algorithm for Feature Selection

The heap-based optimizer (HBO) is an innovative meta-heuristic inspired by human social behavior. In this research, binary adaptations of the heap-based optimizer B_HBO are presented and used to determine the optimal features for classifications in wrapping form. In addition, HBO balances exploration and exploitation by employing self-adaptive parameters that can adaptively search the solution domain for the optimal solution. In the feature selection domain, the presented algorithms for the binary Heap-based optimizer B_HBO are used to find feature subsets that maximize classification performance while lowering the number of selected features. The textitk-nearest neighbor (textitk-NN) classifier ensures that the selected features are significant. The new binary methods are compared to eight common optimization methods recently employed in this field, including Ant Lion Optimization (ALO), Archimedes Optimization Algorithm (AOA), Backtracking Search Algorithm (BSA), Crow Search Algorithm (CSA), Levy flight distribution (LFD), Particle Swarm Optimization (PSO), Slime Mold Algorithm (SMA), and Tree Seed Algorithm (TSA) in terms of fitness, accuracy, precision, sensitivity, F-score, the number of selected features, and statistical tests. Twenty datasets from the UCI repository are evaluated and compared using a set of evaluation indicators. The non-parametric Wilcoxon rank-sum test was used to determine whether the proposed algorithms’ results varied statistically significantly from those of the other compared methods. The comparison analysis demonstrates that B_HBO is superior or equivalent to the other algorithms used in the literature.

ST-AL: a hybridized search based metaheuristic computational algorithm towards optimization of high dimensional industrial datasets

The rapid growth of data generated by several applications like engineering, biotechnology, energy, and others has become a crucial challenge in the high dimensional data mining. The large amounts of data, especially those with high dimensions, may contain many irrelevant, redundant, or noisy features, which may negatively affect the accuracy and efficiency of the industrial data mining process. Recently, several meta-heuristic optimization algorithms have been utilized to evolve feature selection techniques for dealing with the vast dimensionality problem. Despite optimization algorithms' ability to find the near-optimal feature subset of the search space, they still face some global optimization challenges. This paper proposes an improved version of the sooty tern optimization (ST) algorithm, namely the ST-AL method, to improve the search performance for high-dimensional industrial optimization problems. ST-AL method is developed by boosting the performance of STOA by applying four strategies. The first strategy is the use of a control randomization parameters that ensure the balance between the exploration-exploitation stages during the search process; moreover, it avoids falling into local optimums. The second strategy entails the creation of a new exploration phase based on the Ant lion (AL) algorithm. The third strategy is improving the STOA exploitation phase by modifying the main equation of position updating. Finally, the greedy selection is used to ignore the poor generated population and keeps it from diverging from the existing promising regions. To evaluate the performance of the proposed ST-AL algorithm, it has been employed as a global optimization method to discover the optimal value of ten CEC2020 benchmark functions. Also, it has been applied as a feature selection approach on 16 benchmark datasets in the UCI repository and compared with seven well-known optimization feature selection methods. The experimental results reveal the superiority of the proposed algorithm in avoiding local minima and increasing the convergence rate. The experimental result are compared with state-of-the-art algorithms, i.e., ALO, STOA, PSO, GWO, HHO, MFO, and MPA and found that the mean accuracy achieved is in range 0.94-1.00.

An investigation into the deep learning approach in sentimental analysis using graph-based theories

Sentiment analysis is a branch of natural language analytics that aims to correlate what is expressed which comes normally within unstructured format with what is believed and learnt. Several attempts have tried to address this gap (i.e., Naive Bayes, RNN, LSTM, word embedding, etc.), even though the deep learning models achieved high performance, their generative process remains a "black-box" and not fully disclosed due to the high dimensional feature and the non-deterministic weights assignment. Meanwhile, graphs are becoming more popular when modeling complex systems while being traceable and understood. Here, we reveal that a good trade-off transparency and efficiency could be achieved with a Deep Neural Network by exploring the Credit Assignment Paths theory. To this end, we propose a novel algorithm which alleviates the features' extraction mechanism and attributes an importance level of selected neurons by applying a deterministic edge/node embeddings with attention scores on the input unit and backward path respectively. We experiment on the Twitter Health News dataset were the model has been extended to approach different approximations (tweet/aspect and tweets' source levels, frequency, polarity/subjectivity), it was also transparent and traceable. Moreover, results of comparing with four recent models on same data corpus for tweets analysis showed a rapid convergence with an overall accuracy of ≈83% and 94% of correctly identified true positive sentiments. Therefore, weights can be ideally assigned to specific active features by following the proposed method. As opposite to other compared works, the inferred features are conditioned through the users' preferences (i.e., frequency degree) and via the activation's derivatives (i.e., reject feature if not scored). Future direction will address the inductive aspect of graph embeddings to include dynamic graph structures and expand the model resiliency by considering other datasets like SemEval task7, covid-19 tweets, etc.

Feature Extraction of Ancient Chinese Characters Based on Deep Convolution Neural Network and Big Data Analysis

In recent years, deep learning has made good progress and has been applied to face recognition, video monitoring, image processing, and other fields. In this big data background, deep convolution neural network has also received more and more attention. In order to extract the ancient Chinese characters effectively, the paper will discuss the structure model, pool process, and network training of deep convolution neural network and compare the algorithm with the traditional machine learning algorithm. The results show that the accuracy and recall rate of the Chinese characters in the plaque of Ming Dynasty can reach the peak, 81.38% and 81.31%, respectively. When the number of training samples increases to 50, the recognition rate of MFA is 99.72%, which is much higher than other algorithms. This shows that the algorithm based on deep convolution neural network and big data analysis has excellent performance and can effectively identify the Chinese characters under different dynasties, different sample sizes, and different interference factors, which can provide a powerful reference for the extraction of ancient Chinese characters.