Structure Learning for Deep Neural Networks Based on Multiobjective Optimization

Multigranularity Surrogate Modeling for Evolutionary Multiobjective Optimization With Expensive Constraints

Multiobjective optimization problems (MOPs) with expensive constraints pose stiff challenges to existing surrogate-assisted evolutionary algorithms (SAEAs) in a very limited computational cost, due to the fact that the number of expensive constraints for an MOP is often large. For existing SAEAs, they always approximate constraint functions in a single granularity, namely, approximating the constraint violation (CV, coarse-grained) or each constraint (fine-grained). However, the landscape of CV is often too complex to be accurately approximated by a surrogate model. Although the modeling of each constraint function may be simpler than that of CV, approximating all the constraint functions independently may result in tremendous cumulative errors and high computational costs. To address this issue, in this article, we develop a multigranularity surrogate modeling framework for evolutionary algorithms (EAs), where the approximation granularity of constraint surrogates is adaptively determined by the position of the population in the fitness landscape. Moreover, a dedicated model management strategy is also developed to reduce the impact resulting from the errors introduced by constraint surrogates and prevent the population from trapping into local optima. To evaluate the performance of the proposed framework, an implementation called K-MGSAEA is proposed, and the experimental results on a large number of test problems show that the proposed framework is superior to seven state-of-the-art competitors.

Multi-Objective Optimization Method for Signalized Intersections in Intelligent Traffic Network

Urban intersections are one of the most common sources of traffic congestion. Especially for multiple intersections, an appropriate control method should be able to regulate the traffic flow within the control area. The intersection signal-timing problem is crucial for ensuring efficient traffic operations, with the key issues being the determination of a traffic model and the design of an optimization algorithm. So, an optimization method for signalized intersections integrating a multi-objective model and an NSGAIII-DAE algorithm is established in this paper. Firstly, the multi-objective model is constructed including the usual signal control delay and traffic capacity indices. In addition, the conflict delay caused by right-turning vehicles crossing straight-going non-motor vehicles is considered and combined with the proposed algorithm, enabling the traffic model to better balance the traffic efficiency of intersections without adding infrastructure. Secondly, to address the challenges of diversity and convergence faced by the classic NSGA-III algorithm in solving traffic models with high-dimensional search spaces, a denoising autoencoder (DAE) is adopted to learn the compact representation of the original high-dimensional search space. Some genetic operations are performed in the compressed space and then mapped back to the original search space through the DAE. As a result, an appropriate balance between the local and global searching in an iteration can be achieved. To validate the proposed method, numerical experiments were conducted using actual traffic data from intersections in Jinzhou, China. The numerical results show that the signal control delay and conflict delay are significantly reduced compared with the existing algorithm, and the optimal reduction is 33.7% and 31.3%, respectively. The capacity value obtained by the proposed method in this paper is lower than that of the compared algorithm, but it is also 11.5% higher than that of the current scheme in this case. The comparisons and discussions demonstrate the effectiveness of the proposed method designed for improving the efficiency of signalized intersections.

A Survey on Evolutionary Neural Architecture Search

Deep neural networks (DNNs) have achieved great success in many applications. The architectures of DNNs play a crucial role in their performance, which is usually manually designed with rich expertise. However, such a design process is labor-intensive because of the trial-and-error process and also not easy to realize due to the rare expertise in practice. Neural architecture search (NAS) is a type of technology that can design the architectures automatically. Among different methods to realize NAS, the evolutionary computation (EC) methods have recently gained much attention and success. Unfortunately, there has not yet been a comprehensive summary of the EC-based NAS algorithms. This article reviews over 200 articles of most recent EC-based NAS methods in light of the core components, to systematically discuss their design principles and justifications on the design. Furthermore, current challenges and issues are also discussed to identify future research in this emerging field.

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review

The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.

Neuro-Evolutionary Direct Policy Search for Multiobjective Optimal Control

Direct policy search (DPS) is emerging as one of the most effective and widely applied reinforcement learning (RL) methods to design optimal control policies for multiobjective Markov decision processes (MOMDPs). Traditionally, DPS defines the control policy within a preselected functional class and searches its optimal parameterization with respect to a given set of objectives. The functional class should be tailored to the problem at hand and its selection is crucial, as it determines the search space within which solutions can be found. In MOMDPs problems, a different objective tradeoff determines a different fitness landscape, requiring a tradeoff-dynamic functional class selection. Yet, in state-of-the-art applications, the policy class is generally selected a priori and kept constant across the multidimensional objective space. In this work, we present a novel policy search routine called neuro-evolutionary multiobjective DPS (NEMODPS), which extends the DPS problem formulation to conjunctively search the policy functional class and its parameterization in a hyperspace containing policy architectures and coefficients. NEMODPS begins with a population of minimally structured approximating networks and progressively builds more sophisticated architectures by topological and parametrical mutation and crossover, and selection of the fittest individuals concerning multiple objectives. We tested NEMODPS for the problem of designing the control policy of a multipurpose water system. Numerical results show that the tradeoff-dynamic structural and parametrical policy search of NEMODPS is consistent across multiple runs, and outperforms the solutions designed via traditional DPS with predefined policy topologies.

A Gradient-Guided Evolutionary Approach to Training Deep Neural Networks

It has been widely recognized that the efficient training of neural networks (NNs) is crucial to classification performance. While a series of gradient-based approaches have been extensively developed, they are criticized for the ease of trapping into local optima and sensitivity to hyperparameters. Due to the high robustness and wide applicability, evolutionary algorithms (EAs) have been regarded as a promising alternative for training NNs in recent years. However, EAs suffer from the curse of dimensionality and are inefficient in training deep NNs (DNNs). By inheriting the advantages of both the gradient-based approaches and EAs, this article proposes a gradient-guided evolutionary approach to train DNNs. The proposed approach suggests a novel genetic operator to optimize the weights in the search space, where the search direction is determined by the gradient of weights. Moreover, the network sparsity is considered in the proposed approach, which highly reduces the network complexity and alleviates overfitting. Experimental results on single-layer NNs, deep-layer NNs, recurrent NNs, and convolutional NNs (CNNs) demonstrate the effectiveness of the proposed approach. In short, this work not only introduces a novel approach for training DNNs but also enhances the performance of EAs in solving large-scale optimization problems.

Evolving Connections in Group of Neurons for Robust Learning

Artificial neural networks inspired from the learning mechanism of the brain have achieved great successes in machine learning, especially those with deep layers. The commonly used neural networks follow the hierarchical multilayer architecture with no connections between nodes in the same layer. In this article, we propose a new group architectures for neural-network learning. In the new architecture, the neurons are assigned irregularly in a group and a neuron may connect to any neurons in the group. The connections are assigned automatically by optimizing a novel connecting structure learning probabilistic model which is established based on the principle that more relevant input and output nodes deserve a denser connection between them. In order to efficiently evolve the connections, we propose to directly model the architecture without involving weights and biases which significantly reduce the computational complexity of the objective function. The model is optimized via an improved particle swarm optimization algorithm. After the architecture is optimized, the connecting weights and biases are then determined and we find the architecture is robust to corruptions. From experiments, the proposed architecture significantly outperforms existing popular architectures on noise-corrupted images when trained only by pure images.

Artificial Neural Networks Based Optimization Techniques: A Review

In the last few years, intensive research has been done to enhance artificial intelligence (AI) using optimization techniques. In this paper, we present an extensive review of artificial neural networks (ANNs) based optimization algorithm techniques with some of the famous optimization techniques, e.g., genetic algorithm (GA), particle swarm optimization (PSO), artificial bee colony (ABC), and backtracking search algorithm (BSA) and some modern developed techniques, e.g., the lightning search algorithm (LSA) and whale optimization algorithm (WOA), and many more. The entire set of such techniques is classified as algorithms based on a population where the initial population is randomly created. Input parameters are initialized within the specified range, and they can provide optimal solutions. This paper emphasizes enhancing the neural network via optimization algorithms by manipulating its tuned parameters or training parameters to obtain the best structure network pattern to dissolve the problems in the best way. This paper includes some results for improving the ANN performance by PSO, GA, ABC, and BSA optimization techniques, respectively, to search for optimal parameters, e.g., the number of neurons in the hidden layers and learning rate. The obtained neural net is used for solving energy management problems in the virtual power plant system.

Optimization of vector convolutional deep neural network using binary real cumulative incarnation for detection of distributed denial of service attacks

In today’s technological world, distributed denial of service (DDoS) attacks threaten Internet users by flooding huge network traffic to make critical Internet services unavailable to genuine users. Therefore, design of DDoS attack detection system is on urge to mitigate these attacks for protecting the critical services. Nowadays, deep learning techniques are extensively used to detect these attacks. The existing deep feature learning approaches face the lacuna of designing an appropriate deep neural network structure for detection of DDoS attacks which leads to poor performance in terms of accuracy and false alarm. In this article, a tuned vector convolutional deep neural network (TVCDNN) is proposed by optimizing the structure and parameters of the deep neural network using binary and real cumulative incarnation (CuI), respectively. The CuI is a genetic-based optimization technique which optimizes the tuning process by providing values generated from best-fit parents. The TVCDNN is tested with publicly available benchmark network traffic datasets and compared with existing classifiers and optimization techniques. It is evident that the proposed optimization approach yields promising results compared to the existing optimization techniques. Further, the proposed approach achieves significant improvement in performance over the state-of-the-art attack detection systems.

Evolutionary Multi-Objective One-Shot Filter Pruning for Designing Lightweight Convolutional Neural Network

Deep neural networks have achieved significant development and wide applications for their amazing performance. However, their complex structure, high computation and storage resource limit their applications in mobile or embedding devices such as sensor platforms. Neural network pruning is an efficient way to design a lightweight model from a well-trained complex deep neural network. In this paper, we propose an evolutionary multi-objective one-shot filter pruning method for designing a lightweight convolutional neural network. Firstly, unlike some famous iterative pruning methods, a one-shot pruning framework only needs to perform filter pruning and model fine-tuning once. Moreover, we built a constraint multi-objective filter pruning problem in which two objectives represent the filter pruning ratio and the accuracy of the pruned convolutional neural network, respectively. A non-dominated sorting-based evolutionary multi-objective algorithm was used to solve the filter pruning problem, and it provides a set of Pareto solutions which consists of a series of different trade-off pruned models. Finally, some models are uniformly selected from the set of Pareto solutions to be fine-tuned as the output of our method. The effectiveness of our method was demonstrated in experimental studies on four designed models, LeNet and AlexNet. Our method can prune over 85%, 82%, 75%, 65%, 91% and 68% filters with little accuracy loss on four designed models, LeNet and AlexNet, respectively.

You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization

Recently neural architecture search (NAS) has raised great interest in both academia and industry. However, it remains challenging because of its huge and non-continuous search space. Instead of applying evolutionary algorithm or reinforcement learning as previous works, this paper proposes a direct sparse optimization NAS (DSO-NAS) method. The motivation behind DSO-NAS is to address the task in the view of model pruning. To achieve this goal, we start from a completely connected block, and then introduce scaling factors to scale the information flow between operations. Next, sparse regularizations are imposed to prune useless connections in the architecture. Lastly, an efficient and theoretically sound optimization method is derived to solve it. Our method enjoys both advantages of differentiability and efficiency, therefore it can be directly applied to large datasets like ImageNet and tasks beyond classification. Particularly, on the CIFAR-10 dataset, DSO-NAS achieves an average test error 2.74 percent, while on the ImageNet dataset DSO-NAS achieves 25.4 percent test error under 600M FLOPs with 8 GPUs in 18 hours. As for semantic segmentation task, DSO-NAS also achieve competitive result compared with manually designed architectures on the PASCAL VOC dataset. Code is available at https://github.com/XinbangZhang/DSO-NAS.

Solving Large-Scale Multiobjective Optimization Problems With Sparse Optimal Solutions via Unsupervised Neural Networks

Due to the curse of dimensionality of search space, it is extremely difficult for evolutionary algorithms to approximate the optimal solutions of large-scale multiobjective optimization problems (LMOPs) by using a limited budget of evaluations. If the Pareto-optimal subspace is approximated during the evolutionary process, the search space can be reduced and the difficulty encountered by evolutionary algorithms can be highly alleviated. Following the above idea, this article proposes an evolutionary algorithm to solve sparse LMOPs by learning the Pareto-optimal subspace. The proposed algorithm uses two unsupervised neural networks, a restricted Boltzmann machine, and a denoising autoencoder to learn a sparse distribution and a compact representation of the decision variables, where the combination of the learnt sparse distribution and compact representation is regarded as an approximation of the Pareto-optimal subspace. The genetic operators are conducted in the learnt subspace, and the resultant offspring solutions then can be mapped back to the original search space by the two neural networks. According to the experimental results on eight benchmark problems and eight real-world problems, the proposed algorithm can effectively solve sparse LMOPs with 10000 decision variables by only 100000 evaluations.

Evolutionary Multiobjective Optimization Driven by Generative Adversarial Networks (GANs)

Recently, increasing works have been proposed to drive evolutionary algorithms using machine-learning models. Usually, the performance of such model-based evolutionary algorithms is highly dependent on the training qualities of the adopted models. Since it usually requires a certain amount of data (i.e., the candidate solutions generated by the algorithms) for model training, the performance deteriorates rapidly with the increase of the problem scales due to the curse of dimensionality. To address this issue, we propose a multiobjective evolutionary algorithm driven by the generative adversarial networks (GANs). At each generation of the proposed algorithm, the parent solutions are first classified into real and fake samples to train the GANs; then the offspring solutions are sampled by the trained GANs. Thanks to the powerful generative ability of the GANs, our proposed algorithm is capable of generating promising offspring solutions in high-dimensional decision space with limited training data. The proposed algorithm is tested on ten benchmark problems with up to 200 decision variables. The experimental results on these test problems demonstrate the effectiveness of the proposed algorithm.

Optimization design of curved outrigger structure based on buckling analysis and multi-island genetic algorithm

In the present work, the working state of the crane leg is analyzed and discussed, and its structure is optimized. SolidWorks software is used for modeling; ANSYS software is used for finite element analysis. First of all, the constrained finite element method (CFEM) is used to analyze the linear eigenvalue buckling and geometric nonlinear buckling of outriggers with different cross-section shapes. Prove that the curved leg has certain advantages in buckling. At the same time, analyzing the leg along a different path of buckling condition and stress changes provide the basis for the design of the subsequent reinforcement. After selecting the best cross-section shape of the outrigger, the agent-based multi-island genetic algorithm is used to optimize the structural parameters of the outrigger under the transverse stiffened plate reinforced structure and the longitudinally stiffened plate reinforced structure respectively. It is proved that the outrigger with the transverse stiffened plate has a significant effect in improving the bearing capacity and in the lightweight of the structure. Finally, the gap between the movable leg and the fixed leg was changed, the stress of different gaps was analyzed by using the finite element method, and the appropriate gap value was selected according to the high-order fitting curve.

Joint Structure and Parameter Optimization of Multiobjective Sparse Neural Network

This work addresses the problem of network pruning and proposes a novel joint training method based on a multiobjective optimization model. Most of the state-of-the-art pruning methods rely on user experience for selecting the sparsity ratio of the weight matrices or tensors, and thus suffer from severe performance reduction with inappropriate user-defined parameters. Moreover, networks might be inferior due to the inefficient connecting architecture search, especially when it is highly sparse. It is revealed in this work that the network model might maintain sparse characteristic in the early stage of the backpropagation (BP) training process, and evolutionary computation-based algorithms can accurately discover the connecting architecture with satisfying network performance. In particular, we establish a multiobjective sparse model for network pruning and propose an efficient approach that combines BP training and two modified multiobjective evolutionary algorithms (MOEAs). The BP algorithm converges quickly, and the two MOEAs can search for the optimal sparse structure and refine the weights, respectively. Experiments are also included to prove the benefits of the proposed algorithm. We show that the proposed method can obtain a desired Pareto front (PF), leading to a better pruning result comparing to the state-of-the-art methods, especially when the network structure is highly sparse.

Evolving Deep Neural Networks via Cooperative Coevolution With Backpropagation

Deep neural networks (DNNs), characterized by sophisticated architectures capable of learning a hierarchy of feature representations, have achieved remarkable successes in various applications. Learning DNN's parameters is a crucial but challenging task that is commonly resolved by using gradient-based backpropagation (BP) methods. However, BP-based methods suffer from severe initialization sensitivity and proneness to getting trapped into inferior local optima. To address these issues, we propose a DNN learning framework that hybridizes CC-based optimization with BP-based gradient descent, called BPCC, and implement it by devising a computationally efficient CC-based optimization technique dedicated to DNN parameter learning. In BPCC, BP will intermittently execute for multiple training epochs. Whenever the execution of BP in a training epoch cannot sufficiently decrease the training objective function value, CC will kick in to execute by using the parameter values derived by BP as the starting point. The best parameter values obtained by CC will act as the starting point of BP in its next training epoch. In CC-based optimization, the overall parameter learning task is decomposed into many subtasks of learning a small portion of parameters. These subtasks are individually addressed in a cooperative manner. In this article, we treat neurons as basic decomposition units. Furthermore, to reduce the computational cost, we devise a maturity-based subtask selection strategy to selectively solve some subtasks of higher priority. Experimental results demonstrate the superiority of the proposed method over common-practice DNN parameter learning techniques.

Neural Networks Multiobjective Learning With Spherical Representation of Weights

This article presents a novel representation of artificial neural networks (ANNs) that is based on a projection of weights into a new spherical space defined by a radius r and a vector of angles Θ . This spherical representation of ANNs further simplifies the multiobjective learning problem, which is usually treated as a constrained optimization problem that requires great computational effort to maintain the constraints. With the proposed spherical representation, the constrained optimization problem becomes unconstrained, which simplifies the formulation and computational effort required. In addition, it also allows the use of any nonlinear optimization method for the multiobjective learning of ANNs. Results presented in this article show that the proposed spherical representation of weights yields more accurate estimates of the Pareto set than the classical multiobjective approach. Regarding the final solution selected from the Pareto set, our approach was effective and outperformed some state-of-the-art methods on several data sets.

Skip-Connected Covariance Network for Remote Sensing Scene Classification

This paper proposes a novel end-to-end learning model, called skip-connected covariance (SCCov) network, for remote sensing scene classification (RSSC). The innovative contribution of this paper is to embed two novel modules into the traditional convolutional neural network (CNN) model, i.e., skip connections and covariance pooling. The advantages of newly developed SCCov are twofold. First, by means of the skip connections, the multi-resolution feature maps produced by the CNN are combined together, which provides important benefits to address the presence of large-scale variance in RSSC data sets. Second, by using covariance pooling, we can fully exploit the second-order information contained in such multi-resolution feature maps. This allows the CNN to achieve more representative feature learning when dealing with RSSC problems. Experimental results, conducted using three large-scale benchmark data sets, demonstrate that our newly proposed SCCov network exhibits very competitive or superior classification performance when compared with the current state-of-the-art RSSC techniques, using a much lower amount of parameters. Specifically, our SCCov only needs 10% of the parameters used by its counterparts.

A Particle Swarm Optimization-Based Flexible Convolutional Autoencoder for Image Classification

Convolutional autoencoders (CAEs) have shown their remarkable performance in stacking to deep convolutional neural networks (CNNs) for classifying image data during the past several years. However, they are unable to construct the state-of-the-art CNNs due to their intrinsic architectures. In this regard, we propose a flexible CAE (FCAE) by eliminating the constraints on the numbers of convolutional layers and pooling layers from the traditional CAE. We also design an architecture discovery method by exploiting particle swarm optimization, which is capable of automatically searching for the optimal architectures of the proposed FCAE with much less computational resource and without any manual intervention. We test the proposed approach on four extensively used image classification data sets. Experimental results show that our proposed approach in this paper significantly outperforms the peer competitors including the state-of-the-art algorithms.