Continuously Constructive Deep Neural Networks

Traditionally, deep learning algorithms update the network weights, whereas the network architecture is chosen manually using a process of trial and error. In this paper, we propose two novel approaches that automatically update the network structure while also learning its weights. The novelty of our approach lies in our parameterization, where the depth, or additional complexity, is encapsulated continuously in the parameter space through control parameters that add additional complexity. We propose two methods. In tunnel networks, this selection is done at the level of a hidden unit, and in budding perceptrons, this is done at the level of a network layer; updating this control parameter introduces either another hidden unit or layer. We show the effectiveness of our methods on the synthetic two-spiral data and on three real data sets of MNIST, MIRFLICKR, and CIFAR, where we see that our proposed methods, with the same set of hyperparameters, can correctly adjust the network complexity to the task complexity.

Feedforward Neural Networks with a Hidden Layer Regularization Method

In this paper, we propose a group Lasso regularization term as a hidden layer regularization method for feedforward neural networks. Adding a group Lasso regularization term into the standard error function as a hidden layer regularization term is a fruitful approach to eliminate the redundant or unnecessary hidden layer neurons from the feedforward neural network structure. As a comparison, a popular Lasso regularization method is introduced into standard error function of the network. Our novel hidden layer regularization method can force a group of outgoing weights to become smaller during the training process and can eventually be removed after the training process. This means it can simplify the neural network structure and it minimizes the computational cost. Numerical simulations are provided by using K-fold cross-validation method with K = 5 to avoid overtraining and to select the best learning parameters. The numerical results show that our proposed hidden layer regularization method prunes more redundant hidden layer neurons consistently for each benchmark dataset without loss of accuracy. In contrast, the existing Lasso regularization method prunes only the redundant weights of the network, but it cannot prune any redundant hidden layer neurons.

Pruning algorithms of neural networks — a comparative study

The neural network with optimal architecture speeds up the learning process and generalizes the problem well for further knowledge extraction. As a result researchers have developed various techniques for pruning the neural networks. This paper provides a survey of existing pruning techniques that optimize the architecture of neural networks and discusses their advantages and limitations. Also the paper evaluates the effectiveness of various pruning techniques by comparing the performance of some traditional and recent pruning algorithms based on sensitivity analysis, mutual information and significance on four real datasets namely Iris, Wisconsin breast cancer, Hepatitis Domain and Pima Indian Diabetes.

DISGUISED DISCRIMINATION OF LOCALITY-BASED UNSUPERVISED DIMENSIONALITY REDUCTION

Many locality-based unsupervised dimensionality reduction (DR) algorithms have recently been proposed and demonstrated to be effective to a certain degree in some classification tasks. In this paper, we aim to show that: (1) a discriminant disposal is intentionally or unintentionally induced from the construction of locality in these unsupervised algorithms, however, such a discrimination is often inconsistent with the actual class information, so here called disguised discrimination; (2) sensitivities of these algorithms to local neighbor parameters stem from the inconsistency between the disguised discrimination and the actual class information; (3) how such inconsistency impacts the classification performance of these algorithms. The experiments on the benchmark face datasets testify our statements that are expected to provide some insight into the unsupervised leaning based on locality.

Development of artificial neural network based metamodels for inactivation of anthrax spores in ventilated spaces using computational fluid dynamics

Linear, quadratic, and artificial neural network (ANN)-based metamodels were developed for predicting the extent of anthrax spore inactivation by chlorine dioxide in a ventilated three-dimensional space over time from computational fluid dynamics model (CFD) simulation data. Dimensionless groups were developed to define the design space of the problem scenario. The Hammersley sequence sampling (HSS) method was used to determine the sampling points for the numerical experiments within the design space. A CFD model, comprised of multiple submodels, was applied to conduct the numerical experiments. Large eddy simulation (LES) with the Smagorinsky subgridscale model was applied to compute the airflow. Anthrax spores were modeled as a dispersed solid phase using the Lagrangian treatment. The disinfectant transport was calculated by solving a mass transport equation. Kinetic decay constants were included for spontaneous decay of the disinfectant and for the reaction of the disinfectant with the surfaces of the three-dimensional space. To enhance the mixing of the disinfectant with the room air, a momentum source was included in the simulation. An inactivation rate equation accounted for the reaction between the spores and the disinfectant. The ANN-based metamodels were most successful in predicting the number of viable bioaerosols remaining in an arbitrary enclosed space. Sensitivity analysis showed that the mass fraction of the disinfectant, inactivation rate constant, and contact time had the most influence on the inactivation of the spores.