diffGrad: An Optimization Method for Convolutional Neural Networks

Understanding Short-Range Memory Effects in Deep Neural Networks

Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success of SGD is ascribed to the stochastic gradient noise (SGN) incurred in the training process. Based on this consensus, SGD is frequently treated and analyzed as the Euler-Maruyama discretization of stochastic differential equations (SDEs) driven by either Brownian or Lévy stable motion. In this study, we argue that SGN is neither Gaussian nor Lévy stable. Instead, inspired by the short-range correlation emerging in the SGN series, we propose that SGD can be viewed as a discretization of an SDE driven by fractional Brownian motion (FBM). Accordingly, the different convergence behavior of SGD dynamics is well-grounded. Moreover, the first passage time of an SDE driven by FBM is approximately derived. The result suggests a lower escaping rate for a larger Hurst parameter, and thus, SGD stays longer in flat minima. This happens to coincide with the well-known phenomenon that SGD favors flat minima that generalize well. Extensive experiments are conducted to validate our conjecture, and it is demonstrated that short-range memory effects persist across various model architectures, datasets, and training strategies. Our study opens up a new perspective and may contribute to a better understanding of SGD.

Structural prior-driven feature extraction with gradient-momentum combined optimization for convolutional neural network image classification

Recent image classification efforts have achieved certain success by incorporating prior information such as labels and logical rules to learn discriminative features. However, these methods overlook the variability of features, resulting in feature inconsistency and fluctuations in model parameter updates, which further contribute to decreased image classification accuracy and model instability. To address this issue, this paper proposes a novel method combining structural prior-driven feature extraction with gradient-momentum (SPGM), from the perspectives of consistent feature learning and precise parameter updates, to enhance the accuracy and stability of image classification. Specifically, SPGM leverages a structural prior-driven feature extraction (SPFE) approach to calculate gradients of multi-level features and original images to construct structural information, which is then transformed into prior knowledge to drive the network to learn features consistent with the original images. Additionally, an optimization strategy integrating gradients and momentum (GMO) is introduced, dynamically adjusting the direction and step size of parameter updates based on the angle and norm of the sum of gradients and momentum, enabling precise model parameter updates. Extensive experiments on CIFAR10 and CIFAR100 datasets demonstrate that the SPGM method significantly reduces the top-1 error rate in image classification, enhances the classification performance, and outperforms state-of-the-art methods.

Weight Decay With Tailored Adam on Scale-Invariant Weights for Better Generalization

Weight decay (WD) is a fundamental and practical regularization technique in improving generalization of current deep learning models. However, it is observed that the WD does not work effectively for an adaptive optimization algorithm (such as Adam), as it works for SGD. Specifically, the solution found by Adam with the WD often generalizes unsatisfactorily. Though efforts have been made to mitigate this issue, the reason for such deficiency is still vague. In this article, we first show that when using the Adam optimizer, the weight norm increases very fast along with the training procedure, which is in contrast to SGD where the weight norm increases relatively slower and tends to converge. The fast increase of weight norm is adverse to WD; in consequence, the Adam optimizer will lose efficacy in finding solution that generalizes well. To resolve this problem, we propose to tailor Adam by introducing a regularization term on the adaptive learning rate, such that it is friendly to WD. Meanwhile, we introduce first moment on the WD to further enhance the regularization effect. We show that the proposed method is able to find solution with small norm and generalizes better than SGD. We test the proposed method on general image classification and fine-grained image classification tasks with different networks. Experimental results on all these cases substantiate the effectiveness of the proposed method in help improving the generalization. Specifically, the proposed method improves the test accuracy of Adam by a large margin and even improves the performance of SGD by 0.84% on CIFAR 10 and 1.03 % on CIFAR 100 with ResNet-50. The code of this article is public available at xxx.

Discriminative Suprasphere Embedding for Fine-Grained Visual Categorization

Despite the great success of the existing work in fine-grained visual categorization (FGVC), there are still several unsolved challenges, e.g., poor interpretation and vagueness contribution. To circumvent this drawback, motivated by the hypersphere embedding method, we propose a discriminative suprasphere embedding (DSE) framework, which can provide intuitive geometric interpretation and effectively extract discriminative features. Specifically, DSE consists of three modules. The first module is a suprasphere embedding (SE) block, which learns discriminative information by emphasizing weight and phase. The second module is a phase activation map (PAM) used to analyze the contribution of local descriptors to the suprasphere feature representation, which uniformly highlights the object region and exhibits remarkable object localization capability. The last module is a class contribution map (CCM), which quantitatively analyzes the network classification decision and provides insight into the domain knowledge about classified objects. Comprehensive experiments on three benchmark datasets demonstrate the effectiveness of our proposed method in comparison with state-of-the-art methods.

A novel deep learning technique for medical image analysis using improved optimizer

Application of Convolutional neural network in spectrum of Medical image analysis are providing benchmark outputs which converges the interest of many researchers to explore it in depth. Latest preprocessing technique Real ESRGAN (Enhanced super resolution generative adversarial network) and GFPGAN (Generative facial prior GAN) are proving their efficacy in providing high resolution dataset. Objective: Optimizer plays a vital role in upgrading the functioning of CNN model. Different optimizers like Gradient descent, Stochastic Gradient descent, Adagrad, Adadelta and Adam etc. are used for classification and segmentation of Medical image but they suffer from slow processing due to their large memory requirement. Stochastic Gradient descent suffers from high variance and is computationally expensive. Dead neuron problem also proves to detrimental to the performance of most of the optimizers. A new optimization technique Gradient Centralization is providing the unparalleled result in terms of generalization and execution time. Method: Our paper explores the next factor which is the employment of new optimization technique, Gradient centralization (GC) to our integrated framework (Model with advanced preprocessing technique). Result and conclusion: Integrated Framework of Real ESRGAN and GFPGAN with Gradient centralization provides an optimal solution for deep learning models in terms of Execution time and Loss factor improvement.

Ballistic Coefficient Calculation Based on Optical Angle Measurements of Space Debris

Atmospheric drag is an important factor affecting orbit determination and prediction of low-orbit space debris. To obtain accurate ballistic coefficients of space debris, we propose a calculation method based on measured optical angles. Angle measurements of space debris with a perigee height below 1400 km acquired from a photoelectric array were used for orbit determination. Perturbation equations of atmospheric drag were used to calculate the semi-major-axis variation. The ballistic coefficients of space debris were estimated and compared with those published by the North American Aerospace Defense Command in terms of orbit prediction error. The 48 h orbit prediction error of the ballistic coefficients obtained from the proposed method is reduced by 18.65% compared with the published error. Hence, our method seems suitable for calculating space debris ballistic coefficients and supporting related practical applications.

Deep residual architectures and ensemble learning for efficient brain tumour classification

The prompt and accurate detection of brain tumours is essential for disease management and life‐saving. This paper introduces an efficient and robust completely automated system for classifying the three prominent types of brain tumour. The aim is to contribute for enhanced classification accuracy with minimum pre‐processing and less inference time. The power of deep networks is thoroughly investigated, with and without transfer learning. Fine‐tuned deep Residual Networks (ResNets) with depth up to 101 are introduced to manage the complex nature of brain images, and to capture their microstructural information. The proposed residual architectures with their in‐depth representations are evaluated and compared to other fine‐tuned networks (AlexNet, GoogLeNet and VGG16). A novel Convolutional Network (ConvNet) built and trained from scratch is also proposed for tumour type classification. Proven models are integrated by combining their decisions using majority voting to obtain the final classification accuracy. Results show that the residual architectures can be optimized efficiently, and a noticeable accuracy can be gained with them. Although ResNet models are deeper than VGG16, they show lower complexity. Results also indicate that building ensemble of models is a successful strategy to enhance the system performance. Each model in the ensemble learns specific patterns with certain filters. This stochastic nature boosts the classification accuracy. The accuracies obtained from ResNet18, ResNet101, and the proposed ConvNet are 98.91%, 97.39% and 95.43%, respectively. The accuracy based on decision fusion for the three networks is 99.57%, which is better than those of all state‐of‐the‐art techniques. The accuracy obtained with ResNet50 is 98.26%, and its fusion with ResNet18 and the designed network yields a 99.35% accuracy, which is also better than those of previous methods, meanwhile achieving minimum detection time requirements. Finally, visual representation of the learned features is provided to understand what the models have learned.

Detection of Multiple Respiration Patterns Based on 1D SNN from Continuous Human Breathing Signals and the Range Classification Method for Each Respiration Pattern

Human respiratory information is being used as an important source of biometric information that can enable the analysis of health status in the healthcare domain. The analysis of the frequency or duration of a specific respiration pattern and the classification of respiration patterns in the corresponding section for a certain period of time are important for the utilization of respiratory information in various ways. Existing methods require window slide processing to classify sections for each respiration pattern from the breathing data for a certain time period. In this case, when multiple respiration patterns exist within one window, the recognition rate can be lowered. To solve this problem, a 1D Siamese neural network (SNN)-based human respiration pattern detection model and a merge-and-split algorithm for the classification of multiple respiration patterns in each region for all respiration sections are proposed in this study. When calculating the accuracy based on intersection over union (IOU) for the respiration range classification result for each pattern, the accuracy was found to be improved by approximately 19.3% compared with the existing deep neural network (DNN) and 12.4% compared with a 1D convolutional neural network (CNN). The accuracy of detection based on the simple respiration pattern was approximately 14.5% higher than that of the DNN and 5.3% higher than that of the 1D CNN.

Application of big data classification effects based on neural network in video English course and relevant optimization suggestions

Due to the improvement of Internet technology and information technology, more and more students hope to learn and consolidate knowledge through video in the classroom. Teachers are more accustomed to using video in the classroom to improve and improve their teaching quality. In the current English class, teachers and students are more accustomed to using video English for teaching. English teaching videos are informative, intuitive and efficient. Through video teaching, we can make the classroom atmosphere more interesting, thus simplifying complex problems. In this context, this paper analyzes how neural networks can improve the application effect of English video courses in the context of big data, optimizes the PDCNO algorithm by using the neural network principle, and then discusses the impact of the optimized PDCNO algorithm on classification and system performance. This improves the accuracy of English video, reduces the execution time of the algorithm and reduces the memory occupation. Compared with ordinary video, the training time required under the same training parameters is shorter, and the convergence speed of the model itself will be faster. From the students' attitude towards video teaching, we can see that students prefer video English teaching, which also reflects the effectiveness of neural network big data in English video teaching. This paper introduces the neural network and big data technology into the video English course to improve the teaching effectiveness.

Predicting medical device failure: a promise to reduce healthcare facilities cost through smart healthcare management

Background

The advancement of biomedical research generates myriad healthcare-relevant data, including medical records and medical device maintenance information. The COVID-19 pandemic significantly affects the global mortality rate, creating an enormous demand for medical devices. As information technology has advanced, the concept of intelligent healthcare has steadily gained prominence. Smart healthcare utilises a new generation of information technologies, such as the Internet of Things (loT), big data, cloud computing, and artificial intelligence, to completely transform the traditional medical system. With the intention of presenting the concept of smart healthcare, a predictive model is proposed to predict medical device failure for intelligent management of healthcare services.

Methods

Present healthcare device management can be improved by proposing a predictive machine learning model that prognosticates the tendency of medical device failures toward smart healthcare. The predictive model is developed based on 8,294 critical medical devices from 44 different types of equipment extracted from 15 healthcare facilities in Malaysia. The model classifies the device into three classes; (i) class 1, where the device is unlikely to fail within the first 3 years of purchase, (ii) class 2, where the device is likely to fail within 3 years from purchase date, and (iii) class 3 where the device is likely to fail more than 3 years after purchase. The goal is to establish a precise maintenance schedule and reduce maintenance and resource costs based on the time to the first failure event. A machine learning and deep learning technique were compared, and the best robust model for smart healthcare was proposed.

Results

This study compares five algorithms in machine learning and three optimizers in deep learning techniques. The best optimized predictive model is based on ensemble classifier and SGDM optimizer, respectively. An ensemble classifier model produces 77.90%, 87.60%, and 75.39% for accuracy, specificity, and precision compared to 70.30%, 83.71%, and 67.15% for deep learning models. The ensemble classifier model improves to 79.50%, 88.36%, and 77.43% for accuracy, specificity, and precision after significant features are identified. The result concludes although machine learning has better accuracy than deep learning, more training time is required, which is 11.49 min instead of 1 min 5 s when deep learning is applied. The model accuracy shall be improved by introducing unstructured data from maintenance notes and is considered the author's future work because dealing with text data is time-consuming. The proposed model has proven to improve the devices' maintenance strategy with a Malaysian Ringgit (MYR) cost reduction of approximately MYR 326,330.88 per year. Therefore, the maintenance cost would drastically decrease if this smart predictive model is included in the healthcare management system.

An Improved Adam Optimization Algorithm Combining Adaptive Coefficients and Composite Gradients Based on Randomized Block Coordinate Descent

An improved Adam optimization algorithm combining adaptive coefficients and composite gradients based on randomized block coordinate descent is proposed to address issues of the Adam algorithm such as slow convergence, the tendency to miss the global optimal solution, and the ineffectiveness of processing high-dimensional vectors. The adaptive coefficient is used to adjust the gradient deviation value and correct the search direction firstly. Then, the predicted gradient is introduced, and the current gradient and the first-order momentum are combined to form a composite gradient to improve the global optimization ability. Finally, the random block coordinate method is used to determine the gradient update mode, which reduces the computational overhead. Simulation experiments on two standard datasets for classification show that the convergence speed and accuracy of the proposed algorithm are higher than those of the six gradient descent methods, and the CPU and memory utilization are significantly reduced. In addition, based on logging data, the BP neural networks optimized by six algorithms, respectively, are used to predict reservoir porosity. Results show that the proposed method has lower system overhead, higher accuracy, and stronger stability, and the absolute error of more than 86% data is within 0.1%, which further verifies its effectiveness.

Ensemble of Networks for Multilabel Classification

Multilabel learning goes beyond standard supervised learning models by associating a sample with more than one class label. Among the many techniques developed in the last decade to handle multilabel learning best approaches are those harnessing the power of ensembles and deep learners. This work proposes merging both methods by combining a set of gated recurrent units, temporal convolutional neural networks, and long short-term memory networks trained with variants of the Adam optimization approach. We examine many Adam variants, each fundamentally based on the difference between present and past gradients, with step size adjusted for each parameter. We also combine Incorporating Multiple Clustering Centers and a bootstrap-aggregated decision trees ensemble, which is shown to further boost classification performance. In addition, we provide an ablation study for assessing the performance improvement that each module of our ensemble produces. Multiple experiments on a large set of datasets representing a wide variety of multilabel tasks demonstrate the robustness of our best ensemble, which is shown to outperform the state-of-the-art.

An open-set framework for underwater image classification using autoencoders

In this paper, we mainly intend to address the underwater image classification problem in an open-set scenario. Image classification algorithms have been mostly provided with a small set of species, while there exist lots of species not available to the algorithms or even unknown to ourselves. Thus, we deal with an open-set problem and extremely high false alarm rate in real scenarios, especially in the case of unseen species. Motivated by these challenges, our proposed scheme aims to prevent the unseen species from going to the classifier section. To this end, we introduce a new framework based on convolutional neural networks (CNNs) that automatically identifies various species of fishes and then classifies them into certain classes using a novel technique. In the proposed method, an autoencoder is employed to distinguish between seen and unseen species. To clarify, the autoencoder is trained to reconstruct the available species with high accuracy and filter out species that are not in our training set. In the following, a classifier based on EfficientNet is trained to classify the samples that are accepted by the autoencoder (AE), i.e. the samples that have small reconstruction error. Our proposed method is evaluated in terms of precision, recall, and accuracy and compared to the state-of-the-art methods utilizing WildFish dataset. Simulation results reveal the supremacy of the proposed method.

Epoch-Evolving Gaussian Process Guided Learning for Classification

The conventional mini-batch gradient descent algorithms are usually trapped in the local batch-level distribution information, resulting in the ``zig-zag'' effect in the learning process. To characterize the correlation information between the batch-level distribution and the global data distribution, we propose a novel learning scheme called epoch-evolving Gaussian process guided learning (GPGL) to encode the global data distribution information in a non-parametric way. Upon a set of class-aware anchor samples, our GP model is built to estimate the class distribution for each sample in mini-batch through label propagation from the anchor samples to the batch samples. The class distribution, also named the context label, is provided as a complement for the ground-truth one-hot label. Such a class distribution structure has a smooth property and usually carries a rich body of contextual information that is capable of speeding up the convergence process. With the guidance of the context label and ground-truth label, the GPGL scheme provides a more efficient optimization through updating the model parameters with a triangle consistency loss. Furthermore, our GPGL scheme can be generalized and naturally applied to the current deep models, outperforming the state-of-the-art optimization methods on six benchmark datasets.

Robust Spatial-Spectral Squeeze-Excitation AdaBound Dense Network (SE-AB-Densenet) for Hyperspectral Image Classification

Increasing importance in the field of artificial intelligence has led to huge progress in remote sensing. Deep learning approaches have made tremendous progress in hyperspectral image (HSI) classification. However, the complexity in classifying the HSI data using a common convolutional neural network is still a challenge. Further, the network architecture becomes more complex when different spatial-spectral feature information is extracted. Usually, CNN has a large number of trainable parameters, which increases the computational complexity of HSI data. In this paper, an optimized squeeze-excitation AdaBound dense network (SE-AB-DenseNet) is designed to emphasize the significant spatial-spectral features of HSI data. The dense network is combined with the AdaBound and squeeze-excitation modules to give lower computation costs and better classification performance. The AdaBound optimizer gives the proposed model the ability to improve its stability and enhance its classification accuracy by approximately 2%. Additionally, the cutout regularization technique is used for HSI spatial-spectral classification to overcome the problem of overfitting. The experiments were carried out on two commonly used hyperspectral datasets (Indian Pines and Salinas). The experiment results on the datasets show a competitive classification accuracy when compared with state-of-the-art methods with limited training samples. From the SE-AB-DenseNet with the cutout model, the overall accuracies for the Indian Pines and Salinas datasets were observed to be 99.37 and 99.78, respectively.

PFmulDL: a novel strategy enabling multi-class and multi-label protein function annotation by integrating diverse deep learning methods

Bioinformatic annotation of protein function is essential but extremely sophisticated, which asks for extensive efforts to develop effective prediction method. However, the existing methods tend to amplify the representativeness of the families with large number of proteins by misclassifying the proteins in the families with small number of proteins. That is to say, the ability of the existing methods to annotate proteins in the 'rare classes' remains limited. Herein, a new protein function annotation strategy, PFmulDL, integrating multiple deep learning methods, was thus constructed. First, the recurrent neural network was integrated, for the first time, with the convolutional neural network to facilitate the function annotation. Second, a transfer learning method was introduced to the model construction for further improving the prediction performances. Third, based on the latest data of Gene Ontology, the newly constructed model could annotate the largest number of protein families comparing with the existing methods. Finally, this newly constructed model was found capable of significantly elevating the prediction performance for the 'rare classes' without sacrificing that for the 'major classes'. All in all, due to the emerging requirements on improving the prediction performance for the proteins in 'rare classes', this new strategy would become an essential complement to the existing methods for protein function prediction. All the models and source codes are freely available and open to all users at: https://github.com/idrblab/PFmulDL.

A Rectal CT Tumor Segmentation Method Based on Improved U-Net

Automatic and accurate segmentation of tumor area from rectal CT image plays an extremely key role in the treatment and diagnosis of rectal cancer. This paper proposes the MR-U-Net network model. The improvement is that a pair of encoder and decoder is added longitudinally to the U-shaped structure, which is the network structure of the fifth layer, and a residual module is added horizontally to the encoder and decoder of each layer. This model is used to conduct targeted research on the automatic segmentation method of rectal cancer. [H. Gao et al., Rectal tumor segmentation method based on U-Net improved model, J. Comput. Appl.40(8) (2020) 2392–2397] also improved U-Net and used the same dataset as this paper, but the Dice coefficient of all targets was only 83.15%, and the Dice coefficient of small targets was only 87.17%. This paper evaluates the improved MR-U-Net network model with the three indicators of precision, recall and Dice coefficient, and finds that in comparison to Ref. 4 the precision is 95.13%, 2.29% higher than the former work, recall is 94.28%, higher than the former work by 0.34%, Dice coefficient of all targets is 88.45%, increased by 5.3% compared with the former work, and the small targets Dice coefficient is increased by 1.28%, which is the best optimization state of this paper. Experiments show that for datasets with extremely skewed positive and negative samples, the MR-U-Net network structure after improving the hyperparameters in the optimizer can more accurately segment the rectal CT tumor lesion area.

Robust Stochastic Gradient Descent With Student-t Distribution Based First-Order Momentum

Remarkable achievements by deep neural networks stand on the development of excellent stochastic gradient descent methods. Deep-learning-based machine learning algorithms, however, have to find patterns between observations and supervised signals, even though they may include some noise that hides the true relationship between them, more or less especially in the robotics domain. To perform well even with such noise, we expect them to be able to detect outliers and discard them when needed. We, therefore, propose a new stochastic gradient optimization method, whose robustness is directly built in the algorithm, using the robust student-t distribution as its core idea. We integrate our method to some of the latest stochastic gradient algorithms, and in particular, Adam, the popular optimizer, is modified through our method. The resultant algorithm, called t-Adam, along with the other stochastic gradient methods integrated with our core idea is shown to effectively outperform Adam and their original versions in terms of robustness against noise on diverse tasks, ranging from regression and classification to reinforcement learning problems.

Use of machine learning in osteoarthritis research: a systematic literature review

OBJECTIVE

The aim of this systematic literature review was to provide a comprehensive and exhaustive overview of the use of machine learning (ML) in the clinical care of osteoarthritis (OA).

METHODS

A systematic literature review was performed in July 2021 using MEDLINE PubMed with key words and MeSH terms. For each selected article, the number of patients, ML algorithms used, type of data analysed, validation methods and data availability were collected.

RESULTS

From 1148 screened articles, 46 were selected and analysed; most were published after 2017. Twelve articles were related to diagnosis, 7 to prediction, 4 to phenotyping, 12 to severity and 11 to progression. The number of patients included ranged from 18 to 5749. Overall, 35% of the articles described the use of deep learning And 74% imaging analyses. A total of 85% of the articles involved knee OA and 15% hip OA. No study investigated hand OA. Most of the studies involved the same cohort, with data from the OA initiative described in 46% of the articles and the MOST and Cohort Hip and Cohort Knee cohorts in 11% and 7%. Data and source codes were described as publicly available respectively in 54% and 22% of the articles. External validation was provided in only 7% of the articles.

CONCLUSION

This review proposes an up-to-date overview of ML approaches used in clinical OA research and will help to enhance its application in this field.

Automatic Segmentation of Clinical Target Volume and Organs-at-Risk for Breast Conservative Radiotherapy Using a Convolutional Neural Network

Objective

Delineation of clinical target volume (CTV) and organs at risk (OARs) is important for radiotherapy but is time-consuming. We trained and evaluated a U-ResNet model to provide fast and consistent auto-segmentation.

Methods

We collected 160 patients’ CT scans with breast cancer who underwent breast-conserving surgery (BCS) and were treated with radiotherapy. CTV and OARs were delineated manually and were used for model training. The dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (95HD) were used to assess the performance of our model. CTV and OARs were randomly selected as ground truth (GT) masks, and artificial intelligence (AI) masks were generated by the proposed model. Two clinicians randomly compared CTV score differences of the contour. The consistency between two clinicians was tested. Time cost for auto-delineation was evaluated.

Results

The mean DSC values of the proposed method were 0.94, 0.95, 0.94, 0.96, 0.96 and 0.93 for breast CTV, contralateral breast, heart, right lung, left lung and spinal cord, respectively. The mean 95HD values were 4.31mm, 3.59mm, 4.86mm, 3.18mm, 2.79mm and 4.37mm for the above structures, respectively. The average CTV scores for AI and GT were 2.89 versus 2.92 when evaluated by oncologist A (P=0.612), and 2.75 versus 2.83 by oncologist B (P=0.213), with no statistically significant differences. The consistency between two clinicians was poor (kappa=0.282). The time for auto-segmentation of CTV and OARs was 10.03 s.

Conclusion

Our proposed model (U-ResNet) can improve the efficiency and accuracy of delineation compared with U-Net, performing equally well with the segmentation generated by oncologists.

Artificial Neural Networks Based Optimization Techniques: A Review

In the last few years, intensive research has been done to enhance artificial intelligence (AI) using optimization techniques. In this paper, we present an extensive review of artificial neural networks (ANNs) based optimization algorithm techniques with some of the famous optimization techniques, e.g., genetic algorithm (GA), particle swarm optimization (PSO), artificial bee colony (ABC), and backtracking search algorithm (BSA) and some modern developed techniques, e.g., the lightning search algorithm (LSA) and whale optimization algorithm (WOA), and many more. The entire set of such techniques is classified as algorithms based on a population where the initial population is randomly created. Input parameters are initialized within the specified range, and they can provide optimal solutions. This paper emphasizes enhancing the neural network via optimization algorithms by manipulating its tuned parameters or training parameters to obtain the best structure network pattern to dissolve the problems in the best way. This paper includes some results for improving the ANN performance by PSO, GA, ABC, and BSA optimization techniques, respectively, to search for optimal parameters, e.g., the number of neurons in the hidden layers and learning rate. The obtained neural net is used for solving energy management problems in the virtual power plant system.

Closing the Performance Gap between Siamese Networks for Dissimilarity Image Classification and Convolutional Neural Networks

In this paper, we examine two strategies for boosting the performance of ensembles of Siamese networks (SNNs) for image classification using two loss functions (Triplet and Binary Cross Entropy) and two methods for building the dissimilarity spaces (FULLY and DEEPER). With FULLY, the distance between a pattern and a prototype is calculated by comparing two images using the fully connected layer of the Siamese network. With DEEPER, each pattern is described using a deeper layer combined with dimensionality reduction. The basic design of the SNNs takes advantage of supervised k-means clustering for building the dissimilarity spaces that train a set of support vector machines, which are then combined by sum rule for a final decision. The robustness and versatility of this approach are demonstrated on several cross-domain image data sets, including a portrait data set, two bioimage and two animal vocalization data sets. Results show that the strategies employed in this work to increase the performance of dissimilarity image classification using SNN are closing the gap with standalone CNNs. Moreover, when our best system is combined with an ensemble of CNNs, the resulting performance is superior to an ensemble of CNNs, demonstrating that our new strategy is extracting additional information.

Hybrid Spatial–Temporal Graph Convolutional Networks for On-Street Parking Availability Prediction

With the development of sensors and of the Internet of Things (IoT), smart cities can provide people with a variety of information for a more convenient life. Effective on-street parking availability prediction can improve parking efficiency and, at times, alleviate city congestion. Conventional methods of parking availability prediction often do not consider the spatial–temporal features of parking duration distributions. To this end, we propose a parking space prediction scheme called the hybrid spatial–temporal graph convolution networks (HST-GCNs). We use graph convolutional networks and gated linear units (GLUs) with a 1D convolutional neural network to obtain the spatial features and the temporal features, respectively. Then, we construct a spatial–temporal convolutional block to obtain the instantaneous spatial–temporal correlations. Based on the similarity of the parking duration distributions, we propose an attention mechanism called distAtt to measure the similarity of parking duration distributions. Through the distAtt mechanism, we add the long-term spatial–temporal correlations to our spatial–temporal convolutional block, and thus, we can capture complex hybrid spatial–temporal correlations to achieve a higher accuracy of parking availability prediction. Based on real-world datasets, we compare the proposed scheme with the benchmark models. The experimental results show that the proposed scheme has the best performance in predicting the parking occupancy rate.

HyAdamC: A New Adam-Based Hybrid Optimization Algorithm for Convolution Neural Networks

As the performance of devices that conduct large-scale computations has been rapidly improved, various deep learning models have been successfully utilized in various applications. Particularly, convolution neural networks (CNN) have shown remarkable performance in image processing tasks such as image classification and segmentation. Accordingly, more stable and robust optimization methods are required to effectively train them. However, the traditional optimizers used in deep learning still have unsatisfactory training performance for the models with many layers and weights. Accordingly, in this paper, we propose a new Adam-based hybrid optimization method called HyAdamC for training CNNs effectively. HyAdamC uses three new velocity control functions to adjust its search strength carefully in term of initial, short, and long-term velocities. Moreover, HyAdamC utilizes an adaptive coefficient computation method to prevent that a search direction determined by the first momentum is distorted by any outlier gradients. Then, these are combined into one hybrid method. In our experiments, HyAdamC showed not only notable test accuracies but also significantly stable and robust optimization abilities when training various CNN models. Furthermore, we also found that HyAdamC could be applied into not only image classification and image segmentation tasks.

Gradient-Sensitive Optimization for Convolutional Neural Networks

Convolutional neural networks (CNNs) are effective models for image classification and recognition. Gradient descent optimization (GD) is the basic algorithm for CNN model optimization. Since GD appeared, a series of improved algorithms have been derived. Among these algorithms, adaptive moment estimation (Adam) has been widely recognized. However, local changes are ignored in Adam to some extent. In this paper, we introduce an adaptive learning rate factor based on current and recent gradients. According to this factor, we can dynamically adjust the learning rate of each independent parameter to adaptively adjust the global convergence process. We use the factor to adjust the learning rate for each parameter. The convergence of the proposed algorithm is proven by using the regret bound approach of the online learning framework. In the experimental section, comparisons are conducted between the proposed algorithm and other existing algorithms, such as AdaGrad, RMSprop, Adam, diffGrad, and AdaHMG, on test functions and the MNIST dataset. The results show that Adam and RMSprop combined with our algorithm can not only find the global minimum faster in the experiment using the test function but also have a better convergence curve and higher test set accuracy in experiments using datasets. Our algorithm is a supplement to the existing gradient descent algorithms, which can be combined with many other existing gradient descent algorithms to improve the efficiency of iteration, speed up the convergence of the cost function, and improve the final recognition rate.

Age-group determination of living individuals using first molar images based on artificial intelligence

Dental age estimation of living individuals is difficult and challenging, and there is no consensus method in adults with permanent dentition. Thus, we aimed to provide an accurate and robust artificial intelligence (AI)-based diagnostic system for age-group estimation by incorporating a convolutional neural network (CNN) using dental X-ray image patches of the first molars extracted via panoramic radiography. The data set consisted of four first molar images from the right and left sides of the maxilla and mandible of each of 1586 individuals across all age groups, which were extracted from their panoramic radiographs. The accuracy of the tooth-wise estimation was 89.05 to 90.27%. Performance accuracy was evaluated mainly using a majority voting system and area under curve (AUC) scores. The AUC scores ranged from 0.94 to 0.98 for all age groups, which indicates outstanding capacity. The learned features of CNNs were visualized as a heatmap, and revealed that CNNs focus on differentiated anatomical parameters, including tooth pulp, alveolar bone level, or interdental space, depending on the age and location of the tooth. With this, we provided a deeper understanding of the most informative regions distinguished by age groups. The prediction accuracy and heat map analyses support that this AI-based age-group determination model is plausible and useful.

Development of Novel Artificial Intelligence to Detect the Presence of Clinically Meaningful Coronary Atherosclerotic Stenosis in Major Branch from Coronary Angiography Video

Aim: The clinically meaningful coronary stenosis is diagnosed by trained interventional cardiologists. Whether artificial intelligence (AI) could detect coronary stenosis from CAG video is unclear.

Methods: The 199 consecutive patients who underwent coronary arteriography (CAG) with chest pain between December 2018 and May 2019 was enrolled. Each patient underwent CAG with multiple view resulting in total numbers of 1,838 videos. A multi-layer 3-dimensional convolution neural network (CNN) was trained as an AI to detect clinically meaningful coronary artery stenosis diagnosed by the expert interventional cardiologist, using data from 146 patients (resulted in 1,359 videos) randomly selected from the entire dataset (training dataset). This training dataset was further split into 109 patients (989 videos) for derivation and 37 patients (370 videos) for validation. The AI developed in derivation cohort was tuned in validation cohort to make final model.

Results: The final model was selected as the model with best performance in validation dataset. Then, the predictive accuracy of final model was tested with the remaining 53 patients (479 videos) in test dataset. Our AI model showed a c-statistic of 0.61 in validation dataset and 0.61 in test dataset, respectively.

Conclusion: An artificial intelligence applied to CAG videos could detect clinically meaningful coronary atherosclerotic stenosis diagnosed by expert cardiologists with modest predictive value. Further studies with improved AI at larger sample size is necessary.

Hyperparameter Optimization Method Based on Harmony Search Algorithm to Improve Performance of 1D CNN Human Respiration Pattern Recognition System

In this study, we propose a method to find an optimal combination of hyperparameters to improve the accuracy of respiration pattern recognition in a 1D (Dimensional) convolutional neural network (CNN). The proposed method is designed to integrate with a 1D CNN using the harmony search algorithm. In an experiment, we used the depth of the convolutional layer of the 1D CNN, the number and size of kernels in each layer, and the number of neurons in the dense layer as hyperparameters for optimization. The experimental results demonstrate that the proposed method provided a recognition rate for five respiration patterns of approximately 96.7% on average, which is an approximately 2.8% improvement over an existing method. In addition, the number of iterations required to derive the optimal combination of hyperparameters was 2,000,000 in the previous study. In contrast, the proposed method required only 3652 iterations.