A novel single multiplicative neuron model trained by an improved glowworm swarm optimization algorithm for time series prediction

Evaluating single multiplicative neuron models in physics-informed neural networks for differential equations

Machine learning is a prominent and highly effective field of study, renowned for its ability to yield favorable outcomes in estimation and classification tasks. Within this domain, artificial neural networks (ANNs) have emerged as one of the most powerful methodologies. Physics-informed neural networks (PINNs) have proven particularly adept at solving physics problems formulated as differential equations, incorporating boundary and initial conditions into the ANN's loss function. However, a critical challenge in ANNs lies in determining the optimal architecture, encompassing the selection of the appropriate number of neurons and layers. Traditionally, the Single Multiplicative Neuron Model (SMNM) has been explored as a solution to this issue, utilizing a single neuron with a multiplication function in the hidden layer to enhance computational efficiency. This study initially aimed to apply the SMNM within the PINNs framework, targeting the differential equationy ' - y = 0 with boundary conditions y ( 0 ) = 1 and y ( 1 ) = e . Upon implementation, however, it was discovered that while the conventional SMNM approach was theorized to offer significant advantages, multiplicative aggregate function led to a failure in convergence. Consequently, we introduced a "mimic single multiplicative neuron model (mimic-SMNM)" employing an architecture with a single neuron, designed to simulate the SMNM's conceptual advantages while ensuring convergence and computational efficiency. Comparative analysis revealed that the real-PINNs accurately solved the equation, the true SMNM failed to converge, and the mimic model was highlighted for its architectural simplicity and computational feasibility, directly implying it is faster and more efficient than real PINNs for the solution of simple differential equations. Furthermore, our findings demonstrated that our proposed mimic-SMNM model achieves a five-times increase in computational speed compared to real PINNs after 30,000 epochs.

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review

The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.

A new automatic forecasting method based on a new input significancy test of a single multiplicative neuron model artificial neural network

The model adequacy and input significance tests have not been proposed as features for the specification of a single multiplicative neuron model artificial neural networks in the literature. Moreover, there is no systematic approach based on hypothesis tests for using single multiplicative neuron model artificial neural networks for forecasting purposes like classical time series forecasting methods. In this study, new methods are proposed to solve these problems. The performance of the proposed test procedures is investigated in a simulation study. According to simulation results, the proposed tests have very good performance. Moreover, the test procedures are illustrated by using two real-world examples. The second contribution of the paper is that an automatic forecasting method is proposed based on input significance and model adequacy tests and the particle swarm optimization-based learning algorithm. The proposed automatic forecasting method is applied to M4 competition hourly data sets, and it is the best pure machine learning method among others in the competition. The proposed automatic forecasting method is more accurate than all benchmarks, such as MLP, RNN, and ETS, which were proposed by the competition organizers.

Firefly Mating Algorithm for Continuous Optimization Problems

This paper proposes a swarm intelligence algorithm, called firefly mating algorithm (FMA), for solving continuous optimization problems. FMA uses genetic algorithm as the core of the algorithm. The main feature of the algorithm is a novel mating pair selection method which is inspired by the following 2 mating behaviors of fireflies in nature: (i) the mutual attraction between males and females causes them to mate and (ii) fireflies of both sexes are of the multiple-mating type, mating with multiple opposite sex partners. A female continues mating until her spermatheca becomes full, and, in the same vein, a male can provide sperms for several females until his sperm reservoir is depleted. This new feature enhances the global convergence capability of the algorithm. The performance of FMA was tested with 20 benchmark functions (sixteen 30-dimensional functions and four 2-dimensional ones) against FA, ALC-PSO, COA, MCPSO, LWGSODE, MPSODDS, DFOA, SHPSOS, LSA, MPDPGA, DE, and GABC algorithms. The experimental results showed that the success rates of our proposed algorithm with these functions were higher than those of other algorithms and the proposed algorithm also required fewer numbers of iterations to reach the global optima.