A global neural network learning machine: Coupled integer and fractional calculus operator with an adaptive learning scheme

A fractional gradient descent algorithm robust to the initial weights of multilayer perceptron

For multilayer perceptron (MLP), the initial weights will significantly influence its performance. Based on the enhanced fractional derivative extend from convex optimization, this paper proposes a fractional gradient descent (RFGD) algorithm robust to the initial weights of MLP. We analyze the effectiveness of the RFGD algorithm. The convergence of the RFGD algorithm is also analyzed. The computational complexity of the RFGD algorithm is generally larger than that of the gradient descent (GD) algorithm but smaller than that of the Adam, Padam, AdaBelief, and AdaDiff algorithms. Numerical experiments show that the RFGD algorithm has strong robustness to the order of fractional calculus which is the only added parameter compared to the GD algorithm. More importantly, compared to the GD, Adam, Padam, AdaBelief, and AdaDiff algorithms, the experimental results show that the RFGD algorithm has the best robust performance for the initial weights of MLP. Meanwhile, the correctness of the theoretical analysis is verified.

A hybrid algorithm based on state-adaptive slime mold model and fractional-order ant system for the travelling salesman problem

The ant colony optimization (ACO) is one efficient approach for solving the travelling salesman problem (TSP). Here, we propose a hybrid algorithm based on state-adaptive slime mold model and fractional-order ant system (SSMFAS) to address the TSP. The state-adaptive slime mold (SM) model with two targeted auxiliary strategies emphasizes some critical connections and balances the exploration and exploitation ability of SSMFAS. The consideration of fractional-order calculus in the ant system (AS) takes full advantage of the neighboring information. The pheromone update rule of AS is modified to dynamically integrate the flux information of SM. To understand the search behavior of the proposed algorithm, some mathematical proofs of convergence analysis are given. The experimental results validate the efficiency of the hybridization and demonstrate that the proposed algorithm has the competitive ability of finding the better solutions on TSP instances compared with some state-of-the-art algorithms.

Exploring the Effects of Caputo Fractional Derivative in Spiking Neural Network Training

Fractional calculus is an emerging topic in artificial neural network training, especially when using gradient-based methods. This paper brings the idea of fractional derivatives to spiking neural network training using Caputo derivative-based gradient calculation. We focus on conducting an extensive investigation of performance improvements via a case study of small-scale networks using derivative orders in the unit interval. With particle swarm optimization we provide an example of handling the derivative order as an optimizable hyperparameter to find viable values for it. Using multiple benchmark datasets we empirically show that there is no single generally optimal derivative order, rather this value is data-dependent. However, statistics show that a range of derivative orders can be determined where the Caputo derivative outperforms first-order gradient descent with high confidence. Improvements in convergence speed and training time are also examined and explained by the reformulation of the Caputo derivative-based training as an adaptive weight normalization technique.