A one-layer recurrent neural network for nonsmooth pseudoconvex optimization with quasiconvex inequality and affine equality constraints

Inverse-free zeroing neural network for time-variant nonlinear optimization with manipulator applications

In this paper, the problem of time-variant optimization subject to nonlinear equation constraint is studied. To solve the challenging problem, methods based on the neural networks, such as zeroing neural network and gradient neural network, are commonly adopted due to their performance on handling nonlinear problems. However, the traditional zeroing neural network algorithm requires computing the matrix inverse during the solving process, which is a complicated and time-consuming operation. Although the gradient neural network algorithm does not require computing the matrix inverse, its accuracy is not high enough. Therefore, a novel inverse-free zeroing neural network algorithm without matrix inverse is proposed in this paper. The proposed algorithm not only avoids the matrix inverse, but also avoids matrix multiplication, greatly reducing the computational complexity. In addition, detailed theoretical analyses of the convergence performance of the proposed algorithm is provided to guarantee its excellent capability in solving time-variant optimization problems. Numerical simulations and comparative experiments with traditional zeroing neural network and gradient neural network algorithms substantiate the accuracy and superiority of the novel inverse-free zeroing neural network algorithm. To further validate the performance of the novel inverse-free zeroing neural network algorithm in practical applications, path tracking tasks of three manipulators (i.e., Universal Robot 5, Franka Emika Panda, and Kinova JACO2 manipulators) are conducted, and the results verify the applicability of the proposed algorithm.

Binary matrix factorization via collaborative neurodynamic optimization

Binary matrix factorization is an important tool for dimension reduction for high-dimensional datasets with binary attributes and has been successfully applied in numerous areas. This paper presents a collaborative neurodynamic optimization approach to binary matrix factorization based on the original combinatorial optimization problem formulation and quadratic unconstrained binary optimization problem reformulations. The proposed approach employs multiple discrete Hopfield networks operating concurrently in search of local optima. In addition, a particle swarm optimization rule is used to reinitialize neuronal states iteratively to escape from local minima toward better ones. Experimental results on eight benchmark datasets are elaborated to demonstrate the superior performance of the proposed approach against six baseline algorithms in terms of factorization error. Additionally, the viability of the proposed approach is demonstrated for pattern discovery on three datasets.

A Hyper-Transformer model for Controllable Pareto Front Learning with Split Feasibility Constraints

Controllable Pareto front learning (CPFL) approximates the Pareto optimal solution set and then locates a non-dominated point with respect to a given reference vector. However, decision-maker objectives were limited to a constraint region in practice, so instead of training on the entire decision space, we only trained on the constraint region. Controllable Pareto front learning with Split Feasibility Constraints (SFC) is a way to find the best Pareto solutions to a split multi-objective optimization problem that meets certain constraints. In the previous study, CPFL used a Hypernetwork model comprising multi-layer perceptron (Hyper-MLP) blocks. Transformer can be more effective than previous architectures on numerous modern deep learning tasks in certain situations due to their distinctive advantages. Therefore, we have developed a hyper-transformer (Hyper-Trans) model for CPFL with SFC. We use the theory of universal approximation for the sequence-to-sequence function to show that the Hyper-Trans model makes MED errors smaller in computational experiments than the Hyper-MLP model.

A Recurrent Neural Network Approach for Constrained Distributed Fuzzy Convex Optimization

This article investigates a class of constrained distributed fuzzy convex optimization problems, where the objective function is the sum of a set of local fuzzy convex objective functions, and the constraints include partial order relation and closed convex set constraints. In undirected connected node communication network, each node only knows its own objective function and constraints, and the local objective function and partial order relation functions may be nonsmooth. To solve this problem, a recurrent neural network approach based on differential inclusion framework is proposed. The network model is constructed with the help of the idea of penalty function, and the estimation of penalty parameters in advance is eliminated. Through theoretical analysis, it is proven that the state solution of the network enters the feasible region in finite time and does not escape again, and finally reaches consensus at an optimal solution of the distributed fuzzy optimization problem. Furthermore, the stability and global convergence of the network do not depend on the selection of the initial state. A numerical example and an intelligent ship output power optimization problem are given to illustrate the feasibility and effectiveness of the proposed approach.

A novel fixed-time error-monitoring neural network for solving dynamic quaternion-valued Sylvester equations

This paper addresses the dynamic quaternion-valued Sylvester equation (DQSE) using the quaternion real representation and the neural network method. To transform the Sylvester equation in the quaternion field into an equivalent equation in the real field, three different real representation modes for the quaternion are adopted by considering the non-commutativity of quaternion multiplication. Based on the equivalent Sylvester equation in the real field, a novel recurrent neural network model with an integral design formula is proposed to solve the DQSE. The proposed model, referred to as the fixed-time error-monitoring neural network (FTEMNN), achieves fixed-time convergence through the action of a state-of-the-art nonlinear activation function. The fixed-time convergence of the FTEMNN model is theoretically analyzed. Two examples are presented to verify the performance of the FTEMNN model with a specific focus on fixed-time convergence. Furthermore, the chattering phenomenon of the FTEMNN model is discussed, and a saturation function scheme is designed. Finally, the practical value of the FTEMNN model is demonstrated through its application to image fusion denoising.

Two-timescale projection neural networks in collaborative neurodynamic approaches to global optimization and distributed optimization

In this paper, we propose a two-timescale projection neural network (PNN) for solving optimization problems with nonconvex functions. We prove the convergence of the PNN with sufficiently different timescales to a local optimal solution. We develop a collaborative neurodynamic approach with multiple such PNNs to search for global optimal solutions. In addition, we develop a collaborative neurodynamic approach with multiple PNNs connected via a directed graph for distributed global optimization. We elaborate on four numerical examples to illustrate the characteristics of the approaches.

A framework for controllable Pareto front learning with completed scalarization functions and its applications

Pareto Front Learning (PFL) was recently introduced as an efficient method for approximating the entire Pareto front, the set of all optimal solutions to a Multi-Objective Optimization (MOO) problem. In the previous work, the mapping between a preference vector and a Pareto optimal solution is still ambiguous, rendering its results. This study demonstrates the convergence and completion aspects of solving MOO with pseudoconvex scalarization functions and combines them into Hypernetwork in order to offer a comprehensive framework for PFL, called Controllable Pareto Front Learning. Extensive experiments demonstrate that our approach is highly accurate and significantly less computationally expensive than prior methods in term of inference time.

Enhancing neurodynamic approach with physics-informed neural networks for solving non-smooth convex optimization problems

This paper proposes a deep learning approach for solving non-smooth convex optimization problems (NCOPs), which have broad applications in computer science, engineering, and physics. Our approach combines neurodynamic optimization with physics-informed neural networks (PINNs) to provide an efficient and accurate solution. We first use neurodynamic optimization to formulate an initial value problem (IVP) that involves a system of ordinary differential equations for the NCOP. We then introduce a modified PINN as an approximate state solution to the IVP. Finally, we develop a dedicated algorithm to train the model to solve the IVP and minimize the NCOP objective simultaneously. Unlike existing numerical integration methods, a key advantage of our approach is that it does not require the computation of a series of intermediate states to produce a prediction of the NCOP. Our experimental results show that this computational feature results in fewer iterations being required to produce more accurate prediction solutions. Furthermore, our approach is effective in finding feasible solutions that satisfy the NCOP constraint.

Novel adaptive zeroing neural dynamics schemes for temporally-varying linear equation handling applied to arm path following and target motion positioning

While the handling for temporally-varying linear equation (TVLE) has received extensive attention, most methods focused on trading off the conflict between computational precision and convergence rate. Different from previous studies, this paper proposes two complete adaptive zeroing neural dynamics (ZND) schemes, including a novel adaptive continuous ZND (ACZND) model, two general variable time discretization techniques, and two resultant adaptive discrete ZND (ADZND) algorithms, to essentially eliminate the conflict. Specifically, an error-related varying-parameter ACZND model with global and exponential convergence is first designed and proposed. To further adapt to the digital hardware, two novel variable time discretization techniques are proposed to discretize the ACZND model into two ADZND algorithms. The convergence properties with respect to the convergence rate and precision of ADZND algorithms are proved via rigorous mathematical analyses. By comparing with the traditional discrete ZND (TDZND) algorithms, the superiority of ADZND algorithms in convergence rate and computational precision is shown theoretically and experimentally. Finally, simulative experiments, including numerical experiments on a specific TVLE solving as well as four application experiments on arm path following and target motion positioning are successfully conducted to substantiate the efficacy, superiority, and practicability of ADZND algorithms.

Neurodynamic optimization approaches with finite/fixed-time convergence for absolute value equations

This paper proposes three novel accelerated inverse-free neurodynamic approaches to solve absolute value equations (AVEs). The first two are finite-time converging approaches and the third one is a fixed-time converging approach. It is shown that the proposed first two neurodynamic approaches converge to the solution of the concerned AVEs in a finite-time while, under some mild conditions, the third one converges to the solution in a fixed-time. It is also shown that the settling time for the proposed fixed-time converging approach has an uniform upper bound for all initial conditions, while the settling times for the proposed finite-time converging approaches are dependent on initial conditions. The proposed neurodynamic approaches have the advantage that they are all robust against bounded vanishing perturbations. The theoretical results are validated by means of a numerical example and an application in boundary value problems.

A fractional gradient descent algorithm robust to the initial weights of multilayer perceptron

For multilayer perceptron (MLP), the initial weights will significantly influence its performance. Based on the enhanced fractional derivative extend from convex optimization, this paper proposes a fractional gradient descent (RFGD) algorithm robust to the initial weights of MLP. We analyze the effectiveness of the RFGD algorithm. The convergence of the RFGD algorithm is also analyzed. The computational complexity of the RFGD algorithm is generally larger than that of the gradient descent (GD) algorithm but smaller than that of the Adam, Padam, AdaBelief, and AdaDiff algorithms. Numerical experiments show that the RFGD algorithm has strong robustness to the order of fractional calculus which is the only added parameter compared to the GD algorithm. More importantly, compared to the GD, Adam, Padam, AdaBelief, and AdaDiff algorithms, the experimental results show that the RFGD algorithm has the best robust performance for the initial weights of MLP. Meanwhile, the correctness of the theoretical analysis is verified.

Sparse signal reconstruction via collaborative neurodynamic optimization

In this paper, we formulate a mixed-integer problem for sparse signal reconstruction and reformulate it as a global optimization problem with a surrogate objective function subject to underdetermined linear equations. We propose a sparse signal reconstruction method based on collaborative neurodynamic optimization with multiple recurrent neural networks for scattered searches and a particle swarm optimization rule for repeated repositioning. We elaborate on experimental results to demonstrate the outperformance of the proposed approach against ten state-of-the-art algorithms for sparse signal reconstruction.

Boolean matrix factorization based on collaborative neurodynamic optimization with Boltzmann machines

This paper presents a collaborative neurodynamic approach to Boolean matrix factorization. Based on a binary optimization formulation to minimize the Hamming distance between a given data matrix and its low-rank reconstruction, the proposed approach employs a population of Boltzmann machines operating concurrently for scatter search of factorization solutions. In addition, a particle swarm optimization rule is used to re-initialize the neuronal states of Boltzmann machines upon their local convergence to escape from local minima toward global solutions. Experimental results demonstrate the superior convergence and performance of the proposed approach against six baseline methods on ten benchmark datasets.