On Some Separated Algorithms for Separable Nonlinear Least Squares Problems

Online Identification of Nonlinear Systems With Separable Structure

Separable nonlinear models (SNLMs) are of great importance in system modeling, signal processing, and machine learning because of their flexible structure and excellent description of nonlinear behaviors. The online identification of such models is quite challenging, and previous related work usually ignores the special structure where the estimated parameters can be partitioned into a linear and a nonlinear part. In this brief, we propose an efficient first-order recursive algorithm for SNLMs by introducing the variable projection (VP) step. The proposed algorithm utilizes the recursive least-squares method to eliminate the linear parameters, resulting in a reduced function. Then, the stochastic gradient descent (SGD) algorithm is employed to update the parameters of the reduced function. By considering the tight coupling relationship between linear parameters and nonlinear parameters, the proposed first-order VP algorithm is more efficient and robust than the traditional SGD algorithm and alternating optimization algorithm. More importantly, since the proposed algorithm just uses the first-order information, it is easier to apply it to large-scale models. Numerical results on examples of different sizes confirm the effectiveness and efficiency of the proposed algorithm.

Shifted sediment-transport regimes by climate change and amplified hydrological variability in cryosphere-fed rivers

Climate change affects cryosphere-fed rivers and alters seasonal sediment dynamics, affecting cyclical fluvial material supply and year-round water-food-energy provisions to downstream communities. Here, we demonstrate seasonal sediment-transport regime shifts from the 1960s to 2000s in four cryosphere-fed rivers characterized by glacial, nival, pluvial, and mixed regimes, respectively. Spring sees a shift toward pluvial-dominated sediment transport due to less snowmelt and more erosive rainfall. Summer is characterized by intensified glacier meltwater pulses and pluvial events that exceptionally increase sediment fluxes. Our study highlights that the increases in hydroclimatic extremes and cryosphere degradation lead to amplified variability in fluvial fluxes and higher summer sediment peaks, which can threaten downstream river infrastructure safety and ecosystems and worsen glacial/pluvial floods. We further offer a monthly-scale sediment-availability-transport model that can reproduce such regime shifts and thus help facilitate sustainable reservoir operation and river management in wider cryospheric regions under future climate and hydrological change.

Robust Standard Gradient Descent Algorithm for ARX Models Using Aitken Acceleration Technique

A robust standard gradient descent (SGD) algorithm for ARX models using the Aitken acceleration method is developed. Considering that the SGD algorithm has slow convergence rates and is sensitive to the step size, a robust and accelerative SGD (RA-SGD) algorithm is derived. This algorithm is based on the Aitken acceleration method, and its convergence rate is improved from linear convergence to at least quadratic convergence in general. Furthermore, the RA-SGD algorithm is always convergent with no limitation of the step size. Both the convergence analysis and the simulation examples demonstrate that the presented algorithm is effective.

RNN-K: A Reinforced Newton Method for Consensus-Based Distributed Optimization and Control Over Multiagent Systems

With the rise of the processing power of networked agents in the last decade, second-order methods for machine learning have received increasing attention. To solve the distributed optimization problems over multiagent systems, Newton's method has the benefits of fast convergence and high estimation accuracy. In this article, we propose a reinforced network Newton method with K -order control flexibility (RNN-K) in a distributed manner by integrating the consensus strategy and the latest knowledge across the network into local descent direction. The key component of our method is to make the best of intermediate results from the local neighborhood to learn global knowledge, not just for the consensus effect like most existing works, including the gradient descent and Newton methods as well as their refinements. Such a reinforcement enables revitalizing the traditional iterative consensus strategy to accelerate the descent of the Newton direction. The biggest difficulty to design the approximated Newton descent in distributed settings is addressed by using a special Taylor expansion that follows the matrix splitting technique. Based on the truncation on the Taylor series, our method also presents a tradeoff effect between estimation accuracy and computation/communication cost, which provides the control flexibility as a practical consideration. We derive theoretically the sufficient conditions for the convergence of the proposed RNN-K method of at least a linear rate. The simulation results illustrate the performance effectiveness by being applied to three types of distributed optimization problems that arise frequently in machine-learning scenarios.

Recursive Variable Projection Algorithm for a Class of Separable Nonlinear Models

In this article, we study the recursive algorithms for a class of separable nonlinear models (SNLMs) in which the parameters can be partitioned into a linear part and a nonlinear part. Such models are very common in machine learning, system identification, and signal processing. Utilizing the special structure of the SNLMs, we propose a recursive variable projection (RVP) algorithm, in which at each recursion, the linear parameters of the model are eliminated, and the nonlinear parameters are updated by the recursive Levenberg-Marquart algorithm. Then, based on the updated nonlinear parameters, the linear parameters are updated by the recursive least-squares algorithm. According to a convergence analysis of the RVP algorithm, the parameter estimation error is mean-square bounded. Numerical examples confirm the satisfactory performance of the proposed algorithm.

Cooperative Output-Feedback Secure Control of Distributed Linear Cyber-Physical Systems Resist Intermittent DoS Attacks

This article studies a cooperative output-feedback secure control problem for distributed cyber-physical systems over an unreliable communication interaction, which is to achieve coordination tracking in the presence of intermittent denial-of-service (DoS) attacks. Under the switching communication network environment, first, a distributed secure control method for each subsystem is proposed via neighborhood information, which includes the local state estimator and cooperative resilient controller. Second, based on the topology-dependent Lyapunov function approach, the design conditions of secure control protocol are derived such that cooperative tracking errors are uniformly ultimately bounded. Interestingly, by exploiting the topology-allocation-dependent average dwell-time (TADADT) technique, the stability analysis of closed-loop error dynamics is presented, and the proposed coordination design conditions can relax time constraints on interaction topology switching. Finally, two numerical examples are presented to demonstrate the effectiveness of the theoretical results.

Basis Function Matrix-Based Flexible Coefficient Autoregressive Models: A Framework for Time Series and Nonlinear System Modeling

We propose, in this paper, a framework for time series and nonlinear system modeling, called the basis function matrix-based flexible coefficient autoregressive (BFM-FCAR) model. It has very flexible nonlinear structure. We show that many famous nonlinear time series models can be derived under this framework by choosing the proper basis function matrices. Some probabilistic properties (the conditions of geometrical ergodicity) of the BFM-FCAR model are investigated. Taking advantage of the model structure, we present an efficient parameter estimation algorithm for the proposed framework by using the variable projection method. Finally, we show how new models are generated from the proposed framework.

Filtering-Based Parameter Identification Methods for Multivariable Stochastic Systems

This paper presents an adaptive filtering-based maximum likelihood multi-innovation extended stochastic gradient algorithm to identify multivariable equation-error systems with colored noises. The data filtering and model decomposition techniques are used to simplify the structure of the considered system, in which a predefined filter is utilized to filter the observed data, and the multivariable system is turned into several subsystems whose parameters appear in the vectors. By introducing the multi-innovation identification theory to the stochastic gradient method, this study produces improved performances. The simulation numerical results indicate that the proposed algorithm can generate more accurate parameter estimates than the filtering-based maximum likelihood recursive extended stochastic gradient algorithm.

Insights Into Algorithms for Separable Nonlinear Least Squares Problems

Separable nonlinear least squares (SNLLS) problems have attracted interest in a wide range of research fields such as machine learning, computer vision, and signal processing. During the past few decades, several algorithms, including the joint optimization algorithm, alternated least squares (ALS) algorithm, embedded point iterations (EPI) algorithm, and variable projection (VP) algorithms, have been employed for solving SNLLS problems in the literature. The VP approach has been proven to be quite valuable for SNLLS problems and the EPI method has been successful in solving many computer vision tasks. However, no clear explanations about the intrinsic relationships of these algorithms have been provided in the literature. In this paper, we give some insights into these algorithms for SNLLS problems. We derive the relationships among different forms of the VP algorithms, EPI algorithm and ALS algorithm. In addition, the convergence and robustness of some algorithms are investigated. Moreover, the analysis of the VP algorithm generates a negative answer to Kaufman's conjecture. Numerical experiments on the image restoration task, fitting the time series data using the radial basis function network based autoregressive (RBF-AR) model, and bundle adjustment are given to compare the performance of different algorithms.

Separable Nonlinear Least-Squares Parameter Estimation for Complex Dynamic Systems

Nonlinear dynamic models are widely used for characterizing processes that govern complex biological pathway systems. Over the past decade, validation and further development of these models became possible due to data collected via high-throughput experiments using methods from molecular biology. While these data are very beneficial, they are typically incomplete and noisy, which renders the inference of parameter values for complex dynamic models challenging. Fortunately, many biological systems have embedded linear mathematical features, which may be exploited, thereby improving fits and leading to better convergence of optimization algorithms. In this paper, we explore options of inference for dynamic models using a novel method of separable nonlinear least-squares optimization and compare its performance to the traditional nonlinear least-squares method. The numerical results from extensive simulations suggest that the proposed approach is at least as accurate as the traditional nonlinear least-squares, but usually superior, while also enjoying a substantial reduction in computational time.

Motion-corrected MRI with DISORDER: Distributed and incoherent sample orders for reconstruction deblurring using encoding redundancy

PURPOSE

To enable rigid body motion-tolerant parallel volumetric magnetic resonance imaging by retrospective head motion correction on a variety of spatiotemporal scales and imaging sequences.

THEORY AND METHODS

Tolerance against rigid body motion is based on distributed and incoherent sampling orders for boosting a joint retrospective motion estimation and reconstruction framework. Motion resilience stems from the encoding redundancy in the data, as generally provided by the coil array. Hence, it does not require external sensors, navigators or training data, so the methodology is readily applicable to sequences using 3D encodings.

RESULTS

Simulations are performed showing full inter-shot corrections for usual levels of in vivo motion, large number of shots, standard levels of noise and moderate acceleration factors. Feasibility of inter- and intra-shot corrections is shown under controlled motion in vivo. Practical efficacy is illustrated by high-quality results in most corrupted of 208 volumes from a series of 26 clinical pediatric examinations collected using standard protocols.

CONCLUSIONS

The proposed framework addresses the rigid motion problem in volumetric anatomical brain scans with sufficient encoding redundancy which has enabled reliable pediatric examinations without sedation.

Modified Gram-Schmidt Method-Based Variable Projection Algorithm for Separable Nonlinear Models

Separable nonlinear models are very common in various research fields, such as machine learning and system identification. The variable projection (VP) approach is efficient for the optimization of such models. In this paper, we study various VP algorithms based on different matrix decompositions. Compared with the previous method, we use the analytical expression of the Jacobian matrix instead of finite differences. This improves the efficiency of the VP algorithms. In particular, based on the modified Gram-Schmidt (MGS) method, a more robust implementation of the VP algorithm is introduced for separable nonlinear least-squares problems. In numerical experiments, we compare the performance of five different implementations of the VP algorithm. Numerical results show the efficiency and robustness of the proposed MGS method-based VP algorithm.

A Hierarchical Approach for Joint Parameter and State Estimation of a Bilinear System with Autoregressive Noise

This paper is concerned with the joint state and parameter estimation methods for a bilinear system in the state space form, which is disturbed by additive noise. In order to overcome the difficulty that the model contains the product term of the system input and states, we make use of the hierarchical identification principle to present new methods for estimating the system parameters and states interactively. The unknown states are first estimated via a bilinear state estimator on the basis of the Kalman filtering algorithm. Then, a state estimator-based recursive generalized least squares (RGLS) algorithm is formulated according to the least squares principle. To improve the parameter estimation accuracy, we introduce the data filtering technique to derive a data filtering-based two-stage RGLS algorithm. The simulation example indicates the efficiency of the proposed algorithms.

Separate block-based parameter estimation method for Hammerstein systems

Different from the output-input representation-based identification methods of two-block Hammerstein systems, this paper concerns a separate block-based parameter estimation method for each block of a two-block Hammerstein CARMA system, without combining the parameters of two parts together. The idea is to consider each block as a subsystem and to estimate the parameters of the nonlinear block and the linear block separately (interactively), by using two least-squares algorithms in one recursive step. The internal variable between the two blocks (the output of the nonlinear block, and also the input of the linear block) is replaced by different estimates: when estimating the parameters of the nonlinear part, the internal variable between the two blocks is computed by the linear function; when estimating the parameters of the linear part, the internal variable is computed by the nonlinear function. The proposed parameter estimation method possesses property of the higher computational efficiency compared with the previous over-parametrization method in which many redundant parameters need to be computed. The simulation results show the effectiveness of the proposed algorithm.