Using machine learning to predict statistical properties of non-stationary dynamical processes: System climate,regime transitions, and the effect of stochasticity

Time-series-analysis-based detection of critical transitions in real-world non-autonomous systems

Real-world non-autonomous systems are open, out-of-equilibrium systems that evolve in and are driven by temporally varying environments. Such systems can show multiple timescale and transient dynamics together with transitions to very different and, at times, even disastrous dynamical regimes. Since such critical transitions disrupt the systems' intended or desired functionality, it is crucial to understand the underlying mechanisms, to identify precursors of such transitions, and to reliably detect them in time series of suitable system observables to enable forecasts. This review critically assesses the various steps of investigation involved in time-series-analysis-based detection of critical transitions in real-world non-autonomous systems: from the data recording to evaluating the reliability of offline and online detections. It will highlight pros and cons to stimulate further developments, which would be necessary to advance understanding and forecasting nonlinear behavior such as critical transitions in complex systems.

Reservoir-computing based associative memory and itinerancy for complex dynamical attractors

Traditional neural network models of associative memories were used to store and retrieve static patterns. We develop reservoir-computing based memories for complex dynamical attractors, under two common recalling scenarios in neuropsychology: location-addressable with an index channel and content-addressable without such a channel. We demonstrate that, for location-addressable retrieval, a single reservoir computing machine can memorize a large number of periodic and chaotic attractors, each retrievable with a specific index value. We articulate control strategies to achieve successful switching among the attractors, unveil the mechanism behind failed switching, and uncover various scaling behaviors between the number of stored attractors and the reservoir network size. For content-addressable retrieval, we exploit multistability with cue signals, where the stored attractors coexist in the high-dimensional phase space of the reservoir network. As the length of the cue signal increases through a critical value, a high success rate can be achieved. The work provides foundational insights into developing long-term memories and itinerancy for complex dynamical patterns.

A TCN-Linear Hybrid Model for Chaotic Time Series Forecasting

The applications of deep learning and artificial intelligence have permeated daily life, with time series prediction emerging as a focal area of research due to its significance in data analysis. The evolution of deep learning methods for time series prediction has progressed from the Convolutional Neural Network (CNN) and the Recurrent Neural Network (RNN) to the recently popularized Transformer network. However, each of these methods has encountered specific issues. Recent studies have questioned the effectiveness of the self-attention mechanism in Transformers for time series prediction, prompting a reevaluation of approaches to LTSF (Long Time Series Forecasting) problems. To circumvent the limitations present in current models, this paper introduces a novel hybrid network, Temporal Convolutional Network-Linear (TCN-Linear), which leverages the temporal prediction capabilities of the Temporal Convolutional Network (TCN) to enhance the capacity of LSTF-Linear. Time series from three classical chaotic systems (Lorenz, Mackey-Glass, and Rossler) and real-world stock data serve as experimental datasets. Numerical simulation results indicate that, compared to classical networks and novel hybrid models, our model achieves the lowest RMSE, MAE, and MSE with the fewest training parameters, and its R2 value is the closest to 1.

Machine learning approach to detect dynamical states from recurrence measures

We integrate machine learning approaches with nonlinear time series analysis, specifically utilizing recurrence measures to classify various dynamical states emerging from time series. We implement three machine learning algorithms: Logistic Regression, Random Forest, and Support Vector Machine for this study. The input features are derived from the recurrence quantification of nonlinear time series and characteristic measures of the corresponding recurrence networks. For training and testing, we generate synthetic data from standard nonlinear dynamical systems and evaluate the efficiency and performance of the machine learning algorithms in classifying time series into periodic, chaotic, hyperchaotic, or noisy categories. Additionally, we explore the significance of input features in the classification scheme and find that the features quantifying the density of recurrence points are the most relevant. Furthermore, we illustrate how the trained algorithms can successfully predict the dynamical states of two variable stars, SX Her and AC Her, from the data of their light curves. We also indicate how the algorithms can be trained to classify data from discrete systems.

Reconstructing bifurcation diagrams of chaotic circuits with reservoir computing

Model-free reconstruction of bifurcation diagrams of Chua's circuits using the technique of parameter-aware reservoir computing is investigated. We demonstrate that (1) reservoir computer can be utilized as a noise filter to restore the system dynamics from noisy signals; (2) for a single Chua circuit, a machine trained by the noisy time series measured at several sampling states is capable of reconstructing the whole bifurcation diagram of the circuit with a high precision; and (3) for two coupled chaotic Chua circuits with mismatched parameters, the machine trained by the noisy time series measured at several coupling strengths is able to anticipate the variation of the synchronization degree of the coupled circuits with respect to the coupling strength over a wide range. Our studies verify the capability of the technique of parameter-aware reservoir computing in learning the dynamics of chaotic circuits from noisy signals, signifying the potential application of this technique in reconstructing the bifurcation diagram of real-world chaotic systems.

Extrapolating tipping points and simulating non-stationary dynamics of complex systems using efficient machine learning

Model-free and data-driven prediction of tipping point transitions in nonlinear dynamical systems is a challenging and outstanding task in complex systems science. We propose a novel, fully data-driven machine learning algorithm based on next-generation reservoir computing to extrapolate the bifurcation behavior of nonlinear dynamical systems using stationary training data samples. We show that this method can extrapolate tipping point transitions. Furthermore, it is demonstrated that the trained next-generation reservoir computing architecture can be used to predict non-stationary dynamics with time-varying bifurcation parameters. In doing so, post-tipping point dynamics of unseen parameter regions can be simulated.

Predicting nonsmooth chaotic dynamics by reservoir computing

Reservoir computing (RC) has been widely applied to predict the chaotic dynamics in many systems. Yet much broader areas related to nonsmooth dynamics have seldom been touched by the RC community which have great theoretical and practical importance. The generalization of RC to this kind of system is reported in this paper. The numerical work shows that the conventional RC with a hyperbolic tangent activation function is not able to predict the dynamics of nonsmooth systems very well, especially when reconstructing attractors (long-term prediction). A nonsmooth activation function with a piecewise nature is proposed. A kind of physics-informed RC scheme is established based on this activation function. The feasibility of this scheme has been proven by its successful application to the predictions of the short- and long-term (reconstructing chaotic attractor) dynamics of four nonsmooth systems with different complexity, including the tent map, piecewise linear map with a gap, both noninvertible and discontinuous compound circle maps, and Lozi map. The results show that RC with the new activation function is efficient and easy to run. It can make perfectly both short- and long-term predictions. The precision of reconstructing attractors depends on their complexity. This work reveals that, to make efficient predictions, the activation function of an RC approach should match the smooth or nonsmooth nature of the dynamical systems.

Model-free tracking control of complex dynamical trajectories with machine learning

Nonlinear tracking control enabling a dynamical system to track a desired trajectory is fundamental to robotics, serving a wide range of civil and defense applications. In control engineering, designing tracking control requires complete knowledge of the system model and equations. We develop a model-free, machine-learning framework to control a two-arm robotic manipulator using only partially observed states, where the controller is realized by reservoir computing. Stochastic input is exploited for training, which consists of the observed partial state vector as the first and its immediate future as the second component so that the neural machine regards the latter as the future state of the former. In the testing (deployment) phase, the immediate-future component is replaced by the desired observational vector from the reference trajectory. We demonstrate the effectiveness of the control framework using a variety of periodic and chaotic signals, and establish its robustness against measurement noise, disturbances, and uncertainties.

Predicting chaotic dynamics from incomplete input via reservoir computing with (D+1)-dimension input and output

Predicting future evolution based on incomplete information of the past is still a challenge even though data-driven machine learning approaches have been successfully applied to forecast complex nonlinear dynamics. The widely adopted reservoir computing (RC) can hardly deal with this since it usually requires complete observations of the past. In this paper, a scheme of RC with (D+1)-dimension input and output (I/O) vectors is proposed to solve this problem, i.e., the incomplete input time series or dynamical trajectories of a system, in which certain portion of states are randomly removed. In this scheme, the I/O vectors coupled to the reservoir are changed to (D+1)-dimension, where the first D dimensions store the state vector as in the conventional RC, and the additional dimension is the corresponding time interval. We have successfully applied this approach to predict the future evolution of the logistic map and Lorenz, Rössler, and Kuramoto-Sivashinsky systems, where the inputs are the dynamical trajectories with missing data. The dropoff rate dependence of the valid prediction time (VPT) is analyzed. The results show that it can make forecasting with much longer VPT when the dropoff rate θ is lower. The reason for the failure at high θ is analyzed. The predictability of our RC is determined by the complexity of the dynamical systems involved. The more complex they are, the more difficult they are to predict. Perfect reconstructions of chaotic attractors are observed. This scheme is a pretty good generalization to RC and can treat input time series with regular and irregular time intervals. It is easy to use since it does not change the basic architecture of conventional RC. Furthermore, it can make multistep-ahead prediction just by changing the time interval in the output vector into a desired value, which is superior to conventional RC that can only do one-step-ahead forecasting based on complete regular input data.

Reservoir computing as digital twins for nonlinear dynamical systems

We articulate the design imperatives for machine learning based digital twins for nonlinear dynamical systems, which can be used to monitor the "health" of the system and anticipate future collapse. The fundamental requirement for digital twins of nonlinear dynamical systems is dynamical evolution: the digital twin must be able to evolve its dynamical state at the present time to the next time step without further state input-a requirement that reservoir computing naturally meets. We conduct extensive tests using prototypical systems from optics, ecology, and climate, where the respective specific examples are a chaotic CO₂ laser system, a model of phytoplankton subject to seasonality, and the Lorenz-96 climate network. We demonstrate that, with a single or parallel reservoir computer, the digital twins are capable of a variety of challenging forecasting and monitoring tasks. Our digital twin has the following capabilities: (1) extrapolating the dynamics of the target system to predict how it may respond to a changing dynamical environment, e.g., a driving signal that it has never experienced before, (2) making continual forecasting and monitoring with sparse real-time updates under non-stationary external driving, (3) inferring hidden variables in the target system and accurately reproducing/predicting their dynamical evolution, (4) adapting to external driving of different waveform, and (5) extrapolating the global bifurcation behaviors to network systems of different sizes. These features make our digital twins appealing in applications, such as monitoring the health of critical systems and forecasting their potential collapse induced by environmental changes or perturbations. Such systems can be an infrastructure, an ecosystem, or a regional climate system.

Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems

The ability of machine learning (ML) models to "extrapolate" to situations outside of the range spanned by their training data is crucial for predicting the long-term behavior of non-stationary dynamical systems (e.g., prediction of terrestrial climate change), since the future trajectories of such systems may (perhaps after crossing a tipping point) explore regions of state space which were not explored in past time-series measurements used as training data. We investigate the extent to which ML methods can yield useful results by extrapolation of such training data in the task of forecasting non-stationary dynamics, as well as conditions under which such methods fail. In general, we find that ML can be surprisingly effective even in situations that might appear to be extremely challenging, but do (as one would expect) fail when "too much" extrapolation is required. For the latter case, we show that good results can potentially be obtained by combining the ML approach with an available inaccurate conventional model based on scientific knowledge.

Characterizing chaos in systems subjected to parameter drift

To characterize chaos in systems subjected to parameter drift, where a number of traditional methods do not apply, we propose viable alternative approaches, both in the qualitative and quantitative sense. Qualitatively, following stable and unstable foliations is shown to be efficient, which are easy to approximate numerically, without relying on the need for the existence of an analog of hyperbolic periodic orbits. Chaos originates from a Smale horseshoe-like pattern of the foliations, the transverse intersections of which indicate a chaotic set changing in time. In dissipative cases, the unstable foliation is found to be part of the so-called snapshot attractor, but the chaotic set is not dense on it if regular time-dependent attractors also exist. In Hamiltonian cases stable and unstable foliations turn out to be not equivalent due to the lack of time-reversal symmetry. It is the unstable foliation, which is found to correlate with the so-called snapshot chaotic sea. The chaotic set appears to be locally dense in this sea, while tori with originally quasiperiodic character might break up, their motion becoming chaotic as time goes on. A quantity called ensemble-averaged pairwise distance evaluated in relation to unstable foliations is shown to be an appropriate tool to provide the instantaneous strength of time-dependent chaos.

Criticality in reservoir computer of coupled phase oscillators

Accumulating evidence shows that the cerebral cortex is operating near a critical state featured by power-law size distribution of neural avalanche activities, yet evidence of this critical state in artificial neural networks mimicking the cerebral cortex is still lacking. Here we design an artificial neural network of coupled phase oscillators and, by the technique of reservoir computing in machine learning, train it for predicting chaos. It is found that when the machine is properly trained, oscillators in the reservoir are synchronized into clusters whose sizes follow a power-law distribution. This feature, however, is absent when the machine is poorly trained. Additionally, it is found that despite the synchronization degree of the original network, once properly trained, the reservoir network is always developed to the same critical state, exemplifying the "attractor" nature of this state in machine learning. The generality of the results is verified in different reservoir models and by different target systems, and it is found that the scaling exponent of the distribution is independent of the reservoir details and the bifurcation parameters of the target system, but is modified when the dynamics of the target system is changed to a different type. The findings shed light on the nature of machine learning, and are helpful to the design of high-performance machines in physical systems.

Data driven adaptive Gaussian mixture model for solving Fokker-Planck equation

The Fokker-Planck (FP) equation provides a powerful tool for describing the state transition probability density function of complex dynamical systems governed by stochastic differential equations (SDEs). Unfortunately, the analytical solution of the FP equation can be found in very few special cases. Therefore, it has become an interest to find a numerical approximation method of the FP equation suitable for a wider range of nonlinear systems. In this paper, a machine learning method based on an adaptive Gaussian mixture model (AGMM) is proposed to deal with the general FP equations. Compared with previous numerical discretization methods, the proposed method seamlessly integrates data and mathematical models. The prior knowledge generated by the assumed mathematical model can improve the performance of the learning algorithm. Also, it yields more interpretability for machine learning methods. Numerical examples for one-dimensional and two-dimensional SDEs with one and/or two noises are given. The simulation results show the effectiveness and robustness of the AGMM technique for solving the FP equation. In addition, the computational complexity and the optimization algorithm of the model are also discussed.

A Stacking Ensemble Learning Model for Monthly Rainfall Prediction in the Taihu Basin, China

The prediction of monthly rainfall is greatly beneficial for water resources management and flood control projects. Machine learning (ML) techniques, as an increasingly popular approach, have been applied in diverse climatic regions, showing their respective superiority. On top of that, the ensemble learning model that synthesizes the advantages of different ML models deserves more attention. In this study, an ensemble learning model based on stacking approach was proposed. Four prevalent ML models, namely k-nearest neighbors (KNN), extreme gradient boosting (XGB), support vector regression (SVR), and artificial neural networks (ANN) are taken as base models. To combine the outputs from the base models, the weighting algorithm is used as second-layer learner to generate predictions. Large-scale climate indices, large-scale atmospheric variables, and local meteorological variables were used as predictors. R2, RMSE and MAE, were used as evaluation metrics. The results show that the performance of base models varied among the nine stations in the Taihu Basin, while the stacking approach generally performed better than the four base models. The stacking model showed better performance in spring and winter than in summer and autumn. During wet months, the accuracy of model prediction varied more significantly. On the whole, based on performance evaluation measures, it is concluded that the proposed stacking ensemble multi-ML model can provide a flexible and reasonable prediction framework applicable to other regions.