Marginally Stable Triangular Recurrent Neural Network Architecture for Time Series Prediction

Growing Echo State Network With an Inverse-Free Weight Update Strategy

An echo state network (ESN) draws widespread attention and is applied in many scenarios. As the most typical approach for solving the ESN, the matrix inverse operation of high computational complexity is involved. However, in the modern big data era, addressing the heavy computational burden problem is necessary. In order to reduce the computational load, an inverse-free ESN (IFESN) is proposed for the first time in this article. Besides, an incremental IFESN is constructed to attain the network topology with theoretical proof on the training error's monotone decline property. Simulations and experiments are conducted on several numerical and real-world time-series benchmarks, and corresponding results indicate that the proposed model is superior to some existing models and possesses excellent practical application potential. The source code is publicly available at https://github.com/LongJin-lab/the-supplementary-file-for-CYB-E-2021-04-0944.

Modified BBO-Based Multivariate Time-Series Prediction System With Feature Subset Selection and Model Parameter Optimization

Multivariate time-series prediction is a challenging research topic in the field of time-series analysis and modeling, and is continually under research. The echo state network (ESN), a type of efficient recurrent neural network, has been widely used in time-series prediction, but when using ESN, two crucial problems have to be confronted: 1) how to select the optimal subset of input features and 2) how to set the suitable parameters of the model. To solve this problem, the modified biogeography-based optimization ESN (MBBO-ESN) system is proposed for system modeling and multivariate time-series prediction, which can simultaneously achieve feature subset selection and model parameter optimization. The proposed MBBO algorithm is an improved evolutionary algorithm based on biogeography-based optimization (BBO), which utilizes an S -type population migration rate model, a covariance matrix migration strategy, and a Lévy distribution mutation strategy to enhance the rotation invariance and exploration ability. Furthermore, the MBBO algorithm cannot only optimize the key parameters of the ESN model but also uses a hybrid-metric feature selection method to remove the redundancies and distinguish the importance of the input features. Compared with the traditional methods, the proposed MBBO-ESN system can discover the relationship between the input features and the model parameters automatically and make the prediction more accurate. The experimental results on the benchmark and real-world datasets demonstrate that MBBO outperforms the other traditional evolutionary algorithms, and the MBBO-ESN system is more competitive in multivariate time-series prediction than other classic machine-learning models.

Data-driven stochastic model for cross-interacting processes with different time scales

In this work, we propose a new data-driven method for modeling cross-interacting processes with different time scales represented by time series with different sampling steps. It is a generalization of a nonlinear stochastic model of an evolution operator based on neural networks and designed for the case of time series with a constant sampling step. The proposed model has a more complex structure. First, it describes each process by its own stochastic evolution operator with its own time step. Second, it takes into account possible nonlinear connections within each pair of processes in both directions. These connections are parameterized asymmetrically, depending on which process is faster and which process is slower. They make this model essentially different from the set of independent stochastic models constructed individually for each time scale. All evolution operators and connections are trained and optimized using the Bayesian framework, forming a multi-scale stochastic model. We demonstrate the performance of the model on two examples. The first example is a pair of coupled oscillators, with the couplings in both directions which can be turned on and off. Here, we show that inclusion of the connections into the model allows us to correctly reproduce observable effects related to coupling. The second example is a spatially distributed data generated by a global climate model running in the middle 19th century external conditions. In this case, the multi-scale model allows us to reproduce the coupling between the processes which exists in the observed data but is not captured by the model constructed individually for each process.

Maximum Information Exploitation Using Broad Learning System for Large-Scale Chaotic Time-Series Prediction

How to make full use of the evolution information of chaotic systems for time-series prediction is a difficult issue in dynamical system modeling. In this article, we propose a maximum information exploitation broad learning system (MIE-BLS) for extreme information utilization of large-scale chaotic time-series modeling. An improved leaky integrator dynamical reservoir is introduced in order to capture the linear information of chaotic systems effectively. It can not only capture the information of the current state but also achieve the compromise with historical states in the dynamical system. Furthermore, the feature is mapped to the enhancement layer by nonlinear random mapping to exploit nonlinear information. The cascading mechanism promotes the information propagation and achieves feature reactivation in dynamical modeling. Discussions about maximum information exploration and the comparisons with ResNet, DenseNet, and HighwayNet are presented in this article. Simulation results on four large-scale data sets illustrate that MIE-BLS could achieve better performance of information exploration in large-scale dynamical system modeling.

Atmospheric PM2.5 Prediction Based on Multiple Model Adaptive Unscented Kalman Filter

The PM2.5 concentration model is the key to predict PM2.5 concentration. During the prediction of atmospheric PM2.5 concentration based on prediction model, the prediction model of PM2.5 concentration cannot be usually accurately described. For the PM2.5 concentration model in the same period, the dynamic characteristics of the model will change under the influence of many factors. Similarly, for different time periods, the corresponding models of PM2.5 concentration may be different, and the single model cannot play the corresponding ability to predict PM2.5 concentration. The single model leads to the decline of prediction accuracy. To improve the accuracy of PM2.5 concentration prediction in this solution, a multiple model adaptive unscented Kalman filter (MMAUKF) method is proposed in this paper. Firstly, the PM2.5 concentration data in three time periods of the day are taken as the research object, the nonlinear state space model frame of a support vector regression (SVR) method is established. Secondly, the frame of the SVR model in three time periods is combined with an adaptive unscented Kalman filter (AUKF) to predict PM2.5 concentration in the next hour, respectively. Then, the predicted value of three time periods is fused into the final predicted PM2.5 concentration by Bayesian weighting method. Finally, the proposed method is compared with the single support vector regression-adaptive unscented Kalman filter (SVR-AUKF), autoregressive model-Kalman (AR-Kalman), autoregressive model (AR) and back propagation neural network (BP). The prediction results show that the accuracy of PM2.5 concentration prediction is improved in whole time period.

End-to-End Incomplete Time-Series Modeling From Linear Memory of Latent Variables

Time series with missing values (incomplete time series) are ubiquitous in real life on account of noise or malfunctioning sensors. Time-series imputation (replacing missing data) remains a challenge due to the potential for nonlinear dependence on concurrent and previous values of the time series. In this paper, we propose a novel framework for modeling incomplete time series, called a linear memory vector recurrent neural network (LIME-RNN), a recurrent neural network (RNN) with a learned linear combination of previous history states. The technique bears some similarity to residual networks and graph-based temporal dependency imputation. In particular, we introduce a linear memory vector [called the residual sum vector (RSV)] that integrates over previous hidden states of the RNN, and is used to fill in missing values. A new loss function is developed to train our model with time series in the presence of missing values in an end-to-end way. Our framework can handle imputation of both missing-at-random and consecutive missing inputs. Moreover, when conducting time-series prediction with missing values, LIME-RNN allows imputation and prediction simultaneously. We demonstrate the efficacy of the model via extensive experimental evaluation on univariate and multivariate time series, achieving state-of-the-art performance on synthetic and real-world data. The statistical results show that our model is significantly better than most existing time-series univariate or multivariate imputation methods.