Deep belief echo-state network and its application to time series prediction

Hierarchical Echo State Network With Sparse Learning: A Method for Multidimensional Chaotic Time Series Prediction

Echo state network (ESN), a type of special recurrent neural network with a large-scale randomly fixed hidden layer (called a reservoir) and an adaptable linear output layer, has been widely employed in the field of time series analysis and modeling. However, when tackling the problem of multidimensional chaotic time series prediction, due to the randomly generated rules for input and reservoir weights, not only the representation of valuable variables is enriched but also redundant and irrelevant information is accumulated inevitably. To remove the redundant components, reduce the approximate collinearity among echo-state information, and improve the generalization and stability, a new method called hierarchical ESN with sparse learning (HESN-SL) is proposed. The HESN-SL mines and captures the latent evolution patterns hidden from the dynamic system by means of layer-by-layer processing in stacked reservoirs, and leverage monotone accelerated proximal gradient algorithm to train a sparse output layer with variable selection capability. Meanwhile, we further prove that the HESN-SL satisfies the echo state property, which guarantees the stability and convergence of the proposed model when applied to time series prediction. Experimental results on two synthetic chaotic systems and a real-world meteorological dataset illustrate the proposed HESN-SL outperforms both original ESN and existing hierarchical ESN-based models for multidimensional chaotic time series prediction.

Deep belief improved bidirectional LSTM for multivariate time series forecasting

Multivariate time series (MTS) play essential roles in daily life because most real-world time series datasets are multivariate and rich in time-dependent information. Traditional forecasting methods for MTS are time-consuming and filled with complicated limitations. One efficient method being explored within the dynamical systems is the extended short-term memory networks (LSTMs). However, existing MTS models only partially use the hidden spatial relationship as effectively as LSTMs. Shallow LSTMs are inadequate in extracting features from high-dimensional MTS; however, the multilayer bidirectional LSTM (BiLSTM) can learn more MTS features in both directions. This study tries to generate a novel and improved BiLSTM network (DBI-BiLSTM) based on a deep belief network (DBN), bidirectional propagation technique, and a chained structure. The deep structures are constructed by a DBN layer and multiple stacked BiLSTM layers, which increase the feature representation of DBI-BiLSTM and allow for the model to further learn the extended features in two directions. First, the input is processed by DBN to obtain comprehensive features. Then, the known features, divided into clusters based on a global sensitivity analysis method, are used as the inputs of every BiLSTM layer. Meanwhile, the previous outputs of the shallow layer are combined with the clustered features to reconstitute new input signals for the next deep layer. Four experimental real-world time series datasets illustrate our one-step-ahead prediction performance. The simulating results confirm that the DBI-BiLSTM not only outperforms the traditional shallow artificial neural networks (ANNs), deep LSTMs, and some recently improved LSTMs, but also learns more features of the MTS data. As compared with conventional LSTM, the percentage improvement of DBI-BiLSTM on the four MTS datasets is 85.41, 75.47, 61.66 and 30.72%, respectively.

A novel time series prediction method based on pooling compressed sensing echo state network and its application in stock market

In the prediction of time series, the echo state network (ESN) exhibits exclusive strengths and a unique training structure. Based on ESN model, a pooling activation algorithm consisting noise value and adjusted pooling algorithm is proposed to enrich the update strategy of the reservoir layer in ESN. The algorithm optimizes the distribution of reservoir layer nodes. And the nodes set will be more matched to the characteristics of the data. In addition, we introduce a more efficient and accurate compressed sensing technique based on the existing research. The novel compressed sensing technique reduces the amount of spatial computation of methods. The ESN model based on the above two techniques overcomes the limitations in traditional prediction. In the experimental part, the model is validated with different chaotic time series as well as multiple stocks, and the method shows its efficiency and accuracy in prediction.

Estimating the incubated river water quality indicator based on machine learning and deep learning paradigms: BOD5 Prediction

As an indicator measured by incubating organic material from water samples in rivers, the most typical characteristic of water quality items is biochemical oxygen demand (BOD₅) concentration, which is a stream pollutant with an extreme circumstance of organic loading and controlling aquatic behavior in the eco-environment. Leading monitoring approaches including machine leaning and deep learning have been evolved for a correct, trustworthy, and low-cost prediction of BOD₅ concentration. The addressed research investigated the efficiency of three standalone models including machine learning (extreme learning machine (ELM) and support vector regression (SVR)) and deep learning (deep echo state network (Deep ESN)). In addition, the novel double-stage synthesis models (wavelet-extreme learning machine (Wavelet-ELM), wavelet-support vector regression (Wavelet-SVR), and wavelet-deep echo state network (Wavelet-Deep ESN)) were developed by integrating wavelet transformation (WT) with the different standalone models. Five input associations were supplied for evaluating standalone and double-stage synthesis models by determining diverse water quantity and quality items. The proposed models were assessed using the coefficient of determination (R²), Nash-Sutcliffe (NS) efficiency, and root mean square error (RMSE). The significance of addressed research can be found from the overall outcomes that the predictive accuracy of double-stage synthesis models were not always superior to that of standalone models. Overall results showed that the SVR with 3^th distribution (NS = 0.915) and the Wavelet-SVR with 4^th distribution (NS = 0.915) demonstrated more correct outcomes for predicting BOD₅ concentration compared to alternative models at Hwangji station, and the Wavelet-SVR with 4^th distribution (NS = 0.917) was judged to be the most superior model at Toilchun station. In most cases for predicting BOD₅ concentration, the novel double-stage synthesis models can be utilized for efficient and organized data administration and regulation of water pollutants on both stations, South Korea.

Chain-Structure Echo State Network With Stochastic Optimization: Methodology and Application

In this article, a chain-structure echo state network (CESN) with stacked subnetwork modules is newly proposed as a new kind of deep recurrent neural network for multivariate time series prediction. Motivated by the philosophy of "divide and conquer," the related input vectors are first divided into clusters, and the final output results of CESN are then integrated by successively learning the predicted values of each clustered variable. Network structure, mathematical model, training mechanism, and stability analysis are, respectively, studied for the proposed CESN. In the training stage, least-squares regression is first used to pretrain the output weights in a module-by-module way, and stochastic local search (SLS) is developed to fine-tune network weights toward global optima. The loss function of CESN can be effectively reduced by SLS. To avoid overfitting, the optimization process is stopped when the validation error starts to increase. Finally, SLS-CESN is evaluated in chaos prediction benchmarks and real applications. Four different examples are given to verify the effectiveness and robustness of CESN and SLS-CESN.

LSTM Networks for Overcoming the Challenges Associated with Photovoltaic Module Maintenance in Smart Cities

Predictive maintenance is a field of research that has emerged from the need to improve the systems in place. This research focuses on controlling the degradation of photovoltaic (PV) modules in outdoor solar panels, which are exposed to a variety of climatic loads. Improved reliability, operation, and performance can be achieved through monitoring. In this study, a system capable of predicting the output power of a solar module was implemented. It monitors different parameters and uses automatic learning techniques for prediction. Its use improved reliability, operation, and performance. On the other hand, automatic learning algorithms were evaluated with different metrics in order to optimize and find the best configuration that provides an optimal solution to the problem. With the aim of increasing the share of renewable energy penetration, an architectural proposal based on Edge Computing was included to implement the proposed model into a system. The proposed model is designated for outdoor predictions and offers many advantages, such as monitoring of individual panels, optimization of system response, and speed of communication with the Cloud. The final objective of the work was to contribute to the smart Energy system concept, providing solutions for planning the entire energy system together with the identification of suitable energy infrastructure designs and operational strategies.

DERN: Deep Ensemble Learning Model for Short- and Long-Term Prediction of Baltic Dry Index

The Baltic Dry Index (BDI) is a commonly utilized indicator of global shipping and trade activity. It influences stakeholders’ and ship-owners’ decisions respecting investments, chartering, operational plans, and export and import activities. Accurate prediction of the BDI is very challenging due to its volatility, non-stationarity, and complexity. To help stakeholders and ship-owners make sound short- and long-term maritime business decisions and avoid market risk, we performed short- and long-term predictions of BDI using an ensemble deep-learning approach. In this study, we propose to apply recurrent neural network models for BDI prediction. The state-of-the-art of sequential deep-learning models such as RNN, LSTM, and GRU are employed to predict one- and multi-step-ahead BDI values. In order to increase the accuracy, we assemble the models. In experiments, we compared our results with those of traditional methods such as ARIMA and MLP. The results showed that our proposed method outperforms ARIMA, MLP, RNN, LSTM, and GRU in both short- and long-term prediction of BDI.

A sparse deep belief network with efficient fuzzy learning framework

Deep belief network (DBN) is one of the most feasible ways to realize deep learning (DL) technique, and it has been attracting more and more attentions in nonlinear system modeling. However, DBN cannot provide satisfactory results in learning speed, modeling accuracy and robustness, which is mainly caused by dense representation and gradient diffusion. To address these problems and promote DBN's development in cross-models, we propose a Sparse Deep Belief Network with Fuzzy Neural Network (SDBFNN) for nonlinear system modeling. In this novel framework, the sparse DBN is considered as a pre-training technique to realize fast weight-initialization and to obtain feature vectors. It can balance the dense representation to improve its robustness. A fuzzy neural network is developed for supervised modeling so as to eliminate the gradient diffusion. Its input happens to be the obtained feature vector. As a novel cross-model, SDBFNN combines the advantages of both pre-training technique and fuzzy neural network to improve modeling capability. Its convergence is also analyzed as well. A benchmark problem and a practical problem in wastewater treatment are conducted to demonstrate the superiority of SDBFNN. The extensive experimental results show that SDBFNN achieves better performance than the existing methods in learning speed, modeling accuracy and robustness.

Hourly Urban Water Demand Forecasting Using the Continuous Deep Belief Echo State Network

Effective and accurate water demand prediction is an important part of the optimal scheduling of a city water supply system. A novel deep architecture model called the continuous deep belief echo state network (CDBESN) is proposed in this study for the prediction of hourly urban water demand. The CDBESN model uses a continuous deep belief network (CDBN) as the feature extraction algorithm and an echo state network (ESN) as the regression algorithm. The new architecture can model actual water demand data with fast convergence and global optimization ability. The prediction capacity of the CDBESN model is tested using historical hourly water demand data obtained from an urban waterworks in Zhuzhou, China. The performance of the proposed model is compared with those of ESN, continuous deep belief neural network, and support vector regression models. The correlation coefficient (r2), normalized root-mean-square error (NRMSE), and mean absolute percentage error (MAPE) are adopted as assessment criteria. Forecasting results obtained in the testing stage indicate that the CDBESN model has the largest r2 value of 0.995912 and the smallest NRMSE and MAPE values of 0.027163 and 2.469419, respectively. The prediction accuracy of the proposed model clearly outperforms those of the models it is compared with due to the good feature extraction ability of CDBN and the excellent feature learning ability of ESN.

Quantitative structure-property relationship (QSPR) models for predicting the physicochemical properties of polychlorinated biphenyls (PCBs) using deep belief network

Octanol/water partition coefficient (log P), octanol/air partition coefficient (log K_OA) and bioconcentration factor (log BCF) are important physiochemical properties of organic substances. Quantitative structure-property relationship (QSPR) models are a promising alternative method of reducing and replacing experimental steps in determination of log P, log K_OA and log BCF. In the current study, we propose a new QSPR model based on a deep belief network (DBN) to predict the physicochemical properties of polychlorinated biphenyls (PCBs). The prediction accuracy of the proposed model was compared to the results of previous reported models. The predictive ability of the DBN model, validated with a test set, is clearly superior to the other models. All results showed that the proposed model is robust and satisfactory, and can effectively predict the physiochemical properties of PCBs without highly reliable experimental values.