Optimizing the echo state network with a binary particle swarm optimization algorithm

Statistical Analysis on Random Matrices of Echo State Network in PEMFC System’s Lifetime Prediction

The data-driven method of echo state network (ESN) has been successfully used in the proton exchange membrane fuel cell (PEMFC) system’s lifetime prediction area. Nevertheless, the uncertainties of the randomly generated input and internal weight matrices in ESN have not been reported yet. In view of this, an ensemble ESN structure is proposed in this paper to analyze the effects of random matrices from a statistical point of view. For each ESN, the particle swarm optimization (PSO) method is utilized to optimize the hyperparameters of the leaking rate, spectral radius, and regularization coefficient. The statistical results of each ensemble ESN are analyzed from 100 repeated tests whose weight matrices are generated randomly. The mean value of the ensemble ESN and a confidence interval (CI) of 95% are given during the long-term lifetime prediction. The effects of two different distribution shapes, i.e., uniform distribution and Gaussian distribution, are fully compared. Finally, the effects of the ensemble structure and two different distribution shapes are tested by three experimental datasets under steady-state, quasi-dynamic, and full dynamic operating conditions.

Robust Optimization and Validation of Echo State Networks for learning chaotic dynamics

An approach to the time-accurate prediction of chaotic solutions is by learning temporal patterns from data. Echo State Networks (ESNs), which are a class of Reservoir Computing, can accurately predict the chaotic dynamics well beyond the predictability time. Existing studies, however, also showed that small changes in the hyperparameters may markedly affect the network's performance. The overarching aim of this paper is to improve the robustness in the selection of hyperparameters in Echo State Networks for the time-accurate prediction of chaotic solutions. We define the robustness of a validation strategy as its ability to select hyperparameters that perform consistently between validation and test sets. The goal is three-fold. First, we investigate routinely used validation strategies. Second, we propose the Recycle Validation, and the chaotic versions of existing validation strategies, to specifically tackle the forecasting of chaotic systems. Third, we compare Bayesian optimization with the traditional grid search for optimal hyperparameter selection. Numerical tests are performed on prototypical nonlinear systems that have chaotic and quasiperiodic solutions, such as the Lorenz and Lorenz-96 systems, and the Kuznetsov oscillator. Both model-free and model-informed Echo State Networks are analysed. By comparing the networks' performance in learning chaotic (unpredictable) versus quasiperiodic (predictable) solutions, we highlight fundamental challenges in learning chaotic solutions. The proposed validation strategies, which are based on the dynamical systems properties of chaotic time series, are shown to outperform the state-of-the-art validation strategies. Because the strategies are principled - they are based on chaos theory such as the Lyapunov time - they can be applied to other Recurrent Neural Networks architectures with little modification. This work opens up new possibilities for the robust design and application of Echo State Networks, and Recurrent Neural Networks, to the time-accurate prediction of chaotic systems.

Chaotic Time Series Prediction Using a Novel Echo State Network Model with Input Reconstruction, Bayesian Ridge Regression and Independent Component Analysis

This paper presents a novel Echo State Network (ESN) model for chaotic time series prediction, which consists of three steps including input reconstruction, dimensionality reduction and regression. First, phase-space reconstruction is used to reconstruct the original ‘attractor’ of the input time series. Then, Independent Component Analysis (ICA) is used to identify independent components, reduce dimensionality and overcome multicollinearity problem of the reconstructed input matrix. Finally, Bayesian Ridge Regression provides accurate predictions thanks to its regularization effect to avoid over-fitting and its robustness to noise owing to its probabilistic strategy. Our experimental results show that our model significantly outperforms other ESN models in predicting both artificial and real-world chaotic time series.

BrainOS: A Novel Artificial Brain-Alike Automatic Machine Learning Framework

Human intelligence is constituted by a multitude of cognitive functions activated either directly or indirectly by external stimuli of various kinds. Computational approaches to the cognitive sciences and to neuroscience are partly premised on the idea that computational simulations of such cognitive functions and brain operations suspected to correspond to them can help to further uncover knowledge about those functions and operations, specifically, how they might work together. These approaches are also partly premised on the idea that empirical neuroscience research, whether following on from such a simulation (as indeed simulation and empirical research are complementary) or otherwise, could help us build better artificially intelligent systems. This is based on the assumption that principles by which the brain seemingly operate, to the extent that it can be understood as computational, should at least be tested as principles for the operation of artificial systems. This paper explores some of the principles of the brain that seem to be responsible for its autonomous, problem-adaptive nature. The brain operating system (BrainOS) explicated here is an introduction to ongoing work aiming to create a robust, integrated model, combining the connectionist paradigm underlying neural networks and the symbolic paradigm underlying much else of AI. BrainOS is an automatic approach that selects the most appropriate model based on the (a) input at hand, (b) prior experience (a history of results of prior problem solving attempts), and (c) world knowledge (represented in the symbolic way and used as a means to explain its approach). It is able to accept diverse and mixed input data types, process histories and objectives, extract knowledge and infer a situational context. BrainOS is designed to be efficient through its ability to not only choose the most suitable learning model but to effectively calibrate it based on the task at hand.

Improved Bat Algorithm Based on Multipopulation Strategy of Island Model for Solving Global Function Optimization Problem

The bat algorithm (BA) is a heuristic algorithm that globally optimizes by simulating the bat echolocation behavior. In order to improve the search performance and further improve the convergence speed and optimization precision of the bat algorithm, an improved algorithm based on chaotic map is introduced, and the improved bat algorithm of Levy flight search strategy and contraction factor is proposed. The optimal chaotic map operator is selected based on the simulation experiments results. Then, a multipopulation parallel bat algorithm based on the island model is proposed. Finally, the typical test functions are used to carry out the simulation experiments. The simulation results show that the proposed improved algorithm can effectively improve the convergence speed and optimization accuracy.

Online sequential echo state network with sparse RLS algorithm for time series prediction

Recently, the echo state networks (ESNs) have been widely used for time series prediction. To meet the demand of actual applications and avoid the overfitting issue, the online sequential ESN with sparse recursive least squares (OSESN-SRLS) algorithm is proposed. Firstly, the ℓ₀ and ℓ₁ norm sparsity penalty constraints of output weights are separately employed to control the network size. Secondly, the sparse recursive least squares (SRLS) algorithm and the subgradients technique are combined to estimate the output weight matrix. Thirdly, an adaptive selection mechanism for the ℓ₀ or ℓ₁ norm regularization parameter is designed. With the selected regularization parameter, it is proved that the developed SRLS shows comparable or better performance than the regular RLS. Furthermore, the convergence of OSESN-SRLS is theoretically analyzed to guarantee its effectiveness. Simulation results illustrate that the proposed OSESN-SRLS always outperforms other existing ESNs in terms of estimation accuracy and network compactness.

Optimized Parameter Settings of Binary Bat Algorithm for Solving Function Optimization Problems

The bat algorithm (BA) is a new bionic intelligent optimization algorithm to simulate the foraging behavior and the echolocation principle of the bats. The parameter initialization of the discussed binary bat algorithm (BBA) has important influence on the convergence speed, convergence precision, and good global searching ability of the BBA. The convergence speed and algorithm searching precision are determined by the pulse of loudness and pulse rate. The simulation experiments are carried out by using the six typical test functions to discuss this influence. The simulation results show that the convergence speed of the BBA is relatively sensitive to the setting of the algorithm parameters. The convergence precision reduces when increasing the rate of bat transmitted pulse alone and the convergence speed increases the launch loudness alone. The proper combination of BBA parameters (the rate of bat transmitted pulse and the launch loudness) can flexibly improve the algorithm’s convergence velocity and improve the accuracy of the searched solutions.