Impact of time-history terms on reservoir dynamics and prediction accuracy in echo state networks

The echo state network (ESN) is an excellent machine learning model for processing time-series data. This model, utilising the response of a recurrent neural network, called a reservoir, to input signals, achieves high training efficiency. Introducing time-history terms into the neuron model of the reservoir is known to improve the time-series prediction performance of ESN, yet the reasons for this improvement have not been quantitatively explained in terms of reservoir dynamics characteristics. Therefore, we hypothesised that the performance enhancement brought about by time-history terms could be explained by delay capacity, a recently proposed metric for assessing the memory performance of reservoirs. To test this hypothesis, we conducted comparative experiments using ESN models with time-history terms, namely leaky integrator ESNs (LI-ESN) and chaotic echo state networks (ChESN). The results suggest that compared with ESNs without time-history terms, the reservoir dynamics of LI-ESN and ChESN can maintain diversity and stability while possessing higher delay capacity, leading to their superior performance. Explaining ESN performance through dynamical metrics are crucial for evaluating the numerous ESN architectures recently proposed from a general perspective and for the development of more sophisticated architectures, and this study contributes to such efforts.

Echo state networks for the recognition of type 1 Brugada syndrome from conventional 12-LEAD ECG

Artificial Intelligence (AI) applications and Machine Learning (ML) methods have gained much attention in recent years for their ability to automatically detect patterns in data without being explicitly taught rules. Specific features characterise the ECGs of patients with Brugada Syndrome (BrS); however, there is still ambiguity regarding the correct diagnosis of BrS and its differentiation from other pathologies. This work presents an application of Echo State Networks (ESN) in the Recurrent Neural Networks (RNN) class for diagnosing BrS from the ECG time series. 12-lead ECGs were obtained from patients with a definite clinical diagnosis of spontaneous BrS Type 1 pattern (Group A), patients who underwent provocative pharmacological testing to induce BrS type 1 pattern, which resulted in positive (Group B) or negative (Group C), and control subjects (Group D). One extracted beat in the V2 lead was used as input, and the dataset was used to train and evaluate the ESN model using a double cross-validation approach. ESN performance was compared with that of 4 cardiologists trained in electrophysiology. The model performance was assessed in the dataset, with a correct global diagnosis observed in 91.5 % of cases compared to clinicians (88.0 %). High specificity (94.5 %), sensitivity (87.0 %) and AUC (94.7 %) for BrS recognition by ESN were observed in Groups A + B vs. C + D. Our results show that this ML model can discriminate Type 1 BrS ECGs with high accuracy comparable to expert clinicians. Future availability of larger datasets may improve the model performance and increase the potential of the ESN as a clinical support system tool for daily clinical practice.

A novel time series prediction method based on pooling compressed sensing echo state network and its application in stock market

In the prediction of time series, the echo state network (ESN) exhibits exclusive strengths and a unique training structure. Based on ESN model, a pooling activation algorithm consisting noise value and adjusted pooling algorithm is proposed to enrich the update strategy of the reservoir layer in ESN. The algorithm optimizes the distribution of reservoir layer nodes. And the nodes set will be more matched to the characteristics of the data. In addition, we introduce a more efficient and accurate compressed sensing technique based on the existing research. The novel compressed sensing technique reduces the amount of spatial computation of methods. The ESN model based on the above two techniques overcomes the limitations in traditional prediction. In the experimental part, the model is validated with different chaotic time series as well as multiple stocks, and the method shows its efficiency and accuracy in prediction.

Stability analysis of chaotic systems from data

The prediction of the temporal dynamics of chaotic systems is challenging because infinitesimal perturbations grow exponentially. The analysis of the dynamics of infinitesimal perturbations is the subject of stability analysis. In stability analysis, we linearize the equations of the dynamical system around a reference point and compute the properties of the tangent space (i.e. the Jacobian). The main goal of this paper is to propose a method that infers the Jacobian, thus, the stability properties, from observables (data). First, we propose the echo state network (ESN) with the Recycle validation as a tool to accurately infer the chaotic dynamics from data. Second, we mathematically derive the Jacobian of the echo state network, which provides the evolution of infinitesimal perturbations. Third, we analyse the stability properties of the Jacobian inferred from the ESN and compare them with the benchmark results obtained by linearizing the equations. The ESN correctly infers the nonlinear solution and its tangent space with negligible numerical errors. In detail, we compute from data only (i) the long-term statistics of the chaotic state; (ii) the covariant Lyapunov vectors; (iii) the Lyapunov spectrum; (iv) the finite-time Lyapunov exponents; (v) and the angles between the stable, neutral, and unstable splittings of the tangent space (the degree of hyperbolicity of the attractor). This work opens up new opportunities for the computation of stability properties of nonlinear systems from data, instead of equations.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s11071-023-08285-1.

Semi-supervised echo state network with temporal-spatial graph regularization for dynamic soft sensor modeling of industrial processes

Echo state network (ESN) has been successfully applied to industrial soft sensor field because of its strong nonlinear and dynamic modeling capability. Nevertheless, the traditional ESN is intrinsically a supervised learning technique, which only depends on labeled samples, but omits a large number of unlabeled samples. In order to eliminate this limitation, this work proposes a semi-supervised ESN method assisted by a temporal-spatial graph regularization (TSG-SSESN) for constructing soft sensor model with all the available samples. Firstly, the traditional supervised ESN is enhanced to construct the semi-supervised ESN (SSESN) model by integrating both unlabeled and labeled samples in the reservoir computing procedure. The SSESN computes the reservoir states under high sampling rate for better process dynamic information mining. Furthermore, the SSESN's output optimization objective is modified by applying the local adjacency graph of all training samples as a regularization term. Especially, in view of the dynamic data characteristic, a temporal-spatial graph is constructed by considering both the temporal relationship and the spatial distances. The applications to a debutanizer column process and a wastewater treatment plant demonstrate that the TSG-SSESN model can build much smoother model and has better generalization capability than the basic ESN models in terms of soft sensor prediction results.

An internet traffic classification method based on echo state network and improved salp swarm algorithm

Internet traffic classification is fundamental to network monitoring, service quality and security. In this paper, we propose an internet traffic classification method based on the Echo State Network (ESN). To enhance the identification performance, we improve the Salp Swarm Algorithm (SSA) to optimize the ESN. At first, Tent mapping with reversal learning, polynomial operator and dynamic mutation strategy are introduced to improve the SSA, which enhances its optimization performance. Then, the advanced SSA are utilized to optimize the hyperparameters of the ESN, including the size of the reservoir, sparse degree, spectral radius and input scale. Finally, the optimized ESN is adopted to classify Internet traffic. The simulation results show that the proposed ESN-based method performs much better than other traditional machine learning algorithms in terms of per-class metrics and overall accuracy.

Fading memory echo state networks are universal

Echo state networks (ESNs) have been recently proved to be universal approximants for input/output systems with respect to various L^p-type criteria. When 1≤p<∞, only p-integrability hypotheses need to be imposed, while in the case p=∞ a uniform boundedness hypotheses on the inputs is required. This note shows that, in the last case, a universal family of ESNs can be constructed that contains exclusively elements that have the echo state and the fading memory properties. This conclusion could not be drawn with the results and methods available so far in the literature.

Decoding electroencephalographic signals for direction in brain-computer interface using echo state network and Gaussian readouts

BACKGROUND

Noninvasive brain-computer interfaces (BCI) for movement control via an electroencephalogram (EEG) have been extensively investigated. However, most previous studies decoded user intention for movement directions based on sensorimotor rhythms during motor imagery. BCI systems based on mapping imagery movement of body parts (e.g., left or right hands) to movement directions (left or right directional movement of a machine or cursor) are less intuitive and less convenient due to the complex training procedures. Thus, direct decoding methods for detecting user intention about movement directions are urgently needed.

METHODS

Here, we describe a novel direct decoding method for user intention about the movement directions using the echo state network and Gaussian readouts. Importantly parameters in the network were optimized using the genetic algorithm method to achieve better decoding performance. We tested the decoding performance of this method with four healthy subjects and an inexpensive wireless EEG system containing 14 channels and then compared the performance outcome with that of a conventional machine learning method.

RESULTS

We showed that this decoding method successfully classified eight directions of intended movement (approximately 95% of an accuracy).

CONCLUSIONS

We suggest that the echo state network and Gaussian readouts can be a useful decoding method to directly read user intention of movement directions even using an inexpensive and portable EEG system.

Optimal modularity and memory capacity of neural reservoirs

The neural network is a powerful computing framework that has been exploited by biological evolution and by humans for solving diverse problems. Although the computational capabilities of neural networks are determined by their structure, the current understanding of the relationships between a neural network's architecture and function is still primitive. Here we reveal that a neural network's modular architecture plays a vital role in determining the neural dynamics and memory performance of the network of threshold neurons. In particular, we demonstrate that there exists an optimal modularity for memory performance, where a balance between local cohesion and global connectivity is established, allowing optimally modular networks to remember longer. Our results suggest that insights from dynamical analysis of neural networks and information-spreading processes can be leveraged to better design neural networks and may shed light on the brain's modular organization.

Atrial fibrillation frequency tracking in ambulatory ECG signals: The significance of signal quality assessment

An approach to atrial fibrillation (AF) frequency tracking in long-term ambulatory ECG recordings is presented, comprising f-wave extraction, dominant atrial frequency (DAF) tracking, and signal quality assessment. Since poor signal quality is commonly encountered in ambulatory monitoring, a recently proposed index is employed to assess f-wave signal quality in a database containing 38 patients with permanent AF. The index ensures that DAF outliers, typically associated with poor-quality segments, are excluded from further analysis. 40% of all 5-s signal segments were excluded from the database due to poor quality. The exclusion of DAF outliers significantly reduces the standard deviation of the frequency estimates (p≤0.01), allowing more reliable evaluation of the difference between day- and night-time DAF. The results show that signal quality assessment plays a central role in DAF tracking, and therefore should be employed in ambulatory monitoring.