Separation of chaotic signals by reservoir computing

Boosting reservoir computer performance with multiple delays

Time delays play a significant role in dynamical systems, as they affect their transient behavior and the dimensionality of their attractors. The number, values, and spacing of these time delays influences the eigenvalues of a nonlinear delay-differential system at its fixed point. Here we explore a multidelay system as the core computational element of a reservoir computer making predictions on its input in the usual regime close to fixed point instability. Variations in the number and separation of time delays are first examined to determine the effect of such parameters of the delay distribution on the effectiveness of time-delay reservoirs for nonlinear time series prediction. We demonstrate computationally that an optoelectronic device with multiple different delays can improve the mapping of scalar input into higher-dimensional dynamics, and thus its memory and prediction capabilities for input time series generated by low- and high-dimensional dynamical systems. In particular, this enhances the suitability of such reservoir computers for predicting input data with temporal correlations. Additionally, we highlight the pronounced harmful resonance condition for reservoir computing when using an electro-optic oscillator model with multiple delays. We illustrate that the resonance point may shift depending on the task at hand, such as cross prediction or multistep ahead prediction, in both single delay and multiple delay cases.

Global forecasts in reservoir computers

A reservoir computer is a machine learning model that can be used to predict the future state(s) of time-dependent processes, e.g., dynamical systems. In practice, data in the form of an input-signal are fed into the reservoir. The trained reservoir is then used to predict the future state of this signal. We develop a new method for not only predicting the future dynamics of the input-signal but also the future dynamics starting at an arbitrary initial condition of a system. The systems we consider are the Lorenz, Rossler, and Thomas systems restricted to their attractors. This method, which creates a global forecast, still uses only a single input-signal to train the reservoir but breaks the signal into many smaller windowed signals. We examine how well this windowed method is able to forecast the dynamics of a system starting at an arbitrary point on a system's attractor and compare this to the standard method without windows. We find that the standard method has almost no ability to forecast anything but the original input-signal while the windowed method can capture the dynamics starting at most points on an attractor with significant accuracy.

Learning unidirectional coupling using an echo-state network

Reservoir Computing has found many potential applications in the field of complex dynamics. In this article, we explore the exceptional capability of the echo-state network (ESN) model to make it learn a unidirectional coupling scheme from only a few time series data of the system. We show that, once trained with a few example dynamics of a drive-response system, the machine is able to predict the response system's dynamics for any driver signal with the same coupling. Only a few time series data of an A-B type drive-response system in training is sufficient for the ESN to learn the coupling scheme. After training, even if we replace drive system A with a different system C, the ESN can reproduce the dynamics of response system B using the dynamics of new drive system C only.

Accurate separation of mixed high-dimension optical-chaotic signals using optical reservoir computing based on optically pumped VCSELs

In this work, with the mixing fractions being known in advance or unknown, the schemes and theories for the separations of two groups of the mixed optical chaotic signals are proposed in detail, using the VCSEL-based reservoir computing (RC) systems. Here, two groups of the mixed optical chaotic signals are linearly combined with many beams of the chaotic x-polarization components (X-PCs) and Y-PCs emitted by the optically pumped spin-VCSELs operation alone. Two parallel reservoirs are performed by using the chaotic X-PC and Y-PC output by the optically pumped spin-VCSEL with both optical feedback and optical injection. Moreover, we further demonstrate the separation performances of the mixed chaotic signal linearly combined with no more than three beams of the chaotic X-PC or Y-PC. We find that two groups of the mixed optical chaos signals can be effectively separated by using two reservoirs in single RC system based on optically pumped Spin-VCSEL and their corresponding separated errors characterized by the training errors are no more than 0.093, when the mixing fractions are known as a certain value in advance. If the mixing fractions are unknown, we utilize two cascaded RC systems based on optically pumped Spin-VCSELs to separate each group of the mixed optical signals. The mixing fractions can be accurate predicted by using two parallel reservoirs in the first RC system. Based on the values of the predictive mixing fractions, two groups of the mixed optical chaos signals can be effectively separated by utilizing two parallel reservoirs in the second RC system, and their separated errors also are no more than 0.093. In the same way, the mixed optical chaos signal linearly superimposed with more than three beams of optical chaotic signals can be effectively separated. The method and idea for separation of complex optical chaos signals proposed by this paper may provide an impact to development of novel principles of multiple access and demultiplexing in multi-channel chaotic cryptography communication.

Time series reconstructing using calibrated reservoir computing

Reservoir computing, a new method of machine learning, has recently been used to predict the state evolution of various chaotic dynamic systems. It has significant advantages in terms of training cost and adjusted parameters; however, the prediction length is limited. For classic reservoir computing, the prediction length can only reach five to six Lyapunov times. Here, we modified the method of reservoir computing by adding feedback, continuous or discrete, to “calibrate” the input of the reservoir and then reconstruct the entire dynamic systems. The reconstruction length appreciably increased and the training length obviously decreased. The reconstructing of dynamical systems is studied in detail under this method. The reconstruction can be significantly improved both in length and accuracy. Additionally, we summarized the effect of different kinds of input feedback. The more it interacts with others in dynamical equations, the better the reconstructions. Nonlinear terms can reveal more information than linear terms once the interaction terms are equal. This method has proven effective via several classical chaotic systems. It can be superior to traditional reservoir computing in reconstruction, provides new hints in computing promotion, and may be used in some real applications.

Reducing echo state network size with controllability matrices

Echo state networks are a fast training variant of recurrent neural networks excelling at approximating nonlinear dynamical systems and time series prediction. These machine learning models act as nonlinear fading memory filters. While these models benefit from quick training and low complexity, computation demands from a large reservoir matrix are a bottleneck. Using control theory, a reduced size replacement reservoir matrix is found. Starting from a large, task-effective reservoir matrix, we form a controllability matrix whose rank indicates the active sub-manifold and candidate replacement reservoir size. Resulting time speed-ups and reduced memory usage come with minimal error increase to chaotic climate reconstruction or short term prediction. Experiments are performed on simple time series signals and the Lorenz-1963 and Mackey-Glass complex chaotic signals. Observing low error models shows variation of active rank and memory along a sequence of predictions.

Data driven soliton solution of the nonlinear Schrödinger equation with certain P T-symmetric potentials via deep learning

We investigate the physics informed neural network method, a deep learning approach, to approximate soliton solution of the nonlinear Schrödinger equation with parity time symmetric potentials. We consider three different parity time symmetric potentials, namely, Gaussian, periodic, and Rosen-Morse potentials. We use the physics informed neural network to solve the considered nonlinear partial differential equation with the above three potentials. We compare the predicted result with the actual result and analyze the ability of deep learning in solving the considered partial differential equation. We check the ability of deep learning in approximating the soliton solution by taking the squared error between real and predicted values. Further, we examine the factors that affect the performance of the considered deep learning method with different activation functions, namely, ReLU, sigmoid, and tanh. We also use a new activation function, namely, sech, which is not used in the field of deep learning, and analyze whether this new activation function is suitable for the prediction of soliton solution of the nonlinear Schrödinger equation for the aforementioned parity time symmetric potentials. In addition to the above, we present how the network's structure and the size of the training data influence the performance of the physics informed neural network. Our results show that the constructed deep learning model successfully approximates the soliton solution of the considered equation with high accuracy.

A tighter generalization bound for reservoir computing

While reservoir computing (RC) has demonstrated astonishing performance in many practical scenarios, the understanding of its capability for generalization on previously unseen data is limited. To address this issue, we propose a novel generalization bound for RC based on the empirical Rademacher complexity under the probably approximately correct learning framework. Note that the generalization bound for the RC is derived in terms of the model hyperparameters. For this reason, it can explore the dependencies of the generalization bound for RC on its hyperparameters. Compared with the existing generalization bound, our generalization bound for RC is tighter, which is verified by numerical experiments. Furthermore, we study the generalization bound for the RC corresponding to different reservoir graphs, including directed acyclic graph (DAG) and Erdős-R e´nyi undirected random graph (ER graph). Specifically, the generalization bound for the RC whose reservoir graph is designated as a DAG can be refined by leveraging the structural property (i.e., the longest path length) of the DAG. Finally, both theoretical and experimental findings confirm that the generalization bound for the RC of a DAG is lower and less sensitive to the model hyperparameters than that for the RC of an ER graph.

Reservoir time series analysis: Using the response of complex dynamical systems as a universal indicator of change

We present the idea of reservoir time series analysis (RTSA), a method by which the state space representation generated by a reservoir computing (RC) model can be used for time series analysis. We discuss the motivation for this with reference to the characteristics of RC and present three ad hoc methods for generating representative features from the reservoir state space. We then develop and implement a hypothesis test to assess the capacity of these features to distinguish signals from systems with varying parameters. In comparison to a number of benchmark approaches (statistical, Fourier, phase space, and recurrence analysis), we are able to show significant, generalized accuracy across the proposed RTSA features that surpasses the benchmark methods. Finally, we briefly present an application for bearing fault distinction to motivate the use of RTSA in application.

Parameter extraction with reservoir computing: Nonlinear time series analysis and application to industrial maintenance

We study the task of determining parameters of dynamical systems from their time series using variations of reservoir computing. Averages of reservoir activations yield a static set of random features that allows us to separate different parameter values. We study such random feature models in the time and frequency domain. For the Lorenz and Rössler systems throughout stable and chaotic regimes, we achieve accurate and robust parameter extraction. For vibration data of centrifugal pumps, we find a significant ability to recover the operating regime. While the time domain models achieve higher performance for the numerical systems, the frequency domain models are superior in the application context.

Revisiting the memory capacity in reservoir computing of directed acyclic network

Reservoir computing (RC) is an attractive area of research by virtue of its potential for hardware implementation and low training cost. An intriguing research direction in this field is to interpret the underlying dynamics of an RC model by analyzing its short-term memory property, which can be quantified by the global index: memory capacity (MC). In this paper, the global MC of the RC whose reservoir network is specified as a directed acyclic network (DAN) is examined, and first we give that its global MC is theoretically bounded by the length of the longest path of the reservoir DAN. Since the global MC is technically influenced by the model hyperparameters, the dependency of the MC on the hyperparameters of this RC is then explored in detail. In the further study, we employ the improved conventional network embedding method (i.e., struc2vec) to mine the underlying memory community in the reservoir DAN, which can be regarded as the cluster of reservoir nodes with the same memory profile. Experimental results demonstrate that such a memory community structure can provide a concrete interpretation of the global MC of this RC. Finally, the clustered RC is proposed by exploiting the detected memory community structure of DAN, where its prediction performance is verified to be enhanced with lower training cost compared with other RC models on several chaotic time series benchmarks.

Multifunctionality in a reservoir computer

Multifunctionality is a well observed phenomenological feature of biological neural networks and considered to be of fundamental importance to the survival of certain species over time. These multifunctional neural networks are capable of performing more than one task without changing any network connections. In this paper, we investigate how this neurological idiosyncrasy can be achieved in an artificial setting with a modern machine learning paradigm known as "reservoir computing." A training technique is designed to enable a reservoir computer to perform tasks of a multifunctional nature. We explore the critical effects that changes in certain parameters can have on the reservoir computers' ability to express multifunctionality. We also expose the existence of several "untrained attractors"; attractors that dwell within the prediction state space of the reservoir computer were not part of the training. We conduct a bifurcation analysis of these untrained attractors and discuss the implications of our results.

Mapping topological characteristics of dynamical systems into neural networks: A reservoir computing approach

Significant advances have recently been made in modeling chaotic systems with the reservoir computing approach, especially for prediction. We find that although state prediction of the trained reservoir computer will gradually deviate from the actual trajectory of the original system, the associated geometric features remain invariant. Specifically, we show that the typical geometric metrics including the correlation dimension, the multiscale entropy, and the memory effect are nearly identical between the trained reservoir computer and its learned chaotic systems. We further demonstrate this fact on a broad range of chaotic systems ranging from discrete and continuous chaotic systems to hyperchaotic systems. Our findings suggest that the successfully reservoir computer may be topologically conjugate to an observed dynamical system.

Invertible generalized synchronization: A putative mechanism for implicit learning in neural systems

Regardless of the marked differences between biological and artificial neural systems, one fundamental similarity is that they are essentially dynamical systems that can learn to imitate other dynamical systems whose governing equations are unknown. The brain is able to learn the dynamic nature of the physical world via experience; analogously, artificial neural systems such as reservoir computing networks (RCNs) can learn the long-term behavior of complex dynamical systems from data. Recent work has shown that the mechanism of such learning in RCNs is invertible generalized synchronization (IGS). Yet, whether IGS is also the mechanism of learning in biological systems remains unclear. To shed light on this question, we draw inspiration from features of the human brain to propose a general and biologically feasible learning framework that utilizes IGS. To evaluate the framework's relevance, we construct several distinct neural network models as instantiations of the proposed framework. Regardless of their particularities, these neural network models can consistently learn to imitate other dynamical processes with a biologically feasible adaptation rule that modulates the strength of synapses. Further, we observe and theoretically explain the spontaneous emergence of four distinct phenomena reminiscent of cognitive functions: (i) learning multiple dynamics; (ii) switching among the imitations of multiple dynamical systems, either spontaneously or driven by external cues; (iii) filling-in missing variables from incomplete observations; and (iv) deciphering superimposed input from different dynamical systems. Collectively, our findings support the notion that biological neural networks can learn the dynamic nature of their environment through the mechanism of IGS.