Extensive chaos in the Lorenz-96 model

Precise Traits from Sloppy Components: Perception and the Origin of Phenotypic Response

Organisms perceive their environment and respond. The origin of perception-response traits presents a puzzle. Perception provides no value without response. Response requires perception. Recent advances in machine learning may provide a solution. A randomly connected network creates a reservoir of perceptive information about the recent history of environmental states. In each time step, a relatively small number of inputs drives the dynamics of the relatively large network. Over time, the internal network states retain a memory of past inputs. To achieve a functional response to past states or to predict future states, a system must learn only how to match states of the reservoir to the target response. In the same way, a random biochemical or neural network of an organism can provide an initial perceptive basis. With a solution for one side of the two-step perception-response challenge, evolving an adaptive response may not be so difficult. Two broader themes emerge. First, organisms may often achieve precise traits from sloppy components. Second, evolutionary puzzles often follow the same outlines as the challenges of machine learning. In each case, the basic problem is how to learn, either by artificial computational methods or by natural selection.

Stability analysis of chaotic systems from data

The prediction of the temporal dynamics of chaotic systems is challenging because infinitesimal perturbations grow exponentially. The analysis of the dynamics of infinitesimal perturbations is the subject of stability analysis. In stability analysis, we linearize the equations of the dynamical system around a reference point and compute the properties of the tangent space (i.e. the Jacobian). The main goal of this paper is to propose a method that infers the Jacobian, thus, the stability properties, from observables (data). First, we propose the echo state network (ESN) with the Recycle validation as a tool to accurately infer the chaotic dynamics from data. Second, we mathematically derive the Jacobian of the echo state network, which provides the evolution of infinitesimal perturbations. Third, we analyse the stability properties of the Jacobian inferred from the ESN and compare them with the benchmark results obtained by linearizing the equations. The ESN correctly infers the nonlinear solution and its tangent space with negligible numerical errors. In detail, we compute from data only (i) the long-term statistics of the chaotic state; (ii) the covariant Lyapunov vectors; (iii) the Lyapunov spectrum; (iv) the finite-time Lyapunov exponents; (v) and the angles between the stable, neutral, and unstable splittings of the tangent space (the degree of hyperbolicity of the attractor). This work opens up new opportunities for the computation of stability properties of nonlinear systems from data, instead of equations.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s11071-023-08285-1.

Neural Granger Causality

While most classical approaches to Granger causality detection assume linear dynamics, many interactions in real-world applications, like neuroscience and genomics, are inherently nonlinear. In these cases, using linear models may lead to inconsistent estimation of Granger causal interactions. We propose a class of nonlinear methods by applying structured multilayer perceptrons (MLPs) or recurrent neural networks (RNNs) combined with sparsity-inducing penalties on the weights. By encouraging specific sets of weights to be zero-in particular, through the use of convex group-lasso penalties-we can extract the Granger causal structure. To further contrast with traditional approaches, our framework naturally enables us to efficiently capture long-range dependencies between series either via our RNNs or through an automatic lag selection in the MLP. We show that our neural Granger causality methods outperform state-of-the-art nonlinear Granger causality methods on the DREAM3 challenge data. This data consists of nonlinear gene expression and regulation time courses with only a limited number of time points. The successes we show in this challenging dataset provide a powerful example of how deep learning can be useful in cases that go beyond prediction on large datasets. We likewise illustrate our methods in detecting nonlinear interactions in a human motion capture dataset.

Enhanced Serpent algorithm using Lorenz 96 Chaos-based block key generation and parallel computing for RGB image encryption

This paper presents a new approach to enhance the security and performance of the Serpent algorithm. The main concepts of this approach is to generate a sub key for each block using Lorenz 96 chaos and then run the process of encryption and decryption in ECB parallel mode. The proposed method has been implemented in Java, openjdk version "11.0.11"; and for the analysis of the tested RGB images, Python 3.6 was used. Comprehensive experiments on widely used metrics demonstrate the effectiveness of the proposed method against differential attacks, brute force attacks and statistical attacks, while achieving superb results compared to related schemes. Moreover, the encryption quality, Shannon entropy, correlation coefficients, histogram analysis and differential analysis all accomplished affirmative results. Furthermore, the reduction in encryption/decryption time was over 61%. Moreover, the proposed method cipher was tested using the Statistical Test Suite (STS) recommended by the NIST and passed them all ensuring the randomness of the cipher output. Thus, the approach demonstrated the potential of the improved Serpent-ECB algorithm with Lorenz 96 chaos-based block key generation (BKG) and gave favorable results. Specifically, compared to existing encryption schemes, it proclaimed its effectiveness.

Response and flux of information in extended nonequilibrium dynamics

It is well known that entropy production is a proxy to the detection of nonequilibrium, i.e., of the absence of detailed balance; however, due to the global character of this quantity, its knowledge does not allow to identify spatial currents or fluxes of information among specific elements of the system under study. In this respect, much more insight can be gained by studying transfer entropy and response, which allow quantifying the relative influence of parts of the system and the asymmetry of the fluxes. In order to understand the relation between the above-mentioned quantities, we investigate spatially asymmetric extended systems. First, we consider a simplified linear stochastic model, which can be studied analytically; then, we include nonlinear terms in the dynamics. Extensive numerical investigation shows the relation between entropy production and the above-introduced degrees of asymmetry. Finally, we apply our approach to the highly nontrivial dynamics generated by the Lorenz 96 model for Earth oceanic circulation.

Data driven forecasting of aperiodic motions of non-autonomous systems

In the present effort, a data-driven modeling approach is undertaken to forecast aperiodic responses of non-autonomous systems. As a representative non-autonomous system, a harmonically forced Duffing oscillator is considered. Along with it, an experimental prototype of a Duffing oscillator is studied. Data corresponding to chaotic motions are obtained through simulations of forced oscillators with hardening and softening characteristics and experiments with a bistable oscillator. Portions of these datasets are used to train a neural machine and make response predictions and forecasts for motions on the corresponding attractors. The neural machine is constructed by using a deep recurrent neural network architecture. The experiments conducted with the different numerical and experimental chaotic time-series data confirm the effectiveness of the constructed neural network for the forecasting of non-autonomous system responses.

Two methods to approximate the Koopman operator with a reservoir computer

The Koopman operator provides a powerful framework for data-driven analysis of dynamical systems. In the last few years, a wealth of numerical methods providing finite-dimensional approximations of the operator have been proposed [e.g., extended dynamic mode decomposition (EDMD) and its variants]. While convergence results for EDMD require an infinite number of dictionary elements, recent studies have shown that only a few dictionary elements can yield an efficient approximation of the Koopman operator, provided that they are well-chosen through a proper training process. However, this training process typically relies on nonlinear optimization techniques. In this paper, we propose two novel methods based on a reservoir computer to train the dictionary. These methods rely solely on linear convex optimization. We illustrate the efficiency of the method with several numerical examples in the context of data reconstruction, prediction, and computation of the Koopman operator spectrum. These results pave the way for the use of the reservoir computer in the Koopman operator framework.

Finite Cascades of Pitchfork Bifurcations and Multistability in Generalized Lorenz-96 Models

In this paper, we study a family of dynamical systems with circulant symmetry, which are obtained from the Lorenz-96 model by modifying its nonlinear terms. For each member of this family, the dimension n can be arbitrarily chosen and a forcing parameter F acts as a bifurcation parameter. The primary focus in this paper is on the occurrence of finite cascades of pitchfork bifurcations, where the length of such a cascade depends on the divisibility properties of the dimension n. A particularly intriguing aspect of this phenomenon is that the parameter values F of the pitchfork bifurcations seem to satisfy the Feigenbaum scaling law. Further bifurcations can lead to the coexistence of periodic or chaotic attractors. We also describe scenarios in which the number of coexisting attractors can be reduced through collisions with an equilibrium.

Multiscale measures of phase-space trajectories

Characterizing the multiscale nature of fluctuations from nonlinear and nonstationary time series is one of the most intensively studied contemporary problems in nonlinear sciences. In this work, we address this problem by combining two established concepts-empirical mode decomposition (EMD) and generalized fractal dimensions-into a unified analysis framework. Specifically, we demonstrate that the intrinsic mode functions derived by EMD can be used as a source of local (in terms of scales) information about the properties of the phase-space trajectory of the system under study, allowing us to derive multiscale measures when looking at the behavior of the generalized fractal dimensions at different scales. This formalism is applied to three well-known low-dimensional deterministic dynamical systems (the Hénon map, the Lorenz '63 system, and the standard map), three realizations of fractional Brownian motion with different Hurst exponents, and two somewhat higher-dimensional deterministic dynamical systems (the Lorenz '96 model and the on-off intermittency model). These examples allow us to assess the performance of our formalism with respect to practically relevant aspects like additive noise, different initial conditions, the length of the time series under study, low- vs high-dimensional dynamics, and bursting effects. Finally, by taking advantage of two real-world systems whose multiscale features have been widely investigated (a marine stack record providing a proxy of the global ice volume variability of the past 5×10⁶ years and the SYM-H geomagnetic index), we also illustrate the applicability of this formalism to real-world time series.

Using curvature to select the time lag for delay reconstruction

We propose a curvature-based approach for choosing a good value for the time-delay parameter τ in delay reconstructions. The idea is based on the effects of the delay on the geometry of the reconstructions. If the delay is too small, the reconstructed dynamics are flattened along the main diagonal of the embedding space; too-large delays, on the other hand, can overfold the dynamics. Calculating the curvature of a two-dimensional delay reconstruction is an effective way to identify these extremes and to find a middle ground between them: both the sharp reversals at the extremes of an insufficiently unfolded reconstruction and the bends in an overfolded one create spikes in the curvature. We operationalize this observation by computing the mean Menger curvature of a trajectory segment on 2D reconstructions as a function of time delay. We show that the minimum of these values gives an effective heuristic for choosing the time delay. In addition, we show that this curvature-based heuristic is useful even in cases where the customary approach, which uses average mutual information, fails-e.g., noisy or filtered data.

Application of Generalized Polynomial Chaos for Quantification of Uncertainties of Time Averages and Their Sensitivities in Chaotic Systems

In this paper, we consider the effect of stochastic uncertainties on non-linear systems with chaotic behavior. More specifically, we quantify the effect of parametric uncertainties to time-averaged quantities and their sensitivities. Sampling methods for Uncertainty Quantification (UQ), such as the Monte–Carlo (MC), are very costly, while traditional methods for sensitivity analysis, such as the adjoint, fail in chaotic systems. In this work, we employ the non-intrusive generalized Polynomial Chaos (gPC) for UQ, coupled with the Multiple-Shooting Shadowing (MSS) algorithm for sensitivity analysis of chaotic systems. It is shown that the gPC, coupled with MSS, is an appropriate method for conducting UQ in chaotic systems and produces results that match well with those from MC and Finite-Differences (FD).

A Maximum Likelihood Ensemble Filter Via A Modified Cholesky Decomposition For Non-Gaussian Data Assimilation

This paper proposes an efficient and practical implementation of the Maximum Likelihood Ensemble Filter via a Modified Cholesky decomposition (MLEF-MC). The method works as follows: via an ensemble of model realizations, a well-conditioned and full-rank square-root approximation of the background error covariance matrix is obtained. This square-root approximation serves as a control space onto which analysis increments can be computed. These are calculated via Line-Search (LS) optimization. We theoretically prove the convergence of the MLEF-MC. Experimental simulations were performed using an Atmospheric General Circulation Model (AT-GCM) and a highly nonlinear observation operator. The results reveal that the proposed method can obtain posterior error estimates within reasonable accuracies in terms of ℓ−2 error norms. Furthermore, our analysis estimates are similar to those of the MLEF with large ensemble sizes and full observational networks.

A Random Line-Search Optimization Method via Modified Cholesky Decomposition for Non-linear Data Assimilation

This paper proposes a line-search optimization method for non-linear data assimilation via random descent directions. The iterative method works as follows: at each iteration, quadratic approximations of the Three-Dimensional-Variational (3D-Var) cost function are built about current solutions. These approximations are employed to build sub-spaces onto which analysis increments can be estimated. We sample search-directions from those sub-spaces, and for each direction, a line-search optimization method is employed to estimate its optimal step length. Current solutions are updated based on directions along which the 3D-Var cost function decreases faster. We theoretically prove the global convergence of our proposed iterative method. Experimental tests are performed by using the Lorenz-96 model, and for reference, we employ a Maximum-Likelihood-Ensemble-Filter (MLEF) whose ensemble size doubles that of our implementation. The results reveal that, as the degree of observational operators increases, the use of additional directions can improve the accuracy of results in terms of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell _2$$\end{document}-norm of errors, and even more, our numerical results outperform those of the employed MLEF implementation.

The Role of Data in Model Building and Prediction: A Survey Through Examples

The goal of Science is to understand phenomena and systems in order to predict their development and gain control over them. In the scientific process of knowledge elaboration, a crucial role is played by models which, in the language of quantitative sciences, mean abstract mathematical or algorithmical representations. This short review discusses a few key examples from Physics, taken from dynamical systems theory, biophysics, and statistical mechanics, representing three paradigmatic procedures to build models and predictions from available data. In the case of dynamical systems we show how predictions can be obtained in a virtually model-free framework using the methods of analogues, and we briefly discuss other approaches based on machine learning methods. In cases where the complexity of systems is challenging, like in biophysics, we stress the necessity to include part of the empirical knowledge in the models to gain the minimal amount of realism. Finally, we consider many body systems where many (temporal or spatial) scales are at play—and show how to derive from data a dimensional reduction in terms of a Langevin dynamics for their slow components.

Local dimension and recurrent circulation patterns in long-term climate simulations

With the recent advent of a sound mathematical theory for extreme events in dynamical systems, new ways of analyzing a system's inherent properties have become available: Studying only the probabilities of extremely close Poincaré recurrences, we can infer the underlying attractor's local dimensionality-a quantity which is closely linked to the predictability of individual configurations, as well as the information gained from observing them. This study examines possible ways of estimating local and global attractor dimensions, identifies potential pitfalls, and discusses conceivable applications. The Portable University Model of the Atmosphere serves as a test subject of intermediate complexity between simple mathematical toys and truly realistic atmospheric data-sets. It is demonstrated that the introduction of a simple, analytical estimator can streamline the procedure and allows for additional tests of the agreement between theoretical expectation and observed data. We, furthermore, show how the newly gained knowledge about local dimensions can complement classical techniques like principal component analysis and may assist in separating meaningful patterns from mathematical artifacts.

Finding equilibrium in the spatiotemporal chaos of the complex Ginzburg-Landau equation

Equilibrium statistical mechanics allows the prediction of collective behaviors of large numbers of interacting objects from just a few system-wide properties; however, a similar theory does not exist for far-from-equilibrium systems exhibiting complex spatial and temporal behavior. We propose a method for predicting behaviors in a broad class of such systems and apply these ideas to an archetypal example, the spatiotemporal chaotic 1D complex Ginzburg-Landau equation in the defect chaos regime. Building on the ideas of Ruelle and of Cross and Hohenberg that a spatiotemporal chaotic system can be considered a collection of weakly interacting dynamical units of a characteristic size, the chaotic length scale, we identify underlying, mesoscale, chaotic units and effective interaction potentials between them. We find that the resulting equilibrium Takahashi model accurately predicts distributions of particle numbers. These results suggest the intriguing possibility that a class of far-from-equilibrium systems may be well described at coarse-grained scales by the well-established theory of equilibrium statistical mechanics.

Covariant Lyapunov vectors of chaotic Rayleigh-Bénard convection

We explore numerically the high-dimensional spatiotemporal chaos of Rayleigh-Bénard convection using covariant Lyapunov vectors. We integrate the three-dimensional and time-dependent Boussinesq equations for a convection layer in a shallow square box geometry with an aspect ratio of 16 for very long times and for a range of Rayleigh numbers. We simultaneously integrate many copies of the tangent space equations in order to compute the covariant Lyapunov vectors. The dynamics explored has fractal dimensions of 20≲D_{λ}≲50, and we compute on the order of 150 covariant Lyapunov vectors. We use the covariant Lyapunov vectors to quantify the degree of hyperbolicity of the dynamics and the degree of Oseledets splitting and to explore the temporal and spatial dynamics of the Lyapunov vectors. Our results indicate that the chaotic dynamics of Rayleigh-Bénard convection is nonhyperbolic for all of the Rayleigh numbers we have explored. Our results yield that the entire spectrum of covariant Lyapunov vectors that we have computed are tangled as indicated by near tangencies with neighboring vectors. A closer look at the spatiotemporal features of the Lyapunov vectors suggests contributions from structures at two different length scales with differing amounts of localization.

Percolation-based precursors of transitions in extended systems

Abrupt transitions are ubiquitous in the dynamics of complex systems. Finding precursors, i.e. early indicators of their arrival, is fundamental in many areas of science ranging from electrical engineering to climate. However, obtaining warnings of an approaching transition well in advance remains an elusive task. Here we show that a functional network, constructed from spatial correlations of the system’s time series, experiences a percolation transition way before the actual system reaches a bifurcation point due to the collective phenomena leading to the global change. Concepts from percolation theory are then used to introduce early warning precursors that anticipate the system’s tipping point. We illustrate the generality and versatility of our percolation-based framework with model systems experiencing different types of bifurcations and with Sea Surface Temperature time series associated to El Niño phenomenon.

Leveraging information storage to select forecast-optimal parameters for delay-coordinate reconstructions

Delay-coordinate reconstruction is a proven modeling strategy for building effective forecasts of nonlinear time series. The first step in this process is the estimation of good values for two parameters, the time delay and the embedding dimension. Many heuristics and strategies have been proposed in the literature for estimating these values. Few, if any, of these methods were developed with forecasting in mind, however, and their results are not optimal for that purpose. Even so, these heuristics-intended for other applications-are routinely used when building delay coordinate reconstruction-based forecast models. In this paper, we propose an alternate strategy for choosing optimal parameter values for forecast methods that are based on delay-coordinate reconstructions. The basic calculation involves maximizing the shared information between each delay vector and the future state of the system. We illustrate the effectiveness of this method on several synthetic and experimental systems, showing that this metric can be calculated quickly and reliably from a relatively short time series, and that it provides a direct indication of how well a near-neighbor based forecasting method will work on a given delay reconstruction of that time series. This allows a practitioner to choose reconstruction parameters that avoid any pathologies, regardless of the underlying mechanism, and maximize the predictive information contained in the reconstruction.

Prediction in projection

Prediction models that capture and use the structure of state-space dynamics can be very effective. In practice, however, one rarely has access to full information about that structure, and accurate reconstruction of the dynamics from scalar time-series data-e.g., via delay-coordinate embedding-can be a real challenge. In this paper, we show that forecast models that employ incomplete reconstructions of the dynamics-i.e., models that are not necessarily true embeddings-can produce surprisingly accurate predictions of the state of a dynamical system. In particular, we demonstrate the effectiveness of a simple near-neighbor forecast technique that works with a two-dimensional time-delay reconstruction of both low- and high-dimensional dynamical systems. Even though correctness of the topology may not be guaranteed for incomplete reconstructions like this, the dynamical structure that they do capture allows for accurate predictions-in many cases, even more accurate than predictions generated using a traditional embedding. This could be very useful in the context of real-time forecasting, where the human effort required to produce a correct delay-coordinate embedding is prohibitive.

Using waveform information in nonlinear data assimilation

Information in measurements of a nonlinear dynamical system can be transferred to a quantitative model of the observed system to establish its fixed parameters and unobserved state variables. After this learning period is complete, one may predict the model response to new forces and, when successful, these predictions will match additional observations. This adjustment process encounters problems when the model is nonlinear and chaotic because dynamical instability impedes the transfer of information from the data to the model when the number of measurements at each observation time is insufficient. We discuss the use of information in the waveform of the data, realized through a time delayed collection of measurements, to provide additional stability and accuracy to this search procedure. Several examples are explored, including a few familiar nonlinear dynamical systems and small networks of Colpitts oscillators.

Synchronizing spatio-temporal chaos with imperfect models: a stochastic surface growth picture

We study the synchronization of two spatially extended dynamical systems where the models have imperfections. We show that the synchronization error across space can be visualized as a rough surface governed by the Kardar-Parisi-Zhang equation with both upper and lower bounding walls corresponding to nonlinearities and model discrepancies, respectively. Two types of model imperfections are considered: parameter mismatch and unresolved fast scales, finding in both cases the same qualitative results. The consistency between different setups and systems indicates that the results are generic for a wide family of spatially extended systems.

Convolutionless Nakajima-Zwanzig equations for stochastic analysis in nonlinear dynamical systems

Determining the statistical properties of stochastic nonlinear systems is of major interest across many disciplines. Currently, there are no general efficient methods to deal with this challenging problem that involves high dimensionality, low regularity and random frequencies. We propose a framework for stochastic analysis in nonlinear dynamical systems based on goal-oriented probability density function (PDF) methods. The key idea stems from techniques of irreversible statistical mechanics, and it relies on deriving evolution equations for the PDF of quantities of interest, e.g. functionals of the solution to systems of stochastic ordinary and partial differential equations. Such quantities could be low-dimensional objects in infinite dimensional phase spaces. We develop the goal-oriented PDF method in the context of the time-convolutionless Nakajima-Zwanzig-Mori formalism. We address the question of approximation of reduced-order density equations by multi-level coarse graining, perturbation series and operator cumulant resummation. Numerical examples are presented for stochastic resonance and stochastic advection-reaction problems.

Length scale of a chaotic element in Rayleigh-Bénard convection

We describe an approach to quantify the length scale of a chaotic element of a Rayleigh-Bénard convection layer exhibiting spatiotemporal chaos. The length scale of a chaotic element is determined by simultaneously evolving the dynamics of two convection layers with a unidirectional coupling that involves only the time-varying values of the fluid velocity and temperature on the lateral boundaries of the domain. In our results we numerically simulate the full Boussinesq equations for the precise conditions of experiment. By varying the size of the boundary used for the coupling we identify a length scale that describes the size of a chaotic element. The length scale of the chaotic element is of the same order of magnitude, and exhibits similar trends, as the natural chaotic length scale that is based upon the fractal dimension.

Quantifying spatiotemporal chaos in Rayleigh-Bénard convection

Using large-scale parallel numerical simulations we explore spatiotemporal chaos in Rayleigh-Bénard convection in a cylindrical domain with experimentally relevant boundary conditions. We use the variation of the spectrum of Lyapunov exponents and the leading-order Lyapunov vector with system parameters to quantify states of high-dimensional chaos in fluid convection. We explore the relationship between the time dynamics of the spectrum of Lyapunov exponents and the pattern dynamics. For chaotic dynamics we find that all of the Lyapunov exponents are positively correlated with the leading-order Lyapunov exponent, and we quantify the details of their response to the dynamics of defects. The leading-order Lyapunov vector is used to identify topological features of the fluid patterns that contribute significantly to the chaotic dynamics. Our results show a transition from boundary-dominated dynamics to bulk-dominated dynamics as the system size is increased. The spectrum of Lyapunov exponents is used to compute the variation of the fractal dimension with system parameters to quantify how the underlying high-dimensional strange attractor accommodates a range of different chaotic dynamics.

Exploring spiral defect chaos in generalized Swift-Hohenberg models with mean flow

We explore the phenomenon of spiral defect chaos in two types of generalized Swift-Hohenberg model equations that include the effects of long-range drift velocity or mean flow. We use spatially extended domains and integrate the equations for very long times to study the pattern dynamics as the magnitude of the mean flow is varied. The magnitude of the mean flow is adjusted via a real and continuous parameter that accounts for the fluid boundary conditions on the horizontal surfaces in a convecting layer. For weak values of the mean flow, we find that the patterns exhibit a slow coarsening to a state dominated by large and very slowly moving target defects. For strong enough mean flow, we identify the existence of spatiotemporal chaos, which is indicated by a positive leading-order Lyapunov exponent. We compare the spatial features of the mean flow field with that of Rayleigh-Bénard convection and quantify their differences in the neighborhood of spiral defects.