Data-driven prediction in dynamical systems: recent developments

A mathematical framework for evo-devo dynamics

Natural selection acts on phenotypes constructed over development, which raises the question of how development affects evolution. Classic evolutionary theory indicates that development affects evolution by modulating the genetic covariation upon which selection acts, thus affecting genetic constraints. However, whether genetic constraints are relative, thus diverting adaptation from the direction of steepest fitness ascent, or absolute, thus blocking adaptation in certain directions, remains uncertain. This limits understanding of long-term evolution of developmentally constructed phenotypes. Here we formulate a general, tractable mathematical framework that integrates age progression, explicit development (i.e., the construction of the phenotype across life subject to developmental constraints), and evolutionary dynamics, thus describing the evolutionary and developmental (evo-devo) dynamics. The framework yields simple equations that can be arranged in a layered structure that we call the evo-devo process, whereby five core elementary components generate all equations including those mechanistically describing genetic covariation and the evo-devo dynamics. The framework recovers evolutionary dynamic equations in gradient form and describes the evolution of genetic covariation from the evolution of genotype, phenotype, environment, and mutational covariation. This shows that genotypic and phenotypic evolution must be followed simultaneously to yield a dynamically sufficient description of long-term phenotypic evolution in gradient form, such that evolution described as the climbing of a fitness landscape occurs in "geno-phenotype" space. Genetic constraints in geno-phenotype space are necessarily absolute because the phenotype is related to the genotype by development. Thus, the long-term evolutionary dynamics of developed phenotypes is strongly non-standard: (1) evolutionary equilibria are either absent or infinite in number and depend on genetic covariation and hence on development; (2) developmental constraints determine the admissible evolutionary path and hence which evolutionary equilibria are admissible; and (3) evolutionary outcomes occur at admissible evolutionary equilibria, which do not generally occur at fitness landscape peaks in geno-phenotype space, but at peaks in the admissible evolutionary path where "total genotypic selection" vanishes if exogenous plastic response vanishes and mutational variation exists in all directions of genotype space. Hence, selection and development jointly define the evolutionary outcomes if absolute mutational constraints and exogenous plastic response are absent, rather than the outcomes being defined only by selection. Moreover, our framework provides formulas for the sensitivities of a recurrence and an alternative method to dynamic optimization (i.e., dynamic programming or optimal control) to identify evolutionary outcomes in models with developmentally dynamic traits. These results show that development has major evolutionary effects.

Learning nonlinear projections for reduced-order modeling of dynamical systems using constrained autoencoders

Recently developed reduced-order modeling techniques aim to approximate nonlinear dynamical systems on low-dimensional manifolds learned from data. This is an effective approach for modeling dynamics in a post-transient regime where the effects of initial conditions and other disturbances have decayed. However, modeling transient dynamics near an underlying manifold, as needed for real-time control and forecasting applications, is complicated by the effects of fast dynamics and nonnormal sensitivity mechanisms. To begin to address these issues, we introduce a parametric class of nonlinear projections described by constrained autoencoder neural networks in which both the manifold and the projection fibers are learned from data. Our architecture uses invertible activation functions and biorthogonal weight matrices to ensure that the encoder is a left inverse of the decoder. We also introduce new dynamics-aware cost functions that promote learning of oblique projection fibers that account for fast dynamics and nonnormality. To demonstrate these methods and the specific challenges they address, we provide a detailed case study of a three-state model of vortex shedding in the wake of a bluff body immersed in a fluid, which has a two-dimensional slow manifold that can be computed analytically. In anticipation of future applications to high-dimensional systems, we also propose several techniques for constructing computationally efficient reduced-order models using our proposed nonlinear projection framework. This includes a novel sparsity-promoting penalty for the encoder that avoids detrimental weight matrix shrinkage via computation on the Grassmann manifold.

Distilling identifiable and interpretable dynamic models from biological data

Mechanistic dynamical models allow us to study the behavior of complex biological systems. They can provide an objective and quantitative understanding that would be difficult to achieve through other means. However, the systematic development of these models is a non-trivial exercise and an open problem in computational biology. Currently, many research efforts are focused on model discovery, i.e. automating the development of interpretable models from data. One of the main frameworks is sparse regression, where the sparse identification of nonlinear dynamics (SINDy) algorithm and its variants have enjoyed great success. SINDy-PI is an extension which allows the discovery of rational nonlinear terms, thus enabling the identification of kinetic functions common in biochemical networks, such as Michaelis-Menten. SINDy-PI also pays special attention to the recovery of parsimonious models (Occam's razor). Here we focus on biological models composed of sets of deterministic nonlinear ordinary differential equations. We present a methodology that, combined with SINDy-PI, allows the automatic discovery of structurally identifiable and observable models which are also mechanistically interpretable. The lack of structural identifiability and observability makes it impossible to uniquely infer parameter and state variables, which can compromise the usefulness of a model by distorting its mechanistic significance and hampering its ability to produce biological insights. We illustrate the performance of our method with six case studies. We find that, despite enforcing sparsity, SINDy-PI sometimes yields models that are unidentifiable. In these cases we show how our method transforms their equations in order to obtain a structurally identifiable and observable model which is also interpretable.

Optimizing the combination of data-driven and model-based elements in hybrid reservoir computing

Hybrid reservoir computing combines purely data-driven machine learning predictions with a physical model to improve the forecasting of complex systems. In this study, we investigate in detail the predictive capabilities of three different architectures for hybrid reservoir computing: the input hybrid (IH), output hybrid (OH), and full hybrid (FH), which combines IH and OH. By using nine different three-dimensional chaotic model systems and the high-dimensional spatiotemporal chaotic Kuramoto-Sivashinsky system, we demonstrate that all hybrid reservoir computing approaches significantly improve the prediction results, provided that the model is sufficiently accurate. For accurate models, we find that the OH and FH results are equivalent and significantly outperform the IH results, especially for smaller reservoir sizes. For totally inaccurate models, the predictive capabilities of IH and FH may decrease drastically, while the OH architecture remains as accurate as the purely data-driven results. Furthermore, OH allows for the separation of the reservoir and the model contributions to the output predictions. This enables an interpretation of the roles played by the data-driven and model-based elements in output hybrid reservoir computing, resulting in higher explainability of the prediction results. Overall, our findings suggest that the OH approach is the most favorable architecture for hybrid reservoir computing, when taking accuracy, interpretability, robustness to model error, and simplicity into account.

Nonlinear model reduction to fractional and mixed-mode spectral submanifolds

A primary spectral submanifold (SSM) is the unique smoothest nonlinear continuation of a nonresonant spectral subspace E of a dynamical system linearized at a fixed point. Passing from the full nonlinear dynamics to the flow on an attracting primary SSM provides a mathematically precise reduction of the full system dynamics to a very low-dimensional, smooth model in polynomial form. A limitation of this model reduction approach has been, however, that the spectral subspace yielding the SSM must be spanned by eigenvectors of the same stability type. A further limitation has been that in some problems, the nonlinear behavior of interest may be far away from the smoothest nonlinear continuation of the invariant subspace E. Here, we remove both of these limitations by constructing a significantly extended class of SSMs that also contains invariant manifolds with mixed internal stability types and of lower smoothness class arising from fractional powers in their parametrization. We show on examples how fractional and mixed-mode SSMs extend the power of data-driven SSM reduction to transitions in shear flows, dynamic buckling of beams, and periodically forced nonlinear oscillatory systems. More generally, our results reveal the general function library that should be used beyond integer-powered polynomials in fitting nonlinear reduced-order models to data.

How development affects evolution

Natural selection acts on developmentally constructed phenotypes, but how does development affect evolution? This question prompts a simultaneous consideration of development and evolution. However, there has been a lack of general mathematical frameworks mechanistically integrating the two, which may have inhibited progress on the question. Here, we use a new mathematical framework that mechanistically integrates development into evolution to analyse how development affects evolution. We show that, while selection pushes genotypic and phenotypic evolution up the fitness landscape, development determines the admissible evolutionary pathway, such that evolutionary outcomes occur at path peaks rather than landscape peaks. Changes in development can generate path peaks, triggering genotypic or phenotypic diversification, even on constant, single-peak landscapes. Phenotypic plasticity, niche construction, extra-genetic inheritance, and developmental bias alter the evolutionary path and hence the outcome. Thus, extra-genetic inheritance can have permanent evolutionary effects by changing the developmental constraints, even if extra-genetically acquired elements are not transmitted to future generations. Selective development, whereby phenotype construction points in the adaptive direction, may induce adaptive or maladaptive evolution depending on the developmental constraints. Moreover, developmental propagation of phenotypic effects over age enables the evolution of negative senescence. Overall, we find that development plays a major evolutionary role.