Parameter Estimation for Differential Equation Models Using a Framework of Measurement Error in Regression Models

The Michaelis-Menten Reaction at Low Substrate Concentrations: Pseudo-First-Order Kinetics and Conditions for Timescale Separation

We demonstrate that the Michaelis-Menten reaction mechanism can be accurately approximated by a linear system when the initial substrate concentration is low. This leads to pseudo-first-order kinetics, simplifying mathematical calculations and experimental analysis. Our proof utilizes a monotonicity property of the system and Kamke's comparison theorem. This linear approximation yields a closed-form solution, enabling accurate modeling and estimation of reaction rate constants even without timescale separation. Building on prior work, we establish that the sufficient condition for the validity of this approximation iss 0 ≪ K , where K = k 2 / k 1 is the Van Slyke-Cullen constant. This condition is independent of the initial enzyme concentration. Further, we investigate timescale separation within the linear system, identifying necessary and sufficient conditions and deriving the corresponding reduced one-dimensional equations.

Parameter estimation and forecasting with quantified uncertainty for ordinary differential equation models using QuantDiffForecast: A MATLAB toolbox and tutorial

Mathematical models based on systems of ordinary differential equations (ODEs) are frequently applied in various scientific fields to assess hypotheses, estimate key model parameters, and generate predictions about the system's state. To support their application, we present a comprehensive, easy-to-use, and flexible MATLAB toolbox, QuantDiffForecast, and associated tutorial to estimate parameters and generate short-term forecasts with quantified uncertainty from dynamical models based on systems of ODEs. We provide software ( https://github.com/gchowell/paramEstimation_forecasting_ODEmodels/) and detailed guidance on estimating parameters and forecasting time-series trajectories that are characterized using ODEs with quantified uncertainty through a parametric bootstrapping approach. It includes functions that allow the user to infer model parameters and assess forecasting performance for different ODE models specified by the user, using different estimation methods and error structures in the data. The tutorial is intended for a diverse audience, including students training in dynamic systems, and will be broadly applicable to estimate parameters and generate forecasts from models based on ODEs. The functions included in the toolbox are illustrated using epidemic models with varying levels of complexity applied to data from the 1918 influenza pandemic in San Francisco. A tutorial video that demonstrates the functionality of the toolbox is included.

Uncovering personal circadian responses to light through particle swarm optimization

BACKGROUND AND OBJECTIVES

Kronauer's oscillator model of the human central pacemaker is one of the most commonly used approaches to study the human circadian response to light. Two sources of error when applying it to a personal light exposure have been identified: (1) as a populational model, it does not consider inter-individual variability, and (2) the initial conditions needed to integrate the model are usually unknown, and thus subjectively estimated. In this work, we evaluate the ability of particle swarm optimization (PSO) algorithms to simultaneously uncover the optimal initial conditions and individual parameters of a pre-defined Kronauer's oscillator model.

METHODS

A Canonical PSO, a Dynamic Multi-Swarm PSO and a novel modification of the latter, namely Hierarchical Dynamic Multi-Swarm PSO, are evaluated. Two different target models (under a regular and an irregular schedule) are defined, and the same realistic light profile is fed to them. Based on their output, a fitness function is proposed, which is minimized by the algorithms to find the optimum set of parameters and initial conditions of the model.

RESULTS

We demonstrate that Dynamic Multi-Swarm and Hierarchical Dynamic Multi-Swarm algorithms can accurately uncover personal circadian parameters under both regular and irregular schedules, but as expected, optimization is easier under a regular schedule. Circadian parameters play the most important role in the optimization process and should be prioritized over initial conditions, although assessment of the impact of misestimating the latter is recommended. The log-log linear relationship between mean absolute error and computational cost shows that the number of particles to use is at the discretion of the user.

CONCLUSIONS

The robustness and low errors achieved by the algorithms support their further testing, validation and systematic application to empirical data under a regular or irregular schedule. Uncovering personal circadian parameters can improve the assessment of the circadian status of a person and the applicability of personalized light therapies, as well as help to discover other factors that may lie behind the interindividual variability in the circadian response to light.

Direct Estimation of Parameters in ODE Models Using WENDy: Weak-Form Estimation of Nonlinear Dynamics

We introduce the Weak-form Estimation of Nonlinear Dynamics (WENDy) method for estimating model parameters for non-linear systems of ODEs. Without relying on any numerical differential equation solvers, WENDy computes accurate estimates and is robust to large (biologically relevant) levels of measurement noise. For low dimensional systems with modest amounts of data, WENDy is competitive with conventional forward solver-based nonlinear least squares methods in terms of speed and accuracy. For both higher dimensional systems and stiff systems, WENDy is typically both faster (often by orders of magnitude) and more accurate than forward solver-based approaches. The core mathematical idea involves an efficient conversion of the strong form representation of a model to its weak form, and then solving a regression problem to perform parameter inference. The core statistical idea rests on the Errors-In-Variables framework, which necessitates the use of the iteratively reweighted least squares algorithm. Further improvements are obtained by using orthonormal test functions, created from a set of [Formula: see text] bump functions of varying support sizes.We demonstrate the high robustness and computational efficiency by applying WENDy to estimate parameters in some common models from population biology, neuroscience, and biochemistry, including logistic growth, Lotka-Volterra, FitzHugh-Nagumo, Hindmarsh-Rose, and a Protein Transduction Benchmark model. Software and code for reproducing the examples is available at https://github.com/MathBioCU/WENDy .

The identification of piecewise non-linear dynamical system without understanding the mechanism

This paper designs an algorithm to distill the piecewise non-linear dynamical system from the data without prior knowledge. The system to be identified does not have to be written as a known model term or be thoroughly understood. We exploit the fact that an unknown piecewise non-linear system can be decomposed into the Fourier series as long as its equations of motion are Riemann integrable. Based on this property, we reduce the challenge of finding the correct model to discovering the Fourier series approximation. However, the Fourier series approximation of the piecewise function is inaccurate. The new method takes advantage of this weakness to determine whether the model has piecewise features and to find a way to discover the discontinuity set. Then, the dynamical system on each segment is identified as a pure Fourier series. Identification of intricate models can be achieved in simple steps. The results show that the method can accurately discover the equation of motion and precisely capture the non-smooth characteristic. Next, the prediction and further detailed analysis can be carried out.

Direct Estimation of Parameters in ODE Models Using WENDy: Weak-form Estimation of Nonlinear Dynamics

We introduce the Weak-form Estimation of Nonlinear Dynamics (WENDy) method for estimating model parameters for non-linear systems of ODEs. Without relying on any numerical differential equation solvers, WENDy computes accurate estimates and is robust to large (biologically relevant) levels of measurement noise. For low dimensional systems with modest amounts of data, WENDy is competitive with conventional forward solver-based nonlinear least squares methods in terms of speed and accuracy. For both higher dimensional systems and stiff systems, WENDy is typically both faster (often by orders of magnitude) and more accurate than forward solver-based approaches. The core mathematical idea involves an efficient conversion of the strong form representation of a model to its weak form, and then solving a regression problem to perform parameter inference. The core statistical idea rests on the Errors-In-Variables framework, which necessitates the use of the iteratively reweighted least squares algorithm. Further improvements are obtained by using orthonormal test functions, created from a set of C-infinity bump functions of varying support sizes. We demonstrate the high robustness and computational efficiency by applying WENDy to estimate parameters in some common models from population biology, neuroscience, and biochemistry, including logistic growth, Lotka-Volterra, FitzHugh-Nagumo, Hindmarsh-Rose, and a Protein Transduction Benchmark model. Software and code for reproducing the examples is available at (https://github.com/MathBioCU/WENDy).

Estimation of Ordinary Differential Equation Models for Gene Regulatory Networks Through Data Cloning

Ordinary differential equations (ODEs) are widely used for elucidating dynamic processes in various fields. One of the applications of ODEs is to describe dynamics of gene regulatory networks (GRNs), which is a critical step in understanding disease mechanisms. However, estimation of ODE models for GRNs is challenging because of inflexibility of the model and noisy data with complex error structures such as heteroscedasticity, correlations between genes, and time dependency. In addition, either a likelihood or Bayesian approach is commonly used for estimation of ODE models, but both approaches have benefits and drawbacks in their own right. Data cloning is a maximum likelihood (ML) estimation method through the Bayesian framework. Since it works in the Bayesian framework, it is free from local optimum problems that are common drawbacks of ML methods. Also, its inference is invariant for the selection of prior distributions, which is a major issue in Bayesian methods. This study proposes an estimation method of ODE models for GRNs through data cloning. The proposed method is demonstrated through simulation and it is applied to real gene expression time-course data.

Efficient inference and identifiability analysis for differential equation models with random parameters

Heterogeneity is a dominant factor in the behaviour of many biological processes. Despite this, it is common for mathematical and statistical analyses to ignore biological heterogeneity as a source of variability in experimental data. Therefore, methods for exploring the identifiability of models that explicitly incorporate heterogeneity through variability in model parameters are relatively underdeveloped. We develop a new likelihood-based framework, based on moment matching, for inference and identifiability analysis of differential equation models that capture biological heterogeneity through parameters that vary according to probability distributions. As our novel method is based on an approximate likelihood function, it is highly flexible; we demonstrate identifiability analysis using both a frequentist approach based on profile likelihood, and a Bayesian approach based on Markov-chain Monte Carlo. Through three case studies, we demonstrate our method by providing a didactic guide to inference and identifiability analysis of hyperparameters that relate to the statistical moments of model parameters from independent observed data. Our approach has a computational cost comparable to analysis of models that neglect heterogeneity, a significant improvement over many existing alternatives. We demonstrate how analysis of random parameter models can aid better understanding of the sources of heterogeneity from biological data.

Learning delay dynamics for multivariate stochastic processes, with application to the prediction of the growth rate of COVID-19 cases in the United States

Delay differential equations form the underpinning of many complex dynamical systems. The forward problem of solving random differential equations with delay has received increasing attention in recent years. Motivated by the challenge to predict the COVID-19 caseload trajectories for individual states in the U.S., we target here the inverse problem. Given a sample of observed random trajectories obeying an unknown random differential equation model with delay, we use a functional data analysis framework to learn the model parameters that govern the underlying dynamics from the data. We show the existence and uniqueness of the analytical solutions of the population delay random differential equation model when one has discrete time delays in the functional concurrent regression model and also for a second scenario where one has a delay continuum or distributed delay. The latter involves a functional linear regression model with history index. The derivative of the process of interest is modeled using the process itself as predictor and also other functional predictors with predictor-specific delayed impacts. This dynamics learning approach is shown to be well suited to model the growth rate of COVID-19 for the states that are part of the U.S., by pooling information from the individual states, using the case process and concurrently observed economic and mobility data as predictors.

A Joint estimation approach to sparse additive ordinary differential equations

Ordinary differential equations (ODEs) are widely used to characterize the dynamics of complex systems in real applications. In this article, we propose a novel joint estimation approach for generalized sparse additive ODEs where observations are allowed to be non-Gaussian. The new method is unified with existing collocation methods by considering the likelihood, ODE fidelity and sparse regularization simultaneously. We design a block coordinate descent algorithm for optimizing the non-convex and non-differentiable objective function. The global convergence of the algorithm is established. The simulation study and two applications demonstrate the superior performance of the proposed method in estimation and improved performance of identifying the sparse structure.

Kernel Ordinary Differential Equations

Ordinary differential equation (ODE) is widely used in modeling biological and physical processes in science. In this article, we propose a new reproducing kernel-based approach for estimation and inference of ODE given noisy observations. We do not assume the functional forms in ODE to be known, or restrict them to be linear or additive, and we allow pairwise interactions. We perform sparse estimation to select individual functionals, and construct confidence intervals for the estimated signal trajectories. We establish the estimation optimality and selection consistency of kernel ODE under both the low-dimensional and high-dimensional settings, where the number of unknown functionals can be smaller or larger than the sample size. Our proposal builds upon the smoothing spline analysis of variance (SS-ANOVA) framework, but tackles several important problems that are not yet fully addressed, and thus extends the scope of existing SS-ANOVA as well. We demonstrate the efficacy of our method through numerous ODE examples.

Semiparametric Mixed-Effects Ordinary Differential Equation Models with Heavy-Tailed Distributions

Ordinary differential equation (ODE) models are popularly used to describe complex dynamical systems. When estimating ODE parameters from noisy data, a common distribution assumption is using the Gaussian distribution. It is known that the Gaussian distribution is not robust when abnormal data exist. In this article, we develop a hierarchical semiparametric mixed-effects ODE model for longitudinal data under the Bayesian framework. For robust inference on ODE parameters, we consider a class of heavy-tailed distributions to model the random effects of ODE parameters and observations errors. An MCMC method is proposed to sample ODE parameters from the posterior distributions. Our proposed method is illustrated by studying a gene regulation experiment. Simulation studies show that our proposed method provides satisfactory results for the semiparametric mixed-effects ODE models with finite samples.

Supplementary materials accompanying this paper appear online.

Collocation based training of neural ordinary differential equations

The predictive power of machine learning models often exceeds that of mechanistic modeling approaches. However, the interpretability of purely data-driven models, without any mechanistic basis is often complicated, and predictive power by itself can be a poor metric by which we might want to judge different methods. In this work, we focus on the relatively new modeling techniques of neural ordinary differential equations. We discuss how they relate to machine learning and mechanistic models, with the potential to narrow the gulf between these two frameworks: they constitute a class of hybrid model that integrates ideas from data-driven and dynamical systems approaches. Training neural ODEs as representations of dynamical systems data has its own specific demands, and we here propose a collocation scheme as a fast and efficient training strategy. This alleviates the need for costly ODE solvers. We illustrate the advantages that collocation approaches offer, as well as their robustness to qualitative features of a dynamical system, and the quantity and quality of observational data. We focus on systems that exemplify some of the hallmarks of complex dynamical systems encountered in systems biology, and we map out how these methods can be used in the analysis of mathematical models of cellular and physiological processes.

Dynamic estimation of epidemiological parameters of COVID-19 outbreak and effects of interventions on its spread

BACKGROUND

A key challenge in estimating epidemiological parameters for a pandemic such as the initial COVID-19 outbreak in Wuhan is the discrepancy between the officially reported number of infections and the true number of infections. A common approach to tackling the challenge is to use the number of infections exported from the originating city to infer the true number. This approach can only provide a static estimate of the epidemiological parameters before city lockdown because there are almost no exported cases thereafter.

METHODS

We propose a Bayesian estimation method that dynamically estimates the epidemiological parameters by recovering true numbers of infections from day-to-day official numbers. To illustrate the use of this method, we provide a comprehensive retrospection on how the COVID-19 had progressed in Wuhan from January 19 to March 5, 2020. Particularly, we estimate that the outbreak sizes by January 23 and March 5 were 11,239 [95% CI 4,794-22,372] and 124,506 [95% CI 69,526-265,113], respectively.

RESULTS

The effective reproduction number attained its maximum on January 24 (3.42 [95% CI 3.34-3.50]) and became less than 1 from February 7 (0.76 [95% CI 0.65-0.92]). We also estimate the effects of two major government interventions on the spread of COVID-19 in Wuhan.

CONCLUSIONS

This case study by our proposed method affirms the believed importance and effectiveness of imposing tight non-essential travel restrictions and affirm the importance and effectiveness of government interventions (e.g., transportation suspension and large scale hospitalization) for effective mitigation of COVID-19 community spread.

Non-invasive Inference of Thrombus Material Properties with Physics-Informed Neural Networks

We employ physics-informed neural networks (PINNs) to infer properties of biological materials using synthetic data. In particular, we successfully apply PINNs on inferring permeability and viscoelastic modulus from thrombus deformation data, which can be described by the fourth-order Cahn-Hilliard and Navier-Stokes Equations. In PINNs, the partial differential equations are encoded into a loss function, where partial derivatives can be obtained through automatic differentiation (AD). In addition to tackling the challenge of calculating the fourth-order derivative in the Cahn-Hilliard equation with AD, we introduce an auxiliary network along with the main neural network to approximate the second-derivative of the energy potential term. Our model can simultaneously predict unknown material parameters and velocity, pressure, and deformation gradient fields by merely training with partial information among all data, i.e., phase field and pressure measurements, while remaining highly flexible in sampling within the spatio-temporal domain for data acquisition. We validate our model by numerical solutions from the spectral/hp element method (SEM) and demonstrate its robustness by training it with noisy measurements. Our results show that PINNs can infer the material properties from noisy synthetic data, and thus they have great potential for inferring these properties from experimental multi-modality and multi-fidelity data.

R package for statistical inference in dynamical systems using kernel based gradient matching: KGode

Many processes in science and engineering can be described by dynamical systems based on nonlinear ordinary differential equations (ODEs). Often ODE parameters are unknown and not directly measurable. Since nonlinear ODEs typically have no closed form solution, standard iterative inference procedures require a computationally expensive numerical integration of the ODEs every time the parameters are adapted, which in practice restricts statistical inference to rather small systems. To overcome this computational bottleneck, approximate methods based on gradient matching have recently gained much attention. The idea is to circumvent the numerical integration step by using a surrogate cost function that quantifies the discrepancy between the derivatives obtained from a smooth interpolant to the data and the derivatives predicted by the ODEs. The present article describes the software implementation of a recent method that is based on the framework of reproducing kernel Hilbert spaces. We provide an overview of the methods available, illustrate them on a series of widely used benchmark problems, and discuss the accuracy–efficiency trade-off of various regularization methods.

Integration of single-cell multi-omics for gene regulatory network inference

The advancement of single-cell sequencing technology in recent years has provided an opportunity to reconstruct gene regulatory networks (GRNs) with the data from thousands of single cells in one sample. This uncovers regulatory interactions in cells and speeds up the discoveries of regulatory mechanisms in diseases and biological processes. Therefore, more methods have been proposed to reconstruct GRNs using single-cell sequencing data. In this review, we introduce technologies for sequencing single-cell genome, transcriptome, and epigenome. At the same time, we present an overview of current GRN reconstruction strategies utilizing different single-cell sequencing data. Bioinformatics tools were grouped by their input data type and mathematical principles for reader's convenience, and the fundamental mathematics inherent in each group will be discussed. Furthermore, the adaptabilities and limitations of these different methods will also be summarized and compared, with the hope to facilitate researchers recognizing the most suitable tools for them.

Gene expression dynamic analysis reveals co-activation of Sonic Hedgehog and epidermal growth factor followed by dynamic silencing

Aberrant activation of the Sonic Hedgehog (SHH) gene is observed in various cancers. Previous studies have shown a “cross-talk” effect between the canonical Hedgehog signaling pathway and the Epidermal Growth Factor (EGF) pathway when SHH is active in the presence of EGF. However, the precise mechanism of the cross-talk effect on the entire gene population has not been investigated. Here, we re-analyzed publicly available data to study how SHH and EGF cooperate to affect the dynamic activity of the gene population. We used genome dynamic analysis to explore the expression profiles under different conditions in a human medulloblastoma cell line. Ordinary differential equations, equipped with solid statistical and computational tools, were exploited to extract the information hidden in the dynamic behavior of the gene population. Our results revealed that EGF stimulation plays a dominant role, overshadowing most of the SHH effects. We also identified cross-talk genes that exhibited expression profiles dissimilar to that seen under SHH or EGF stimulation alone. These unique cross-talk patterns were validated in a cell culture model. These cross-talk genes identified here may serve as valuable markers to study or test for EGF co-stimulatory effects in an SHH+ environment. Furthermore, these cross-talk genes may play roles in cancer progression, thus they may be further explored as cancer treatment targets.

Modelling of hypoxia gene expression for three different cancer cell lines

Gene dynamic analysis is essential in identifying target genes involved pathogenesis of various diseases, including cancer. Cancer prognosis is often influenced by hypoxia. We apply a multi-step pipeline to study dynamic gene expressions in response to hypoxia in three cancer cell lines: prostate (DU145), colon (HT29), and breast (MCF7) cancers. We identified 26 distinct temporal expression patterns for prostate cell line, and 29 patterns for colon and breast cell lines. The module-based dynamic networks have been developed for all three cell lines. Our analyses improve the existing results in multiple ways. It exploits the time-dependence nature of gene expression values in identifying the dynamically significant genes; hence, more key significant genes and transcription factors have been identified. Our gene network returns significant information regarding biologically important modules of genes. Furthermore, the network has potential in learning the regulatory path between transcription factors and the downstream genes. In addition, our findings suggest that changes in genes BMP6 and ARSJ expression might have a key role in the time-dependent response to hypoxia in breast cancer.

A big data pipeline: Identifying dynamic gene regulatory networks from time-course Gene Expression Omnibus data with applications to influenza infection

A biological host response to an external stimulus or intervention such as a disease or infection is a dynamic process, which is regulated by an intricate network of many genes and their products. Understanding the dynamics of this gene regulatory network allows us to infer the mechanisms involved in a host response to an external stimulus, and hence aids the discovery of biomarkers of phenotype and biological function. In this article, we propose a modeling/analysis pipeline for dynamic gene expression data, called Pipeline4DGEData, which consists of a series of statistical modeling techniques to construct dynamic gene regulatory networks from the large volumes of high-dimensional time-course gene expression data that are freely available in the Gene Expression Omnibus repository. This pipeline has a consistent and scalable structure that allows it to simultaneously analyze a large number of time-course gene expression data sets, and then integrate the results across different studies. We apply the proposed pipeline to influenza infection data from nine studies and demonstrate that interesting biological findings can be discovered with its implementation.

Nonparametric inference of interaction laws in systems of agents from trajectory data

Inferring the laws of interaction in agent-based systems from observational data is a fundamental challenge in a wide variety of disciplines. We propose a nonparametric statistical learning approach for distance-based interactions, with no reference or assumption on their analytical form, given data consisting of sampled trajectories of interacting agents. We demonstrate the effectiveness of our estimators both by providing theoretical guarantees that avoid the curse of dimensionality and by testing them on a variety of prototypical systems used in various disciplines. These systems include homogeneous and heterogeneous agent systems, ranging from particle systems in fundamental physics to agent-based systems that model opinion dynamics under the social influence, prey-predator dynamics, flocking and swarming, and phototaxis in cell dynamics.

Parametric and non-parametric gradient matching for network inference: a comparison

Background

Reverse engineering of gene regulatory networks from time series gene-expression data is a challenging problem, not only because of the vast sets of candidate interactions but also due to the stochastic nature of gene expression. We limit our analysis to nonlinear differential equation based inference methods. In order to avoid the computational cost of large-scale simulations, a two-step Gaussian process interpolation based gradient matching approach has been proposed to solve differential equations approximately.

Results

We apply a gradient matching inference approach to a large number of candidate models, including parametric differential equations or their corresponding non-parametric representations, we evaluate the network inference performance under various settings for different inference objectives. We use model averaging, based on the Bayesian Information Criterion (BIC), to combine the different inferences. The performance of different inference approaches is evaluated using area under the precision-recall curves.

Conclusions

We found that parametric methods can provide comparable, and often improved inference compared to non-parametric methods; the latter, however, require no kinetic information and are computationally more efficient.

Electronic supplementary material

The online version of this article (10.1186/s12859-018-2590-7) contains supplementary material, which is available to authorized users.

Parameter Estimation and Variable Selection for Big Systems of Linear Ordinary Differential Equations: A Matrix-Based Approach

Ordinary differential equations (ODEs) are widely used to model the dynamic behavior of a complex system. Parameter estimation and variable selection for a "Big System" with linear ODEs are very challenging due to the need of nonlinear optimization in an ultra-high dimensional parameter space. In this article, we develop a parameter estimation and variable selection method based on the ideas of similarity transformation and separable least squares (SLS). Simulation studies demonstrate that the proposed matrix-based SLS method could be used to estimate the coefficient matrix more accurately and perform variable selection for a linear ODE system with thousands of dimensions and millions of parameters much better than the direct least squares (LS) method and the vector-based two-stage method that are currently available. We applied this new method to two real data sets: a yeast cell cycle gene expression data set with 30 dimensions and 930 unknown parameters and the Standard & Poor 1500 index stock price data with 1250 dimensions and 1,563,750 unknown parameters, to illustrate the utility and numerical performance of the proposed parameter estimation and variable selection method for big systems in practice.

Parameter Estimation for Semiparametric Ordinary Differential Equation Models

We propose a new class of two-stage parameter estimation methods for semiparametric ordinary differential equation (ODE) models. In the first stage, state variables are estimated using a penalized spline approach; In the second stage, form of numerical discretization algorithms for an ODE solver is used to formulate estimating equations. Estimated state variables from the first stage are used to obtain more data points for the second stage. Asymptotic properties for the proposed estimators are established. Simulation studies show that the method performs well, especially for small sample. Real life use of the method is illustrated using Influenza specific cell-trafficking study.

Independence screening for high dimensional nonlinear additive ODE models with applications to dynamic gene regulatory networks

Mechanism-driven low-dimensional ordinary differential equation (ODE) models are often used to model viral dynamics at cellular levels and epidemics of infectious diseases. However, low-dimensional mechanism-based ODE models are limited for modeling infectious diseases at molecular levels such as transcriptomic or proteomic levels, which is critical to understand pathogenesis of diseases. Although linear ODE models have been proposed for gene regulatory networks (GRNs), nonlinear regulations are common in GRNs. The reconstruction of large-scale nonlinear networks from time-course gene expression data remains an unresolved issue. Here, we use high-dimensional nonlinear additive ODEs to model GRNs and propose a 4-step procedure to efficiently perform variable selection for nonlinear ODEs. To tackle the challenge of high dimensionality, we couple the 2-stage smoothing-based estimation method for ODEs and a nonlinear independence screening method to perform variable selection for the nonlinear ODE models. We have shown that our method possesses the sure screening property and it can handle problems with non-polynomial dimensionality. Numerical performance of the proposed method is illustrated with simulated data and a real data example for identifying the dynamic GRN of Saccharomyces cerevisiae.

Estimation of Dynamic Systems for Gene Regulatory Networks from Dependent Time-Course Data

Dynamic system consisting of ordinary differential equations (ODEs) is a well-known tool for describing dynamic nature of gene regulatory networks (GRNs), and the dynamic features of GRNs are usually captured through time-course gene expression data. Owing to high-throughput technologies, time-course gene expression data have complex structures such as heteroscedasticity, correlations between genes, and time dependence. Since gene experiments typically yield highly noisy data with small sample size, for a more accurate prediction of the dynamics, the complex structures should be taken into account in ODE models. Hence, this study proposes an ODE model considering such data structures and a fast and stable estimation method for the ODE parameters based on the generalized profiling approach with data smoothing techniques. The proposed method also provides statistical inference for the ODE estimator and it is applied to a zebrafish retina cell network.

Using single-index ODEs to study dynamic gene regulatory network

With the development of biotechnology, high-throughput studies on protein-protein, protein-gene, and gene-gene interactions become possible and attract remarkable attention. To explore the interactions in dynamic gene regulatory networks, we propose a single-index ordinary differential equation (ODE) model and develop a variable selection procedure. We employ the smoothly clipped absolute deviation penalty (SCAD) penalized function for variable selection. We analyze a yeast cell cycle gene expression data set to illustrate the usefulness of the single-index ODE model. In real data analysis, we group genes into functional modules using the smoothing spline clustering approach. We estimate state functions and their first derivatives for functional modules using penalized spline-based nonparametric mixed-effects models and the spline method. We substitute the estimates into the single-index ODE models, and then use the penalized profile least-squares procedure to identify network structures among the models. The results indicate that our model fits the data better than linear ODE models and our variable selection procedure identifies the interactions that may be missed by linear ODE models but confirmed in biological studies. In addition, Monte Carlo simulation studies are used to evaluate and compare the methods.

Statistical inference in mechanistic models: time warping for improved gradient matching

Inference in mechanistic models of non-linear differential equations is a challenging problem in current computational statistics. Due to the high computational costs of numerically solving the differential equations in every step of an iterative parameter adaptation scheme, approximate methods based on gradient matching have become popular. However, these methods critically depend on the smoothing scheme for function interpolation. The present article adapts an idea from manifold learning and demonstrates that a time warping approach aiming to homogenize intrinsic length scales can lead to a significant improvement in parameter estimation accuracy. We demonstrate the effectiveness of this scheme on noisy data from two dynamical systems with periodic limit cycle, a biopathway, and an application from soft-tissue mechanics. Our study also provides a comparative evaluation on a wide range of signal-to-noise ratios.

Identifying the dynamic gene regulatory network during latent HIV-1 reactivation using high-dimensional ordinary differential equations

Reactivation of latently infected cells has emerged as an important strategy for eradication of HIV. However, genetic mechanisms of regulation after reactivation remain unclear. We describe a five-step pipeline to study the dynamics of the gene regulatory network following a viral reactivation using high-dimensional ordinary differential equations. Our pipeline implements a combination of five different methods, by detecting temporally differentially expressed genes (step 1), clustering genes with similar temporal expression patterns into a small number of response modules (step2), performing a functional enrichment analysis within each gene response module (step 3), identifying a network structure based on the gene response modules using ordinary differential equations (ODE) and a high-dimensional variable selection technique (step 4), and obtaining a gene regulatory model based on refined parameter estimates using nonlinear least squares (step 5). We applied our pipeline to a time course gene expression data of latently infected T-cells following a latency-reversion.

Modelling and parameter inference of predator-prey dynamics in heterogeneous environments using the direct integral approach

Most bacterial habitats are topographically complex in the micro scale. Important examples include the gastrointestinal and tracheal tracts, and the soil. Although there are myriad theoretical studies that explore the role of spatial structures on antagonistic interactions (predation, competition) among animals, there are many fewer experimental studies that have explored, validated and quantified their predictions. In this study, we experimentally monitored the temporal dynamic of the predatory bacterium Bdellovibrio bacteriovorus, and its prey, the bacterium Burkholderia stabilis in a structured habitat consisting of sand under various regimes of wetness. We constructed a dynamic model, and estimated its parameters by further developing the direct integral method, a novel estimation procedure that exploits the separability of the states and parameters in the model. We also verified that one of our parameter estimates was consistent with its known, directly measured value from the literature. The ability of the model to fit the data combined with realistic parameter estimates indicate that bacterial predation in the sand can be described by a relatively simple model, and stress the importance of prey refuge on predation dynamics in heterogeneous environments.

Prediction of the containment of HIV infection by antiretroviral therapy - a variable structure control approach

It is demonstrated that the reachability paradigm from variable structure control theory is a suitable framework to monitor and predict the progression of the human immunodeficiency virus (HIV) infection following initiation of antiretroviral therapy (ART). A manifold is selected which characterises the infection-free steady-state. A model of HIV infection together with an associated reachability analysis is used to formulate a dynamical condition for the containment of HIV infection on the manifold. This condition is tested using data from two different HIV clinical trials which contain measurements of the CD4+ T cell count and HIV load in the peripheral blood collected from HIV infected individuals for the six month period following initiation of ART. The biological rates of the model are estimated using the multi-point identification method and data points collected in the initial period of the trial. Using the parameter estimates and the numerical solutions of the model, the predictions of the reachability analysis are shown to be consistent with the clinical diagnosis at the conclusion of the trial. The methodology captures the dynamical characteristics of eventual successful, failed and marginal outcomes. The findings evidence that the reachability analysis is an appropriate tool to monitor and develop personalised antiretroviral treatment.

Dynamical systems modeling of early childhood self-regulation

Self-regulation can be conceptualized in terms of dynamic tension between highly probable reactions (prepotent responses) and use of strategies that can modulate those reactions (executive processes). This study investigated the value of a dynamical systems approach to the study of early childhood self-regulation. Specifically, ordinary differential equations (ODEs) were used to model the interactive influences of 115 36-month-olds' executive processes (strategy use) and prepotent responses to waiting to open a gift (desire for the gift and frustration about waiting to open it). Using a pair of coupled second-order ODEs in a nonlinear mixed effects framework, the study tested predictions for specific within- and between-child patterns of prepotent response-executive process coupling. Dynamic modeling results articulated the limits of 36-month olds' strategic efforts. They engaged executive processes when their prepotent responding levels were high, which delayed the resurgence of prepotent responses, but ultimately did not damp prepotent responding over the course of the wait. There was, however, preliminary evidence that the effectiveness of 36-month-olds' self-regulation depended upon child characteristics. Externalizing behavior problems were associated with more regulatory interference. Temperamental negative affectivity was marginally associated with more regulatory inefficiency. Compared with conventional methods of studying self-regulation, dynamic modeling yielded complementary and unique findings, suggesting its potential. (PsycINFO Database Record

Network Reconstruction From High-Dimensional Ordinary Differential Equations

We consider the task of learning a dynamical system from high-dimensional time-course data. For instance, we might wish to estimate a gene regulatory network from gene expression data measured at discrete time points. We model the dynamical system nonparametrically as a system of additive ordinary differential equations. Most existing methods for parameter estimation in ordinary differential equations estimate the derivatives from noisy observations. This is known to be challenging and inefficient. We propose a novel approach that does not involve derivative estimation. We show that the proposed method can consistently recover the true network structure even in high dimensions, and we demonstrate empirical improvement over competing approaches. Supplementary materials for this article are available online.

Analysis of Practical Identifiability of a Viral Infection Model

Mathematical modelling approaches have granted a significant contribution to life sciences and beyond to understand experimental results. However, incomplete and inadequate assessments in parameter estimation practices hamper the parameter reliability, and consequently the insights that ultimately could arise from a mathematical model. To keep the diligent works in modelling biological systems from being mistrusted, potential sources of error must be acknowledged. Employing a popular mathematical model in viral infection research, existing means and practices in parameter estimation are exemplified. Numerical results show that poor experimental data is a main source that can lead to erroneous parameter estimates despite the use of innovative parameter estimation algorithms. Arbitrary choices of initial conditions as well as data asynchrony distort the parameter estimates but are often overlooked in modelling studies. This work stresses the existence of several sources of error buried in reports of modelling biological systems, voicing the need for assessing the sources of error, consolidating efforts in solving the immediate difficulties, and possibly reconsidering the use of mathematical modelling to quantify experimental data.

Approximate parameter inference in systems biology using gradient matching: a comparative evaluation

BACKGROUND

A challenging problem in current systems biology is that of parameter inference in biological pathways expressed as coupled ordinary differential equations (ODEs). Conventional methods that repeatedly numerically solve the ODEs have large associated computational costs. Aimed at reducing this cost, new concepts using gradient matching have been proposed, which bypass the need for numerical integration. This paper presents a recently established adaptive gradient matching approach, using Gaussian processes (GPs), combined with a parallel tempering scheme, and conducts a comparative evaluation with current state-of-the-art methods used for parameter inference in ODEs. Among these contemporary methods is a technique based on reproducing kernel Hilbert spaces (RKHS). This has previously shown promising results for parameter estimation, but under lax experimental settings. We look at a range of scenarios to test the robustness of this method. We also change the approach of inferring the penalty parameter from AIC to cross validation to improve the stability of the method.

METHODS

Methodology for the recently proposed adaptive gradient matching method using GPs, upon which we build our new method, is provided. Details of a competing method using RKHS are also described here.

RESULTS

We conduct a comparative analysis for the methods described in this paper, using two benchmark ODE systems. The analyses are repeated under different experimental settings, to observe the sensitivity of the techniques.

CONCLUSIONS

Our study reveals that for known noise variance, our proposed method based on GPs and parallel tempering achieves overall the best performance. When the noise variance is unknown, the RKHS method proves to be more robust.

Statistical analysis of differential equations: introducing probability measures on numerical solutions

In this paper, we present a formal quantification of uncertainty induced by numerical solutions of ordinary and partial differential equation models. Numerical solutions of differential equations contain inherent uncertainties due to the finite-dimensional approximation of an unknown and implicitly defined function. When statistically analysing models based on differential equations describing physical, or other naturally occurring, phenomena, it can be important to explicitly account for the uncertainty introduced by the numerical method. Doing so enables objective determination of this source of uncertainty, relative to other uncertainties, such as those caused by data contaminated with noise or model error induced by missing physical or inadequate descriptors. As ever larger scale mathematical models are being used in the sciences, often sacrificing complete resolution of the differential equation on the grids used, formally accounting for the uncertainty in the numerical method is becoming increasingly more important. This paper provides the formal means to incorporate this uncertainty in a statistical model and its subsequent analysis. We show that a wide variety of existing solvers can be randomised, inducing a probability measure over the solutions of such differential equations. These measures exhibit contraction to a Dirac measure around the true unknown solution, where the rates of convergence are consistent with the underlying deterministic numerical method. Furthermore, we employ the method of modified equations to demonstrate enhanced rates of convergence to stochastic perturbations of the original deterministic problem. Ordinary differential equations and elliptic partial differential equations are used to illustrate the approach to quantify uncertainty in both the statistical analysis of the forward and inverse problems.

A Comparison of Two-Stage Approaches for Fitting Nonlinear Ordinary Differential Equation Models with Mixed Effects

Several approaches exist for estimating the derivatives of observed data for model exploration purposes, including functional data analysis (FDA; Ramsay & Silverman, 2005 ), generalized local linear approximation (GLLA; Boker, Deboeck, Edler, & Peel, 2010 ), and generalized orthogonal local derivative approximation (GOLD; Deboeck, 2010 ). These derivative estimation procedures can be used in a two-stage process to fit mixed effects ordinary differential equation (ODE) models. While the performance and utility of these routines for estimating linear ODEs have been established, they have not yet been evaluated in the context of nonlinear ODEs with mixed effects. We compared properties of the GLLA and GOLD to an FDA-based two-stage approach denoted herein as functional ordinary differential equation with mixed effects (FODEmixed) in a Monte Carlo (MC) study using a nonlinear coupled oscillators model with mixed effects. Simulation results showed that overall, the FODEmixed outperformed both the GLLA and GOLD across all the embedding dimensions considered, but a novel use of a fourth-order GLLA approach combined with very high embedding dimensions yielded estimation results that almost paralleled those from the FODEmixed. We discuss the strengths and limitations of each approach and demonstrate how output from each stage of FODEmixed may be used to inform empirical modeling of young children's self-regulation.