Optimization and model reduction in the high dimensional parameter space of a budding yeast cell cycle model

Mini-batch optimization enables training of ODE models on large-scale datasets

Quantitative dynamic models are widely used to study cellular signal processing. A critical step in modelling is the estimation of unknown model parameters from experimental data. As model sizes and datasets are steadily growing, established parameter optimization approaches for mechanistic models become computationally extremely challenging. Mini-batch optimization methods, as employed in deep learning, have better scaling properties. In this work, we adapt, apply, and benchmark mini-batch optimization for ordinary differential equation (ODE) models, thereby establishing a direct link between dynamic modelling and machine learning. On our main application example, a large-scale model of cancer signaling, we benchmark mini-batch optimization against established methods, achieving better optimization results and reducing computation by more than an order of magnitude. We expect that our work will serve as a first step towards mini-batch optimization tailored to ODE models and enable modelling of even larger and more complex systems than what is currently possible.

Efficient gradient-based parameter estimation for dynamic models using qualitative data

MOTIVATION

Unknown parameters of dynamical models are commonly estimated from experimental data. However, while various efficient optimization and uncertainty analysis methods have been proposed for quantitative data, methods for qualitative data are rare and suffer from bad scaling and convergence.

RESULTS

Here, we propose an efficient and reliable framework for estimating the parameters of ordinary differential equation models from qualitative data. In this framework, we derive a semi-analytical algorithm for gradient calculation of the optimal scaling method developed for qualitative data. This enables the use of efficient gradient-based optimization algorithms. We demonstrate that the use of gradient information improves performance of optimization and uncertainty quantification on several application examples. On average, we achieve a speedup of more than one order of magnitude compared to gradient-free optimization. In addition, in some examples, the gradient-based approach yields substantially improved objective function values and quality of the fits. Accordingly, the proposed framework substantially improves the parameterization of models from qualitative data.

AVAILABILITY AND IMPLEMENTATION

The proposed approach is implemented in the open-source Python Parameter EStimation TOolbox (pyPESTO). pyPESTO is available at https://github.com/ICB-DCM/pyPESTO. All application examples and code to reproduce this study are available at https://doi.org/10.5281/zenodo.4507613.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Efficient gradient-based parameter estimation for dynamic models using qualitative data

Motivation

Unknown parameters of dynamical models are commonly estimated from experimental data. However, while various efficient optimization and uncertainty analysis methods have been proposed for quantitative data, methods for qualitative data are rare and suffer from bad scaling and convergence.

Results

Here, we propose an efficient and reliable framework for estimating the parameters of ordinary differential equation models from qualitative data. In this framework, we derive a semi-analytical algorithm for gradient calculation of the optimal scaling method developed for qualitative data. This enables the use of efficient gradient-based optimization algorithms. We demonstrate that the use of gradient information improves performance of optimization and uncertainty quantification on several application examples. On average, we achieve a speedup of more than one order of magnitude compared to gradient-free optimization. In addition, in some examples, the gradient-based approach yields substantially improved objective function values and quality of the fits. Accordingly, the proposed framework substantially improves the parameterization of models from qualitative data.

Availability and implementation

The proposed approach is implemented in the open-source Python Parameter EStimation TOolbox (pyPESTO). pyPESTO is available at https://github.com/ICB-DCM/pyPESTO. All application examples and code to reproduce this study are available at https://doi.org/10.5281/zenodo.4507613.

Supplementary information

Supplementary data are available at Bioinformatics online.

Bayesian inference using qualitative observations of underlying continuous variables

MOTIVATION

Recent work has demonstrated the feasibility of using non-numerical, qualitative data to parameterize mathematical models. However, uncertainty quantification (UQ) of such parameterized models has remained challenging because of a lack of a statistical interpretation of the objective functions used in optimization.

RESULTS

We formulated likelihood functions suitable for performing Bayesian UQ using qualitative observations of underlying continuous variables or a combination of qualitative and quantitative data. To demonstrate the resulting UQ capabilities, we analyzed a published model for immunoglobulin E (IgE) receptor signaling using synthetic qualitative and quantitative datasets. Remarkably, estimates of parameter values derived from the qualitative data were nearly as consistent with the assumed ground-truth parameter values as estimates derived from the lower throughput quantitative data. These results provide further motivation for leveraging qualitative data in biological modeling.

AVAILABILITY AND IMPLEMENTATION

The likelihood functions presented here are implemented in a new release of PyBioNetFit, an open-source application for analyzing Systems Biology Markup Language- and BioNetGen Language-formatted models, available online at www.github.com/lanl/PyBNF.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Parameter Estimation and Uncertainty Quantification for Systems Biology Models

Mathematical models can provide quantitative insights into immunoreceptor signaling, and other biological processes, but require parameterization and uncertainty quantification before reliable predictions become possible. We review currently available methods and software tools to address these problems. We consider gradient-based and gradient-free methods for point estimation of parameter values, and methods of profile likelihood, bootstrapping, and Bayesian inference for uncertainty quantification. We consider recent and potential future applications of these methods to systems-level modeling of immune-related phenomena.

PyBioNetFit and the Biological Property Specification Language

In systems biology modeling, important steps include model parameterization, uncertainty quantification, and evaluation of agreement with experimental observations. To help modelers perform these steps, we developed the software PyBioNetFit, which in addition supports checking models against known system properties and solving design problems. PyBioNetFit introduces Biological Property Specification Language (BPSL) for the formal declaration of system properties. BPSL allows qualitative data to be used alone or in combination with quantitative data. PyBioNetFit performs parameterization with parallelized metaheuristic optimization algorithms that work directly with existing model definition standards: BioNetGen Language (BNGL) and Systems Biology Markup Language (SBML). We demonstrate PyBioNetFit's capabilities by solving various example problems, including the challenging problem of parameterizing a 153-parameter model of cell cycle control in yeast based on both quantitative and qualitative data. We demonstrate the model checking and design applications of PyBioNetFit and BPSL by analyzing a model of targeted drug interventions in autophagy signaling.

Quasi-Newton Stochastic Optimization Algorithm for Parameter Estimation of a Stochastic Model of the Budding Yeast Cell Cycle

Parameter estimation in discrete or continuous deterministic cell cycle models is challenging for several reasons, including the nature of what can be observed, and the accuracy and quantity of those observations. The challenge is even greater for stochastic models, where the number of simulations and amount of empirical data must be even larger to obtain statistically valid parameter estimates. The two main contributions of this work are (1) stochastic model parameter estimation based on directly matching multivariate probability distributions, and (2) a new quasi-Newton algorithm class QNSTOP for stochastic optimization problems. QNSTOP directly uses the random objective function value samples rather than creating ensemble statistics. QNSTOP is used here to directly match empirical and simulated joint probability distributions rather than matching summary statistics. Results are given for a current state-of-the-art stochastic cell cycle model of budding yeast, whose predictions match well some summary statistics and one-dimensional distributions from empirical data, but do not match well the empirical joint distributions. The nature of the mismatch provides insight into the weakness in the stochastic model.

Using both qualitative and quantitative data in parameter identification for systems biology models

In systems biology, qualitative data are often generated, but rarely used to parameterize models. We demonstrate an approach in which qualitative and quantitative data can be combined for parameter identification. In this approach, qualitative data are converted into inequality constraints imposed on the outputs of the model. These inequalities are used along with quantitative data points to construct a single scalar objective function that accounts for both datasets. To illustrate the approach, we estimate parameters for a simple model describing Raf activation. We then apply the technique to a more elaborate model characterizing cell cycle regulation in yeast. We incorporate both quantitative time courses (561 data points) and qualitative phenotypes of 119 mutant yeast strains (1647 inequalities) to perform automated identification of 153 model parameters. We quantify parameter uncertainty using a profile likelihood approach. Our results indicate the value of combining qualitative and quantitative data to parameterize systems biology models.

Much of the data generated in biology is qualitative, but exploiting such data to inform models of biological systems remains a challenge. Here, the authors demonstrate an approach that allows use of both quantitative and qualitative data for parameterising dynamical models.

Computational Model Predicts the Effects of Targeting Cellular Metabolism in Pancreatic Cancer

Reprogramming of energy metabolism is a hallmark of cancer that enables the cancer cells to meet the increased energetic requirements due to uncontrolled proliferation. One prominent example is pancreatic ductal adenocarcinoma, an aggressive form of cancer with an overall 5-year survival rate of 5%. The reprogramming mechanism in pancreatic cancer involves deregulated uptake of glucose and glutamine and other opportunistic modes of satisfying energetic demands in a hypoxic and nutrient-poor environment. In the current study, we apply systems biology approaches to enable a better understanding of the dynamics of the distinct metabolic alterations in KRAS-mediated pancreatic cancer, with the goal of impeding early cell proliferation by identifying the optimal metabolic enzymes to target. We have constructed a kinetic model of metabolism represented as a set of ordinary differential equations that describe time evolution of the metabolite concentrations in glycolysis, glutaminolysis, tricarboxylic acid cycle and the pentose phosphate pathway. The model is comprised of 46 metabolites and 53 reactions. The mathematical model is fit to published enzyme knockdown experimental data. We then applied the model to perform in silico enzyme modulations and evaluate the effects on cell proliferation. Our work identifies potential combinations of enzyme knockdown, metabolite inhibition, and extracellular conditions that impede cell proliferation. Excitingly, the model predicts novel targets that can be tested experimentally. Therefore, the model is a tool to predict the effects of inhibiting specific metabolic reactions within pancreatic cancer cells, which is difficult to measure experimentally, as well as test further hypotheses toward targeted therapies.

Predicting network modules of cell cycle regulators using relative protein abundance statistics

BACKGROUND

Parameter estimation in systems biology is typically done by enforcing experimental observations through an objective function as the parameter space of a model is explored by numerical simulations. Past studies have shown that one usually finds a set of "feasible" parameter vectors that fit the available experimental data equally well, and that these alternative vectors can make different predictions under novel experimental conditions. In this study, we characterize the feasible region of a complex model of the budding yeast cell cycle under a large set of discrete experimental constraints in order to test whether the statistical features of relative protein abundance predictions are influenced by the topology of the cell cycle regulatory network.

RESULTS

Using differential evolution, we generate an ensemble of feasible parameter vectors that reproduce the phenotypes (viable or inviable) of wild-type yeast cells and 110 mutant strains. We use this ensemble to predict the phenotypes of 129 mutant strains for which experimental data is not available. We identify 86 novel mutants that are predicted to be viable and then rank the cell cycle proteins in terms of their contributions to cumulative variability of relative protein abundance predictions. Proteins involved in "regulation of cell size" and "regulation of G1/S transition" contribute most to predictive variability, whereas proteins involved in "positive regulation of transcription involved in exit from mitosis," "mitotic spindle assembly checkpoint" and "negative regulation of cyclin-dependent protein kinase by cyclin degradation" contribute the least. These results suggest that the statistics of these predictions may be generating patterns specific to individual network modules (START, S/G2/M, and EXIT). To test this hypothesis, we develop random forest models for predicting the network modules of cell cycle regulators using relative abundance statistics as model inputs. Predictive performance is assessed by the areas under receiver operating characteristics curves (AUC). Our models generate an AUC range of 0.83-0.87 as opposed to randomized models with AUC values around 0.50.

CONCLUSIONS

By using differential evolution and random forest modeling, we show that the model prediction statistics generate distinct network module-specific patterns within the cell cycle network.

Molecular Network Dynamics of Cell Cycle Control: Periodicity of Start and Finish

The cell division cycle is controlled by a complex regulatory network which ensures that the phases of the cell cycle are executed in the right order. This regulatory network receives signals from the environment, monitors the state of the DNA, and decides timings of cell cycle events. The underlying transcriptional and post-translational regulatory interactions lead to complex dynamical responses, such as the oscillations in the levels of cell cycle proteins driven by intertwined biochemical reactions. A cell moves between different phases of its cycle similar to a dynamical system switching between its steady states. The complex molecular network driving these phases has been investigated in previous computational systems biology studies. Here, we review the critical physiological and molecular transitions that occur in the cell cycle and discuss the role of mathematical modeling in elucidating these transitions and understand cell cycle synchronization.

A Model of Yeast Cell-Cycle Regulation Based on a Standard Component Modeling Strategy for Protein Regulatory Networks

To understand the molecular mechanisms that regulate cell cycle progression in eukaryotes, a variety of mathematical modeling approaches have been employed, ranging from Boolean networks and differential equations to stochastic simulations. Each approach has its own characteristic strengths and weaknesses. In this paper, we propose a “standard component” modeling strategy that combines advantageous features of Boolean networks, differential equations and stochastic simulations in a framework that acknowledges the typical sorts of reactions found in protein regulatory networks. Applying this strategy to a comprehensive mechanism of the budding yeast cell cycle, we illustrate the potential value of standard component modeling. The deterministic version of our model reproduces the phenotypic properties of wild-type cells and of 125 mutant strains. The stochastic version of our model reproduces the cell-to-cell variability of wild-type cells and the partial viability of the CLB2-dbΔ clb5Δ mutant strain. Our simulations show that mathematical modeling with “standard components” can capture in quantitative detail many essential properties of cell cycle control in budding yeast.

Coherent organization in gene regulation: a study on six networks

Structural and dynamical fingerprints of evolutionary optimization in biological networks are still unclear. Here we analyze the dynamics of genetic regulatory networks responsible for the regulation of cell cycle and cell differentiation in three organisms or cell types each, and show that they follow a version of Hebb's rule which we have termed coherence. More precisely, we find that simultaneously expressed genes with a common target are less likely to act antagonistically at the attractors of the regulatory dynamics. We then investigate the dependence of coherence on structural parameters, such as the mean number of inputs per node and the activatory/repressory interaction ratio, as well as on dynamically determined quantities, such as the basin size and the number of expressed genes.

Model-Based Analysis of Cell Cycle Responses to Dynamically Changing Environments

Cell cycle progression is carefully coordinated with a cell's intra- and extracellular environment. While some pathways have been identified that communicate information from the environment to the cell cycle, a systematic understanding of how this information is dynamically processed is lacking. We address this by performing dynamic sensitivity analysis of three mathematical models of the cell cycle in Saccharomyces cerevisiae. We demonstrate that these models make broadly consistent qualitative predictions about cell cycle progression under dynamically changing conditions. For example, it is shown that the models predict anticorrelated changes in cell size and cell cycle duration under different environments independently of the growth rate. This prediction is validated by comparison to available literature data. Other consistent patterns emerge, such as widespread nonmonotonic changes in cell size down generations in response to parameter changes. We extend our analysis by investigating glucose signalling to the cell cycle, showing that known regulation of Cln3 translation and Cln1,2 transcription by glucose is sufficient to explain the experimentally observed changes in cell cycle dynamics at different glucose concentrations. Together, these results provide a framework for understanding the complex responses the cell cycle is capable of producing in response to dynamic environments.

Conditional robustness analysis for fragility discovery and target identification in biochemical networks and in cancer systems biology

Background

The study of cancer therapy is a key issue in the field of oncology research and the development of target therapies is one of the main problems currently under investigation. This is particularly relevant in different types of tumor where traditional chemotherapy approaches often fail, such as lung cancer.

Results

We started from the general definition of robustness introduced by Kitano and applied it to the analysis of dynamical biochemical networks, proposing a new algorithm based on moment independent analysis of input/output uncertainty. The framework utilizes novel computational methods which enable evaluating the model fragility with respect to quantitative performance measures and parameters such as reaction rate constants and initial conditions. The algorithm generates a small subset of parameters that can be used to act on complex networks and to obtain the desired behaviors. We have applied the proposed framework to the EGFR-IGF1R signal transduction network, a crucial pathway in lung cancer, as an example of Cancer Systems Biology application in drug discovery. Furthermore, we have tested our framework on a pulse generator network as an example of Synthetic Biology application, thus proving the suitability of our methodology to the characterization of the input/output synthetic circuits.

Conclusions

The achieved results are of immediate practical application in computational biology, and while we demonstrate their use in two specific examples, they can in fact be used to study a wider class of biological systems.

Electronic supplementary material

The online version of this article (doi:10.1186/s12918-015-0216-5) contains supplementary material, which is available to authorized users.

Mechanistic modeling confronts the complexity of molecular cell biology

Mechanistic modeling has the potential to transform how cell biologists contend with the inescapable complexity of modern biology. I am a physiologist–electrical engineer–systems biologist who has been working at the level of cell biology for the past 24 years. This perspective aims 1) to convey why we build models, 2) to enumerate the major approaches to modeling and their philosophical differences, 3) to address some recurrent concerns raised by experimentalists, and then 4) to imagine a future in which teams of experimentalists and modelers build—and subject to exhaustive experimental tests—models covering the entire spectrum from molecular cell biology to human pathophysiology. There is, in my view, no technical obstacle to this future, but it will require some plasticity in the biological research mind-set.

Summary of the DREAM8 Parameter Estimation Challenge: Toward Parameter Identification for Whole-Cell Models

Whole-cell models that explicitly represent all cellular components at the molecular level have the potential to predict phenotype from genotype. However, even for simple bacteria, whole-cell models will contain thousands of parameters, many of which are poorly characterized or unknown. New algorithms are needed to estimate these parameters and enable researchers to build increasingly comprehensive models. We organized the Dialogue for Reverse Engineering Assessments and Methods (DREAM) 8 Whole-Cell Parameter Estimation Challenge to develop new parameter estimation algorithms for whole-cell models. We asked participants to identify a subset of parameters of a whole-cell model given the model’s structure and in silico “experimental” data. Here we describe the challenge, the best performing methods, and new insights into the identifiability of whole-cell models. We also describe several valuable lessons we learned toward improving future challenges. Going forward, we believe that collaborative efforts supported by inexpensive cloud computing have the potential to solve whole-cell model parameter estimation.

Whole-cell models promise to enable rational bioengineering by predicting how cells behave. Even for simple bacteria, whole-cell models require thousands of parameters, many of which are poorly characterized or unknown. New approaches are needed to estimate these parameters. We organized the Dialogue for Reverse Engineering Assessments and Methods (DREAM) 8 Whole-Cell Parameter Estimation Challenge to develop new approaches for whole-cell model parameter identification. Here we describe the challenge, the best performing methods, new insights into the identifiability of whole-cell models, and several lessons we learned for improving future challenges. Going forward, we believe that collaborative efforts have the potential to produce powerful tools for identifying whole-cell models.

A Mathematical Framework for Understanding Four-Dimensional Heterogeneous Differentiation of CD4+ T Cells

At least four distinct lineages of CD4+ T cells play diverse roles in the immune system. Both in vivo and in vitro, naïve CD4+ T cells often differentiate into a variety of cellular phenotypes. Previously, we developed a mathematical framework to study heterogeneous differentiation of two lineages governed by a mutual-inhibition motif. To understand heterogeneous differentiation of CD4+ T cells involving more than two lineages, we present here a mathematical framework for the analysis of multiple stable steady states in dynamical systems with multiple state variables interacting through multiple mutual-inhibition loops. A mathematical model for CD4+ T cells based on this framework can reproduce known properties of heterogeneous differentiation involving multiple lineages of this cell differentiation system, such as heterogeneous differentiation of TH1-TH2, TH1-TH17 and iTReg-TH17 under single or mixed types of differentiation stimuli. The model shows that high concentrations of differentiation stimuli favor the formation of phenotypes with co-expression of lineage-specific master regulators.

Quantitative determinants of aerobic glycolysis identify flux through the enzyme GAPDH as a limiting step

Aerobic glycolysis or the Warburg Effect (WE) is characterized by the increased metabolism of glucose to lactate. It remains unknown what quantitative changes to the activity of metabolism are necessary and sufficient for this phenotype. We developed a computational model of glycolysis and an integrated analysis using metabolic control analysis (MCA), metabolomics data, and statistical simulations. We identified and confirmed a novel mode of regulation specific to aerobic glycolysis where flux through GAPDH, the enzyme separating lower and upper glycolysis, is the rate-limiting step in the pathway and the levels of fructose (1,6) bisphosphate (FBP), are predictive of the rate and control points in glycolysis. Strikingly, negative flux control was found and confirmed for several steps thought to be rate-limiting in glycolysis. Together, these findings enumerate the biochemical determinants of the WE and suggest strategies for identifying the contexts in which agents that target glycolysis might be most effective.

DOI:http://dx.doi.org/10.7554/eLife.03342.001

Cells generate energy from a sugar called glucose via a process called glycolysis. This process involves many enzymes that catalyze 10 different chemical reactions, and it essentially converts glucose step-by-step into a simpler chemical called pyruvate.

Pyruvate is then normally transported into structures within the cell called mitochondria, where it is further broken down using oxygen to release more energy. However, in cells that are rapidly dividing, pyruvate is converted into another chemical called lactate—which releases energy more quickly, but releases less energy overall. Cancer cells often convert most of their glucose into lactate, rather than breaking down pyruvate in their mitochondria: an observation known as the ‘Warburg effect’. And while many factors affect how a cell releases energy from pyruvate, it remains unclear what regulates which of these biochemical processes is most common in a living cell.

In this study, Shestov et al. have developed a computational model for the process of glycolysis and used this to investigate the causes of the Warburg Effect. The model was based on the known characteristics of the enzymes and chemical reactions involved at each step. It predicted that the activity of the enzyme called GAPDH, which carries out the sixth step in glycolysis, in many cases affects how much lactate is produced. This suggests that this enzyme represents a bottleneck in the pathway.

Next, Shestov et al. performed experiments where they used drugs to block different stages of the glycolysis pathway, and confirmed that the GAPDH enzyme is important for regulating this pathway in living cancer cells too. In these treated cells, the levels of a chemical called fructose-1,6-biphosphate (which is made in a step in the pathway between glucose and pyruvate) were either very high or very low. Shestov et al. proposed that the flow of chemicals through the glycolysis pathway is controlled by the GAPDH enzyme when the chemicals used by the enzymes upstream of GAPDH in the pathway (which includes fructose-1,6-biphosphate) are plentiful. However, if these chemicals are limited, other enzymes that are involved in earlier steps of the pathway regulate the process instead.

The findings of Shestov et al. reveal that the regulation of glycolysis is more complex than previously thought, and is also very different when cells are undergoing the Warburg Effect. In the future, these findings might help to identify the types of cancer that could be effectively treated using drugs that target the glycolysis process, which are currently being tested in pre-clinical studies.

DOI:http://dx.doi.org/10.7554/eLife.03342.002

A stochastic model correctly predicts changes in budding yeast cell cycle dynamics upon periodic expression of CLN2

In this study, we focus on a recent stochastic budding yeast cell cycle model. First, we estimate the model parameters using extensive data sets: phenotypes of 110 genetic strains, single cell statistics of wild type and cln3 strains. Optimization of stochastic model parameters is achieved by an automated algorithm we recently used for a deterministic cell cycle model. Next, in order to test the predictive ability of the stochastic model, we focus on a recent experimental study in which forced periodic expression of CLN2 cyclin (driven by MET3 promoter in cln3 background) has been used to synchronize budding yeast cell colonies. We demonstrate that the model correctly predicts the experimentally observed synchronization levels and cell cycle statistics of mother and daughter cells under various experimental conditions (numerical data that is not enforced in parameter optimization), in addition to correctly predicting the qualitative changes in size control due to forced CLN2 expression. Our model also generates a novel prediction: under frequent CLN2 expression pulses, G1 phase duration is bimodal among small-born cells. These cells originate from daughters with extended budded periods due to size control during the budded period. This novel prediction and the experimental trends captured by the model illustrate the interplay between cell cycle dynamics, synchronization of cell colonies, and size control in budding yeast.