Optimal experiment design for model selection in biochemical networks

Rationalised experiment design for parameter estimation with sensitivity clustering

Quantitative experiments are essential for investigating, uncovering, and confirming our understanding of complex systems, necessitating the use of effective and robust experimental designs. Despite generally outperforming other approaches, the broader adoption of model-based design of experiments (MBDoE) has been hindered by oversimplified assumptions and computational overhead. To address this, we present PARameter SEnsitivity Clustering (PARSEC), an MBDoE framework that identifies informative measurable combinations through parameter sensitivity (PS) clustering. We combined PARSEC with a new variant of Approximate Bayesian Computation-based parameter estimation for rapid, automated assessment and ranking of experiment designs. Using two kinetic model systems with distinct dynamical features, we show that PARSEC-based experiments improve the parameter estimation of a complex system. By its inherent formulation, PARSEC can account for experimental restrictions and parameter variability. Moreover, we demonstrate that there is a strong correlation between sample size and the optimal number of PS clusters in PARSEC, offering a novel method to determine the ideal sampling for experiments. This validates our argument for employing parameter sensitivity in experiment design and illustrates the potential to leverage both model architecture and system dynamics to effectively explore the experimental design space.

An OpenSim-Based Closed-Loop Biomechanical Wrist Model for Subject-Specific Pathological Tremor Simulation

OBJECTIVE

A pathological tremor (PT) is an involuntary rhythmic movement of varying frequency and amplitude that affects voluntary motion, thus compromising individuals' independence. A comprehensive model incorporating PT's physiological and biomechanical aspects can enhance our understanding of the disorder and provide valuable insights for therapeutic approaches. This study aims to build a biomechanical model of pathological tremors using OpenSim's realistic musculoskeletal representation of the human wrist with two degrees of freedom.

METHODS

We implemented a Matlab/OpenSim interface for a forward dynamics simulation, which allows for the modeling, simulation, and design of a physiological H∞ closed-loop control. This system replicates pathological tremors similar to those observed in patients when their arm is extended forward, the wrist is pronated, and the hand is subject to gravity forces. The model was individually tuned to five subjects (four Parkinson's disease patients and one diagnosed with essential tremor), each exhibiting distinct tremor characteristics measured by an inertial sensor and surface EMG electrodes. Simulation agreement with the experiments for EMGs, central frequency, joint angles, and angular velocities were evaluated by Jensen-Shannon divergence, histogram centroid error, and histogram intersection.

RESULTS

The model emulated individual tremor statistical characteristics, including muscle activations, frequency, variability, and wrist kinematics, with greater accuracy for the four Parkinson's patients than the essential tremor.

CONCLUSION

The proposed model replicated the main statistical features of subject-specific wrist tremor kinematics.

SIGNIFICANCE

Our methodology may facilitate the design of patient-specific rehabilitation devices for tremor suppression, such as neural prostheses and electromechanical orthoses.

Advanced data analysis in inertial confinement fusion and high energy density physics

Bayesian analysis enables flexible and rigorous definition of statistical model assumptions with well-characterized propagation of uncertainties and resulting inferences for single-shot, repeated, or even cross-platform data. This approach has a strong history of application to a variety of problems in physical sciences ranging from inference of particle mass from multi-source high-energy particle data to analysis of black-hole characteristics from gravitational wave observations. The recent adoption of Bayesian statistics for analysis and design of high-energy density physics (HEDP) and inertial confinement fusion (ICF) experiments has provided invaluable gains in expert understanding and experiment performance. In this Review, we discuss the basic theory and practical application of the Bayesian statistics framework. We highlight a variety of studies from the HEDP and ICF literature, demonstrating the power of this technique. Due to the computational complexity of multi-physics models needed to analyze HEDP and ICF experiments, Bayesian inference is often not computationally tractable. Two sections are devoted to a review of statistical approximations, efficient inference algorithms, and data-driven methods, such as deep-learning and dimensionality reduction, which play a significant role in enabling use of the Bayesian framework. We provide additional discussion of various applications of Bayesian and machine learning methods that appear to be sparse in the HEDP and ICF literature constituting possible next steps for the community. We conclude by highlighting community needs, the resolution of which will improve trust in data-driven methods that have proven critical for accelerating the design and discovery cycle in many application areas.

Socio-technical scales in socio-environmental modeling: Managing a system-of-systems modeling approach

System-of-systems approaches for integrated assessments have become prevalent in recent years. Such approaches integrate a variety of models from different disciplines and modeling paradigms to represent a socio-environmental (or social-ecological) system aiming to holistically inform policy and decision-making processes. Central to the system-of-systems approaches is the representation of systems in a multi-tier framework with nested scales. Current modeling paradigms, however, have disciplinary-specific lineage, leading to inconsistencies in the conceptualization and integration of socio-environmental systems. In this paper, a multidisciplinary team of researchers, from engineering, natural and social sciences, have come together to detail socio-technical practices and challenges that arise in the consideration of scale throughout the socio-environmental modeling process. We identify key paths forward, focused on explicit consideration of scale and uncertainty, strengthening interdisciplinary communication, and improvement of the documentation process. We call for a grand vision (and commensurate funding) for holistic system-of-systems research that engages researchers, stakeholders, and policy makers in a multi-tiered process for the co-creation of knowledge and solutions to major socio-environmental problems.

An experimental design tool to optimize inference precision in data-driven mathematical models of bacterial infections in vivo

The management of bacterial diseases calls for a detailed knowledge about the dynamic changes in host-bacteria interactions. Biological insights are gained by integrating experimental data with mechanistic mathematical models to infer experimentally unobservable quantities. This inter-disciplinary field would benefit from experiments with maximal information content yielding high-precision inference. Here, we present a computationally efficient tool for optimizing experimental design in terms of parameter inference in studies using isogenic-tagged strains. We study the effect of three experimental design factors: number of biological replicates, sampling timepoint selection and number of copies per tagged strain. We conduct a simulation study to establish the relationship between our optimality criterion and the size of parameter estimate confidence intervals, and showcase its application in a range of biological scenarios reflecting different dynamics patterns observed in experimental infections. We show that in low-variance systems with low killing and replication rates, predicting high-precision experimental designs is consistently achieved; higher replicate sizes and strategic timepoint selection yield more precise estimates. Finally, we address the question of resource allocation under constraints; given a fixed number of host animals and a constraint on total inoculum size per host, infections with fewer strains at higher copies per strain lead to higher-precision inference.

Postural instability via a loss of intermittent control in elderly and patients with Parkinson's disease: A model-based and data-driven approach

Postural instability is one of the major symptoms of Parkinson's disease. Here, we assimilated a model of intermittent delay feedback control during quiet standing into postural sway data from healthy young and elderly individuals as well as patients with Parkinson's disease to elucidate the possible mechanisms of instability. Specifically, we estimated the joint probability distribution of a set of parameters in the model using the Bayesian parameter inference such that the model with the inferred parameters can best-fit sway data for each individual. It was expected that the parameter values for three populations would distribute differently in the parameter space depending on their balance capability. Because the intermittent control model is parameterized by a parameter associated with the degree of intermittency in the control, it can represent not only the intermittent model but also the traditional continuous control model with no intermittency. We showed that the inferred parameter values for the three groups of individuals are classified into two major groups in the parameter space: one represents the intermittent control mostly for healthy people and patients with mild postural symptoms and the other the continuous control mostly for some elderly and patients with severe postural symptoms. The results of this study may be interpreted by postulating that increased postural instability in most Parkinson's patients and some elderly persons might be characterized as a dynamical disease.

Identifiability of a Binomial Synapse

Synapses are highly stochastic transmission units. A classical model describing this stochastic transmission is called the binomial model, and its underlying parameters can be estimated from postsynaptic responses to evoked stimuli. The accuracy of parameter estimates obtained via such a model-based approach depends on the identifiability of the model. A model is said to be structurally identifiable if its parameters can be uniquely inferred from the distribution of its outputs. However, this theoretical property does not necessarily imply practical identifiability. For instance, if the number of observations is low or if the recording noise is high, the model's parameters can only be loosely estimated. Structural identifiability, which is an intrinsic property of a model, has been widely characterized; but practical identifiability, which is a property of both the model and the experimental protocol, is usually only qualitatively assessed. Here, we propose a formal definition for the practical identifiability domain of a statistical model. For a given experimental protocol, this domain corresponds to the set of parameters for which the model is correctly identified as the ground truth compared to a simpler alternative model. Considering a model selection problem instead of a parameter inference problem allows to derive a non-arbitrary criterion for practical identifiability. We apply our definition to the study of neurotransmitter release at a chemical synapse. Our contribution to the analysis of synaptic stochasticity is three-fold: firstly, we propose a quantitative criterion for the practical identifiability of a statistical model, and compute the identifiability domains of different variants of the binomial release model (uni or multi-quantal, with or without short-term plasticity); secondly, we extend the Bayesian Information Criterion (BIC), a classically used tool for model selection, to models with correlated data (which is the case for most models of chemical synapses); finally, we show that our approach allows to perform data free model selection, i.e., to verify if a model used to fit data was indeed identifiable even without access to the data, but having only access to the fitted parameters.

A scalable method for parameter identification in kinetic models of metabolism using steady-state data

MOTIVATION

In kinetic models of metabolism, the parameter values determine the dynamic behaviour predicted by these models. Estimating parameters from in vivo experimental data require the parameters to be structurally identifiable, and the data to be informative enough to estimate these parameters. Existing methods to determine the structural identifiability of parameters in kinetic models of metabolism can only be applied to models of small metabolic networks due to their computational complexity. Additionally, a priori experimental design, a necessity to obtain informative data for parameter estimation, also does not account for using steady-state data to estimate parameters in kinetic models.

RESULTS

Here, we present a scalable methodology to structurally identify parameters for each flux in a kinetic model of metabolism based on the availability of steady-state data. In doing so, we also address the issue of determining the number and nature of experiments for generating steady-state data to estimate these parameters. By using a small metabolic network as an example, we show that most parameters in fluxes expressed by mechanistic enzyme kinetic rate laws can be identified using steady-state data, and the steady-state data required for their estimation can be obtained from selective experiments involving both substrate and enzyme level perturbations. The methodology can be used in combination with other identifiability and experimental design algorithms that use dynamic data to determine the most informative experiments requiring the least resources to perform.

AVAILABILITY AND IMPLEMENTATION

https://github.com/LMSE/ident.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Simulation and inference algorithms for stochastic biochemical reaction networks: from basic concepts to state-of-the-art

Stochasticity is a key characteristic of intracellular processes such as gene regulation and chemical signalling. Therefore, characterizing stochastic effects in biochemical systems is essential to understand the complex dynamics of living things. Mathematical idealizations of biochemically reacting systems must be able to capture stochastic phenomena. While robust theory exists to describe such stochastic models, the computational challenges in exploring these models can be a significant burden in practice since realistic models are analytically intractable. Determining the expected behaviour and variability of a stochastic biochemical reaction network requires many probabilistic simulations of its evolution. Using a biochemical reaction network model to assist in the interpretation of time-course data from a biological experiment is an even greater challenge due to the intractability of the likelihood function for determining observation probabilities. These computational challenges have been subjects of active research for over four decades. In this review, we present an accessible discussion of the major historical developments and state-of-the-art computational techniques relevant to simulation and inference problems for stochastic biochemical reaction network models. Detailed algorithms for particularly important methods are described and complemented with Matlab^® implementations. As a result, this review provides a practical and accessible introduction to computational methods for stochastic models within the life sciences community.

Robust Model Selection: Flatness-Based Optimal Experimental Design for a Biocatalytic Reaction

Considering the competitive and strongly regulated pharmaceutical industry, mathematical modeling and process systems engineering might be useful tools for implementing quality by design (QbD) and quality by control (QbC) strategies for low-cost but high-quality drugs. However, a crucial task in modeling (bio)pharmaceutical manufacturing processes is the reliable identification of model candidates from a set of various model hypotheses. To identify the best experimental design suitable for a reliable model selection and system identification is challenging for nonlinear (bio)pharmaceutical process models in general. This paper is the first to exploit differential flatness for model selection problems under uncertainty, and thus translates the model selection problem to advanced concepts of systems theory and controllability aspects, respectively. Here, the optimal controls for improved model selection trajectories are expressed analytically with low computational costs. We further demonstrate the impact of parameter uncertainties on the differential flatness-based method and provide an effective robustification strategy with the point estimate method for uncertainty quantification. In a simulation study, we consider a biocatalytic reaction step simulating the carboligation of aldehydes, where we successfully derive optimal controls for improved model selection trajectories under uncertainty.

Optimal parameter identification of synthetic gene networks using harmony search algorithm

Computational modeling of engineered gene circuits is an important while challenged task in systems biology. In order to describe and predict the response behaviors of genetic circuits using reliable model parameters, this paper applies an optimal experimental design(OED) method to obtain input signals. In order to obtain informative observations, this study focuses on maximizing Fisher information matrix(FIM)-based optimal criteria and to provide optimal inputs. Furthermore, this paper designs a two-stage optimization with the modified E-optimal criteria and applies harmony search(HS)-based OED algorithm to minimize estimation errors. The proposed optimal identification methodology involves estimation errors and the sample size to pursue a trade-off between estimation accuracy and measurement cost in modeling gene networks. The designed cost function takes two major factors into account, in which experimental costs are proportional to the number of time points. Experiments select two types of synthetic genetic networks to validate the effectiveness of the proposed HS-OED approach. Identification outcomes and analysis indicate the proposed HS-OED method outperforms two candidate OED approaches, with reduced computational effort.

Plant Synthetic Biology: Quantifying the "Known Unknowns" and Discovering the "Unknown Unknowns"

Biosensors, advanced microscopy, and single- cell transcriptomics are advancing plant synthetic biology.

Estimating parametric phenotypes that determine anthesis date in Zea mays: Challenges in combining ecophysiological models with genetics

Ecophysiological crop models encode intra-species behaviors using parameters that are presumed to summarize genotypic properties of individual lines or cultivars. These genotype-specific parameters (GSP’s) can be interpreted as quantitative traits that can be mapped or otherwise analyzed, as are more conventional traits. The goal of this study was to investigate the estimation of parameters controlling maize anthesis date with the CERES-Maize model, based on 5,266 maize lines from 11 plantings at locations across the eastern United States. High performance computing was used to develop a database of 356 million simulated anthesis dates in response to four CERES-Maize model parameters. Although the resulting estimates showed high predictive value (R2 = 0.94), three issues presented serious challenges for use of GSP’s as traits. First (expressivity), the model was unable to express the observed data for 168 to 3,339 lines (depending on the combination of site-years), many of which ended up sharing the same parameter value irrespective of genetics. Second, for 2,254 lines, the model reproduced the data, but multiple parameter sets were equally effective (equifinality). Third, parameter values were highly dependent (p<10⁻⁶⁹¹⁹) on the sets of environments used to estimate them (instability), calling in to question the assumption that they represent fundamental genetic traits. The issues of expressivity, equifinality and instability must be addressed before the genetic mapping of GSP’s becomes a robust means to help solve the genotype-to-phenotype problem in crops.

Optimal Objective-Based Experimental Design for Uncertain Dynamical Gene Networks with Experimental Error

In systems biology, network models are often used to study interactions among cellular components, a salient aim being to develop drugs and therapeutic mechanisms to change the dynamical behavior of the network to avoid undesirable phenotypes. Owing to limited knowledge, model uncertainty is commonplace and network dynamics can be updated in different ways, thereby giving multiple dynamic trajectories, that is, dynamics uncertainty. In this manuscript, we propose an experimental design method that can effectively reduce the dynamics uncertainty and improve performance in an interaction-based network. Both dynamics uncertainty and experimental error are quantified with respect to the modeling objective, herein, therapeutic intervention. The aim of experimental design is to select among a set of candidate experiments the experiment whose outcome, when applied to the network model, maximally reduces the dynamics uncertainty pertinent to the intervention objective.

Optimal Quantification of Contact Inhibition in Cell Populations

Contact inhibition refers to a reduction in the rate of cell migration and/or cell proliferation in regions of high cell density. Under normal conditions, contact inhibition is associated with the proper functioning tissues, whereas abnormal regulation of contact inhibition is associated with pathological conditions, such as tumor spreading. Unfortunately, standard mathematical modeling practices mask the importance of parameters that control contact inhibition through scaling arguments. Furthermore, standard experimental protocols are insufficient to quantify the effects of contact inhibition because they focus on data describing early time, low-density dynamics only. Here we use the logistic growth equation as a caricature model of contact inhibition to make recommendations as to how to best mitigate these issues. Taking a Bayesian approach, we quantify the trade off between different features of experimental design and estimates of parameter uncertainty so that we can reformulate a standard cell proliferation assay to provide estimates of both the low-density intrinsic growth rate, λ, and the carrying capacity density, K, which is a measure of contact inhibition.

Genome-Scale Reconstruction of the Human Astrocyte Metabolic Network

Astrocytes are the most abundant cells of the central nervous system; they have a predominant role in maintaining brain metabolism. In this sense, abnormal metabolic states have been found in different neuropathological diseases. Determination of metabolic states of astrocytes is difficult to model using current experimental approaches given the high number of reactions and metabolites present. Thus, genome-scale metabolic networks derived from transcriptomic data can be used as a framework to elucidate how astrocytes modulate human brain metabolic states during normal conditions and in neurodegenerative diseases. We performed a Genome-Scale Reconstruction of the Human Astrocyte Metabolic Network with the purpose of elucidating a significant portion of the metabolic map of the astrocyte. This is the first global high-quality, manually curated metabolic reconstruction network of a human astrocyte. It includes 5,007 metabolites and 5,659 reactions distributed among 8 cell compartments, (extracellular, cytoplasm, mitochondria, endoplasmic reticle, Golgi apparatus, lysosome, peroxisome and nucleus). Using the reconstructed network, the metabolic capabilities of human astrocytes were calculated and compared both in normal and ischemic conditions. We identified reactions activated in these two states, which can be useful for understanding the astrocytic pathways that are affected during brain disease. Additionally, we also showed that the obtained flux distributions in the model, are in accordance with literature-based findings. Up to date, this is the most complete representation of the human astrocyte in terms of inclusion of genes, proteins, reactions and metabolic pathways, being a useful guide for in-silico analysis of several metabolic behaviors of the astrocyte during normal and pathologic states.

Prediction uncertainty and optimal experimental design for learning dynamical systems

Dynamical systems are frequently used to model biological systems. When these models are fit to data, it is necessary to ascertain the uncertainty in the model fit. Here, we present prediction deviation, a metric of uncertainty that determines the extent to which observed data have constrained the model's predictions. This is accomplished by solving an optimization problem that searches for a pair of models that each provides a good fit for the observed data, yet has maximally different predictions. We develop a method for estimating a priori the impact that additional experiments would have on the prediction deviation, allowing the experimenter to design a set of experiments that would most reduce uncertainty. We use prediction deviation to assess uncertainty in a model of interferon-alpha inhibition of viral infection, and to select a sequence of experiments that reduces this uncertainty. Finally, we prove a theoretical result which shows that prediction deviation provides bounds on the trajectories of the underlying true model. These results show that prediction deviation is a meaningful metric of uncertainty that can be used for optimal experimental design.

Requirements for multi-level systems pharmacology models to reach end-usage: the case of type 2 diabetes

We are currently in the middle of a major shift in biomedical research: unprecedented and rapidly growing amounts of data may be obtained today, from in vitro, in vivo and clinical studies, at molecular, physiological and clinical levels. To make use of these large-scale, multi-level datasets, corresponding multi-level mathematical models are needed, i.e. models that simultaneously capture multiple layers of the biological, physiological and disease-level organization (also referred to as quantitative systems pharmacology-QSP-models). However, today's multi-level models are not yet embedded in end-usage applications, neither in drug research and development nor in the clinic. Given the expectations and claims made historically, this seemingly slow adoption may seem surprising. Therefore, we herein consider a specific example-type 2 diabetes-and critically review the current status and identify key remaining steps for these models to become mainstream in the future. This overview reveals how, today, we may use models to ask scientific questions concerning, e.g., the cellular origin of insulin resistance, and how this translates to the whole-body level and short-term meal responses. However, before these multi-level models can become truly useful, they need to be linked with the capabilities of other important existing models, in order to make them 'personalized' (e.g. specific to certain patient phenotypes) and capable of describing long-term disease progression. To be useful in drug development, it is also critical that the developed models and their underlying data and assumptions are easily accessible. For clinical end-usage, in addition, model links to decision-support systems combined with the engagement of other disciplines are needed to create user-friendly and cost-efficient software packages.

Analyzing Th17 cell differentiation dynamics using a novel integrative modeling framework for time-course RNA sequencing data

Background

The differentiation of naive CD 4⁺ helper T (Th) cells into effector Th17 cells is steered by extracellular cytokines that activate and control the lineage specific transcriptional program. While the inducing cytokine signals and core transcription factors driving the differentiation towards Th17 lineage are well known, detailed mechanistic interactions between the key components are poorly understood.

Results

We develop an integrative modeling framework which combines RNA sequencing data with mathematical modeling and enables us to construct a mechanistic model for the core Th17 regulatory network in a data-driven manner.

Conclusions

Our results show significant evidence, for instance, for inhibitory mechanisms between the transcription factors and reveal a previously unknown dependency between the dosage of the inducing cytokine TGF β and the expression of the master regulator of competing (induced) regulatory T cell lineage. Further, our experimental validation approves this dependency in Th17 polarizing conditions.

Electronic supplementary material

The online version of this article (doi:10.1186/s12918-015-0223-6) contains supplementary material, which is available to authorized users.

Rational selection of experimental readout and intervention sites for reducing uncertainties in computational model predictions

Background

Understanding the dynamics of biological processes can substantially be supported by computational models in the form of nonlinear ordinary differential equations (ODE). Typically, this model class contains many unknown parameters, which are estimated from inadequate and noisy data. Depending on the ODE structure, predictions based on unmeasured states and associated parameters are highly uncertain, even undetermined. For given data, profile likelihood analysis has been proven to be one of the most practically relevant approaches for analyzing the identifiability of an ODE structure, and thus model predictions. In case of highly uncertain or non-identifiable parameters, rational experimental design based on various approaches has shown to significantly reduce parameter uncertainties with minimal amount of effort.

Results

In this work we illustrate how to use profile likelihood samples for quantifying the individual contribution of parameter uncertainty to prediction uncertainty. For the uncertainty quantification we introduce the profile likelihood sensitivity (PLS) index. Additionally, for the case of several uncertain parameters, we introduce the PLS entropy to quantify individual contributions to the overall prediction uncertainty. We show how to use these two criteria as an experimental design objective for selecting new, informative readouts in combination with intervention site identification. The characteristics of the proposed multi-criterion objective are illustrated with an in silico example. We further illustrate how an existing practically non-identifiable model for the chlorophyll fluorescence induction in a photosynthetic organism, D. salina, can be rendered identifiable by additional experiments with new readouts.

Conclusions

Having data and profile likelihood samples at hand, the here proposed uncertainty quantification based on prediction samples from the profile likelihood provides a simple way for determining individual contributions of parameter uncertainties to uncertainties in model predictions. The uncertainty quantification of specific model predictions allows identifying regions, where model predictions have to be considered with care. Such uncertain regions can be used for a rational experimental design to render initially highly uncertain model predictions into certainty. Finally, our uncertainty quantification directly accounts for parameter interdependencies and parameter sensitivities of the specific prediction.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-014-0436-5) contains supplementary material, which is available to authorized users.