Benchmarking tools for a priori identifiability analysis

Employing Observability Rank Conditions for Taking into Account Experimental Information a priori

The concept of identifiability describes the possibility of inferring the parameters of a dynamic model by observing its output. It is common and useful to distinguish between structural and practical identifiability. The former property is fully determined by the model equations, while the latter is also influenced by the characteristics of the available experimental data. Structural identifiability can be determined by means of symbolic computations, which may be performed before collecting experimental data, and are hence sometimes called a priori analyses. Practical identifiability is typically assessed numerically, with methods that require simulations-and often also optimization-and are applied a posteriori. An approach to study structural local identifiability is to consider it as a particular case of observability, which is the possibility of inferring the internal state of a system from its output. Thus, both properties can be analysed jointly, by building a generalized observability matrix and computing its rank. The aim of this paper is to investigate to which extent such observability-based methods can also inform about practical aspects related with the experimental setup, which are usually not approached in this way. To this end, we explore a number of possible extensions of the rank tests, and discuss the purposes for which they can be informative as well as others for which they cannot.

A Systematic Computational Framework for Practical Identifiability Analysis in Mathematical Models Arising from Biology

Practical identifiability is a critical concern in data-driven modeling of mathematical systems. In this paper, we propose a novel framework for practical identifiability analysis to evaluate parameter identifiability in mathematical models of biological systems. Starting with a rigorous mathematical definition of practical identifiability, we demonstrate its equivalence to the invertibility of the Fisher Information Matrix. Our framework establishes the relationship between practical identifiability and coordinate identifiability, introducing a novel metric that simplifies and accelerates the evaluation of parameter identifiability compared to the profile likelihood method. Additionally, we introduce new regularization terms to address non-identifiable parameters, enabling uncertainty quantification and improving model reliability. To guide experimental design, we present an optimal data collection algorithm that ensures all model parameters are practically identifiable. Applications to Hill functions, neural networks, and dynamic biological models demonstrate the feasibility and efficiency of the proposed computational framework in uncovering critical biological processes and identifying key observable variables.

Global and local identifiability analysis of a nonlinear biphasic constitutive model in confined compression

Application of biomechanical models relies on model parameters estimated from experimental data. Parameter non-identifiability, when the same model output can be produced by many sets of parameter values, introduces severe errors yet has received relatively little attention in biomechanics and is subtle enough to remain unnoticed in the absence of deliberate verification. The present work develops a global identifiability analysis method in which cluster analysis and singular value decomposition are applied to vectors of parameter-output variable correlation coefficients. This method provides a visual representation of which specific experimental design elements are beneficial or harmful in terms of parameter identifiability, supporting the correction of deficiencies in the test protocol prior to testing physical specimens. The method was applied to a representative nonlinear biphasic model for cartilaginous tissue, demonstrating that confined compression data does not provide identifiability for the biphasic model parameters. This result was confirmed by two independent analyses: local analysis of the Hessian of a sum-of-squares error cost function and observation of the behaviour of two optimization algorithms. Therefore, confined compression data are insufficient for the calibration of general-purpose biphasic models. Identifiability analysis by these or other methods is strongly recommended when planning future experiments.

Identifiability investigation of within-host models of acute virus infection

Uncertainty in parameter estimates from fitting within-host models to empirical data limits the model's ability to uncover mechanisms of infection, disease progression, and to guide pharmaceutical interventions. Understanding the effect of model structure and data availability on model predictions is important for informing model development and experimental design. To address sources of uncertainty in parameter estimation, we used four mathematical models of influenza A infection with increased degrees of biological realism. We tested the ability of each model to reveal its parameters in the presence of unlimited data by performing structural identifiability analyses. We then refined the results by predicting practical identifiability of parameters under daily influenza A virus titers alone or together with daily adaptive immune cell data. Using these approaches, we presented insight into the sources of uncertainty in parameter estimation and provided guidelines for the types of model assumptions, optimal experimental design, and biological information needed for improved predictions.

Modeling the CD8＋ T cell immune response to influenza infection in adult and aged mice

The CD8＋ T cell response is the main determinant of viral clearance during influenza infection. However, influenza viral dynamics and the respective immune responses are affected by the host's age. To investigate age-related differences in the CD8＋ T cell immune response dynamics, we propose 16 ordinary differential equation models of existing experimental data. These data consist of viral titer and CD8＋ T cell counts collected periodically over a period of 19 days from adult and aged mice infected with influenza A/Puerto Rico/8/34 (H1N1). We use the corrected Akaike Information Criterion to identify the models which best represent the considered data. Our model selection process indicates differences in mechanisms which reduce the CD8＋ T cell response: linear downregulation is favored for adult mice, while baseline exponential decay is favored for aged mice. Parameter fitting of the top ranked models suggests that the aged population has reduced CD8＋ T cell proliferation compared to the adult population. More experimental work is needed to determine the specific immunological features through which age might cause these differences. A better understanding of the immunological mechanisms by which aging leads to discrepant CD8＋ T cell dynamics may inform future treatment strategies.

Mathematical analysis of simple behavioral epidemic models

COVID-19 highlighted the importance of considering human behavior change when modeling disease dynamics. This led to developing various models that incorporate human behavior. Our objective is to contribute to an in-depth, mathematical examination of such models. Here, we consider a simple deterministic compartmental model with endogenous incorporation of human behavior (i.e., behavioral feedback) through transmission in a classic Susceptible-Exposed-Infectious-Recovered (SEIR) structure. Despite its simplicity, the SEIR structure with behavior (SEIRb) was shown to perform well in forecasting, especially compared to more complicated models. We contrast this model with an SEIR model that excludes endogenous incorporation of behavior. Both models assume permanent immunity to COVID-19, so we also consider a modification of the models which include waning immunity (SEIRS and SEIRSb). We perform equilibria, sensitivity, and identifiability analyses on all models and examine the fidelity of the models to replicate COVID-19 data across the United States. Endogenous incorporation of behavior significantly improves a model's ability to produce realistic outbreaks. While the two endogenous models are similar with respect to identifiability and sensitivity, the SEIRSb model, with the more accurate assumption of the waning immunity, strengthens the initial SEIRb model by allowing for the existence of an endemic equilibrium, a realistic feature of COVID-19 dynamics. When fitting the model to data, we further consider the addition of simple seasonality affecting disease transmission to highlight the explanatory power of the models.

Identifiability investigation of within-host models of acute virus infection

Uncertainty in parameter estimates from fitting within-host models to empirical data limits the model's ability to uncover mechanisms of infection, disease progression, and to guide pharmaceutical interventions. Understanding the effect of model structure and data availability on model predictions is important for informing model development and experimental design. To address sources of uncertainty in parameter estimation, we use four mathematical models of influenza A infection with increased degrees of biological realism. We test the ability of each model to reveal its parameters in the presence of unlimited data by performing structural identifiability analyses. We then refine the results by predicting practical identifiability of parameters under daily influenza A virus titers alone or together with daily adaptive immune cell data. Using these approaches, we present insight into the sources of uncertainty in parameter estimation and provide guidelines for the types of model assumptions, optimal experimental design, and biological information needed for improved predictions.

Endothelial cells signaling and patterning under hypoxia: a mechanistic integrative computational model including the Notch-Dll4 pathway

Introduction: Several signaling pathways are activated during hypoxia to promote angiogenesis, leading to endothelial cell patterning, interaction, and downstream signaling. Understanding the mechanistic signaling differences between endothelial cells under normoxia and hypoxia and their response to different stimuli can guide therapies to modulate angiogenesis. We present a novel mechanistic model of interacting endothelial cells, including the main pathways involved in angiogenesis. Methods: We calibrate and fit the model parameters based on well-established modeling techniques that include structural and practical parameter identifiability, uncertainty quantification, and global sensitivity. Results: Our results indicate that the main pathways involved in patterning tip and stalk endothelial cells under hypoxia differ, and the time under hypoxia interferes with how different stimuli affect patterning. Additionally, our simulations indicate that Notch signaling might regulate vascular permeability and establish different Nitric Oxide release patterns for tip/stalk cells. Following simulations with various stimuli, our model suggests that factors such as time under hypoxia and oxygen availability must be considered for EC pattern control. Discussion: This project provides insights into the signaling and patterning of endothelial cells under various oxygen levels and stimulation by VEGFA and is our first integrative approach toward achieving EC control as a method for improving angiogenesis. Overall, our model provides a computational framework that can be built on to test angiogenesis-related therapies by modulation of different pathways, such as the Notch pathway.

Practical Identifiability of Plant Growth Models: A Unifying Framework and Its Specification for Three Local Indices

Amid the rise of machine learning models, a substantial portion of plant growth models remains mechanistic, seeking to capture an in-depth understanding of the underlying phenomena governing the system's dynamics. The development of these models typically involves parameter estimation from experimental data. Ensuring that the estimated parameters align closely with their respective "true" values is crucial since they hold biological interpretation, leading to the challenge of uniqueness in the solutions. Structural identifiability analysis addresses this issue under the assumption of perfect observations of system dynamics, whereas practical identifiability considers limited measurements and the accompanying noise. In the literature, definitions for structural identifiability vary only slightly among authors, whereas the concept and quantification of practical identifiability lack consensus, with several indices coexisting. In this work, we provide a unified framework for studying identifiability, accommodating different definitions that need to be instantiated depending on each application case. In a more applicative second step, we focus on three widely used methods for quantifying practical identifiability: collinearity indices, profile likelihood, and average relative error. We show the limitations of their local versions, and we propose a new risk index built on the profile likelihood-based confidence intervals. We illustrate the usefulness of these concepts for plant growth modeling using a discrete-time individual plant growth model, LNAS, and a continuous-time plant population epidemics model. Through this work, we aim to underline the significance of identifiability analysis as a complement to any parameter estimation study and offer guidance to the modeler.

Implementing measurement error models with mechanistic mathematical models in a likelihood-based framework for estimation, identifiability analysis and prediction in the life sciences

Throughout the life sciences, we routinely seek to interpret measurements and observations using parametrized mechanistic mathematical models. A fundamental and often overlooked choice in this approach involves relating the solution of a mathematical model with noisy and incomplete measurement data. This is often achieved by assuming that the data are noisy measurements of the solution of a deterministic mathematical model, and that measurement errors are additive and normally distributed. While this assumption of additive Gaussian noise is extremely common and simple to implement and interpret, it is often unjustified and can lead to poor parameter estimates and non-physical predictions. One way to overcome this challenge is to implement a different measurement error model. In this review, we demonstrate how to implement a range of measurement error models in a likelihood-based framework for estimation, identifiability analysis and prediction, called profile-wise analysis. This frequentist approach to uncertainty quantification for mechanistic models leverages the profile likelihood for targeting parameters and understanding their influence on predictions. Case studies, motivated by simple caricature models routinely used in systems biology and mathematical biology literature, illustrate how the same ideas apply to different types of mathematical models. Open-source Julia code to reproduce results is available on GitHub.

Distilling identifiable and interpretable dynamic models from biological data

Mechanistic dynamical models allow us to study the behavior of complex biological systems. They can provide an objective and quantitative understanding that would be difficult to achieve through other means. However, the systematic development of these models is a non-trivial exercise and an open problem in computational biology. Currently, many research efforts are focused on model discovery, i.e. automating the development of interpretable models from data. One of the main frameworks is sparse regression, where the sparse identification of nonlinear dynamics (SINDy) algorithm and its variants have enjoyed great success. SINDy-PI is an extension which allows the discovery of rational nonlinear terms, thus enabling the identification of kinetic functions common in biochemical networks, such as Michaelis-Menten. SINDy-PI also pays special attention to the recovery of parsimonious models (Occam's razor). Here we focus on biological models composed of sets of deterministic nonlinear ordinary differential equations. We present a methodology that, combined with SINDy-PI, allows the automatic discovery of structurally identifiable and observable models which are also mechanistically interpretable. The lack of structural identifiability and observability makes it impossible to uniquely infer parameter and state variables, which can compromise the usefulness of a model by distorting its mechanistic significance and hampering its ability to produce biological insights. We illustrate the performance of our method with six case studies. We find that, despite enforcing sparsity, SINDy-PI sometimes yields models that are unidentifiable. In these cases we show how our method transforms their equations in order to obtain a structurally identifiable and observable model which is also interpretable.

Structural identifiability of biomolecular controller motifs with and without flow measurements as model output

Controller motifs are simple biomolecular reaction networks with negative feedback. They can explain how regulatory function is achieved and are often used as building blocks in mathematical models of biological systems. In this paper we perform an extensive investigation into structural identifiability of controller motifs, specifically the so-called basic and antithetic controller motifs. Structural identifiability analysis is a useful tool in the creation and evaluation of mathematical models: it can be used to ensure that model parameters can be determined uniquely and to examine which measurements are necessary for this purpose. This is especially useful for biological models where parameter estimation can be difficult due to limited availability of measureable outputs. Our aim with this work is to investigate how structural identifiability is affected by controller motif complexity and choice of measurements. To increase the number of potential outputs we propose two methods for including flow measurements and show how this affects structural identifiability in combination with, or in the absence of, concentration measurements. In our investigation, we analyze 128 different controller motif structures using a combination of flow and/or concentration measurements, giving a total of 3648 instances. Among all instances, 34% of the measurement combinations provided structural identifiability. Our main findings for the controller motifs include: i) a single measurement is insufficient for structural identifiability, ii) measurements related to different chemical species are necessary for structural identifiability. Applying these findings result in a reduced subset of 1568 instances, where 80% are structurally identifiable, and more complex/interconnected motifs appear easier to structurally identify. The model structures we have investigated are commonly used in models of biological systems, and our results demonstrate how different model structures and measurement combinations affect structural identifiability of controller motifs.

Endothelial cells signaling and patterning under hypoxia: a mechanistic integrative computational model including the Notch-Dll4 pathway

Several signaling pathways are activated during hypoxia to promote angiogenesis, leading to endothelial cell patterning, interaction, and downstream signaling. Understanding the mechanistic signaling differences between normoxia and hypoxia can guide therapies to modulate angiogenesis. We present a novel mechanistic model of interacting endothelial cells, including the main pathways involved in angiogenesis. We calibrate and fit the model parameters based on well-established modeling techniques. Our results indicate that the main pathways involved in the patterning of tip and stalk endothelial cells under hypoxia differ, and the time under hypoxia affects how a reaction affects patterning. Interestingly, the interaction of receptors with Neuropilin1 is also relevant for cell patterning. Our simulations under different oxygen concentrations indicate time- and oxygen-availability-dependent responses for the two cells. Following simulations with various stimuli, our model suggests that factors such as period under hypoxia and oxygen availability must be considered for pattern control. This project provides insights into the signaling and patterning of endothelial cells under hypoxia, contributing to studies in the field.

Structural Identifiability and Observability of Microbial Community Models

Biological communities are populations of various species interacting in a common location. Microbial communities, which are formed by microorganisms, are ubiquitous in nature and are increasingly used in biotechnological and biomedical applications. They are nonlinear systems whose dynamics can be accurately described by models of ordinary differential equations (ODEs). A number of ODE models have been proposed to describe microbial communities. However, the structural identifiability and observability of most of them-that is, the theoretical possibility of inferring their parameters and internal states by observing their output-have not been determined yet. It is important to establish whether a model possesses these properties, because, in their absence, the ability of a model to make reliable predictions may be compromised. Hence, in this paper, we analyse these properties for the main families of microbial community models. We consider several dimensions and measurements; overall, we analyse more than a hundred different configurations. We find that some of them are fully identifiable and observable, but a number of cases are structurally unidentifiable and/or unobservable under typical experimental conditions. Our results help in deciding which modelling frameworks may be used for a given purpose in this emerging area, and which ones should be avoided.