The impact of uncertainty on predictions of the CovidSim epidemiological code

Trajectory-based global sensitivity analysis in multiscale models

This research introduces a novel global sensitivity analysis (GSA) framework for agent-based models (ABMs) that explicitly handles their distinctive features, such as multi-level structure and temporal dynamics. The framework uses Grassmannian diffusion maps to reduce output data dimensionality and sparse polynomial chaos expansion (PCE) to compute sensitivity indices for stochastic input parameters. To demonstrate the versatility of the proposed GSA method, we applied it to a non-linear system dynamics model and epidemiological and economic ABMs, depicting different dynamics. Unlike traditional GSA approaches, the proposed method enables a more general estimation of parametric sensitivities spanning from the micro level (individual agents) to the macro level (entire population). The new framework encourages the use of manifold-based techniques in uncertainty quantification, enhances understanding of complex spatio-temporal processes, and equips ABM practitioners with robust tools for detailed model analysis. This empowers them to make more informed decisions when developing, fine-tuning, and verifying models, thereby advancing the field and improving routine practice for GSA in ABMs.

Uncertainty quantification of the lattice Boltzmann method focussing on studies of human-scale vascular blood flow

Uncertainty quantification is becoming a key tool to ensure that numerical models can be sufficiently trusted to be used in domains such as medical device design. Demonstration of how input parameters impact the quantities of interest generated by any numerical model is essential to understanding the limits of its reliability. With the lattice Boltzmann method now a widely used approach for computational fluid dynamics, building greater understanding of its numerical uncertainty characteristics will support its further use in science and industry. In this study we apply an in-depth uncertainty quantification study of the lattice Boltzmann method in a canonical bifurcating geometry that is representative of the vascular junctions present in arterial and venous domains. These campaigns examine how quantities of interest-pressure and velocity along the central axes of the bifurcation-are influenced by the algorithmic parameters of the lattice Boltzmann method and the parameters controlling the values imposed at inlet velocity and outlet pressure boundary conditions. We also conduct a similar campaign on a set of personalised vessels to further illustrate the application of these techniques. Our work provides insights into how input parameters and boundary conditions impact the velocity and pressure distributions calculated in a simulation and can guide the choices of such values when applied to vascular studies of patient specific geometries. We observe that, from an algorithmic perspective, the number of time steps and the size of the grid spacing are the most influential parameters. When considering the influence of boundary conditions, we note that the magnitude of the inlet velocity and the mean pressure applied within sinusoidal pressure outlets have the greatest impact on output quantities of interest. We also observe that, when comparing the magnitude of variation imposed in the input parameters with that observed in the output quantities, this variability is particularly magnified when the input velocity is altered. This study also demonstrates how open-source toolkits for validation, verification and uncertainty quantification can be applied to numerical models deployed on high-performance computers without the need for modifying the simulation code itself. Such an ability is key to the more widespread adoption of the analysis of uncertainty in numerical models by significantly reducing the complexity of their execution and analysis.

Recommendations for quantitative uncertainty consideration in ecology and evolution

Ecological and evolutionary studies are currently failing to achieve complete and consistent reporting of model-related uncertainty. We identify three key barriers - a focus on parameter-related uncertainty, obscure uncertainty metrics, and limited recognition of uncertainty propagation - which have led to gaps in uncertainty consideration. However, these gaps can be closed. We propose that uncertainty reporting in ecology and evolution can be improved through wider application of existing statistical solutions and by adopting good practice from other scientific fields. Our recommendations include greater consideration of input data and model structure uncertainties, field-specific uncertainty standards for methods and reporting, and increased uncertainty propagation through the use of hierarchical models.

Uncertainty quantification of the impact of peripheral arterial disease on abdominal aortic aneurysms in blood flow simulations

Peripheral arterial disease (PAD) and abdominal aortic aneurysms (AAAs) often coexist and pose significant risks of mortality, yet their mutual interactions remain largely unexplored. Here, we introduce a fluid mechanics model designed to simulate the haemodynamic impact of PAD on AAA-associated risk factors. Our focus lies on quantifying the uncertainty inherent in controlling the flow rates within PAD-affected vessels and predicting AAA risk factors derived from wall shear stress. We perform a sensitivity analysis on nine critical model parameters through simulations of three-dimensional blood flow within a comprehensive arterial geometry. Our results show effective control of the flow rates using two-element Windkessel models, although specific outlets need attention. Quantities of interest like endothelial cell activation potential (ECAP) and relative residence time are instructive for identifying high-risk regions, with ECAP showing greater reliability and adaptability. Our analysis reveals that the uncertainty in the quantities of interest is 187% of that of the input parameters. Notably, parameters governing the amplitude and frequency of the inlet velocity exert the strongest influence on the risk factors' variability and warrant precise determination. This study forms the foundation for patient-specific simulations involving PAD and AAAs which should ultimately improve patient outcomes and reduce associated mortality rates.

An epidemiological modeling framework to inform institutional-level response to infectious disease outbreaks: a Covid-19 case study

Institutions have an enhanced ability to implement tailored mitigation measures during infectious disease outbreaks. However, macro-level predictive models are inefficient for guiding institutional decision-making due to uncertainty in local-level model input parameters. We present an institutional-level modeling toolkit used to inform prediction, resource procurement and allocation, and policy implementation at Clemson University throughout the Covid-19 pandemic. Through incorporating real-time estimation of disease surveillance and epidemiological measures based on institutional data, we argue this approach helps minimize uncertainties in input parameters presented in the broader literature and increases prediction accuracy. We demonstrate this through case studies at Clemson and other university settings during the Omicron BA.1 and BA.4/BA.5 variant surges. The input parameters of our toolkit are easily adaptable to other institutional settings during future health emergencies. This methodological approach has potential to improve public health response through increasing the capability of institutions to make data-informed decisions that better prioritize the health and safety of their communities while minimizing operational disruptions.

Introduction to Computational Biomedicine

The domain of computational biomedicine is a new and burgeoning one. Its areas of concern cover all scales of human biology, physiology, and pathology, commonly referred to as medicine, from the genomic to the whole human and beyond, including epidemiology and population health. Computational biomedicine aims to provide high-fidelity descriptions and predictions of the behavior of biomedical systems of both fundamental scientific and clinical importance. Digital twins and virtual humans aim to reproduce the extremely accurate duplicate of real-world human beings in cyberspace, which can be used to make highly accurate predictions that take complicated conditions into account. When that can be done reliably enough for the predictions to be actionable, such an approach will make an impact in the pharmaceutical industry by reducing or even replacing the extremely laboratory-intensive preclinical process of making and testing compounds in laboratories, and in clinical applications by assisting clinicians to make diagnostic and treatment decisions.

Bayesian monitoring of COVID-19 in Sweden

In an effort to provide regional decision support for the public healthcare, we design a data-driven compartment-based model of COVID-19 in Sweden. From national hospital statistics we derive parameter priors, and we develop linear filtering techniques to drive the simulations given data in the form of daily healthcare demands. We additionally propose a posterior marginal estimator which provides for an improved temporal resolution of the reproduction number estimate as well as supports robustness checks via a parametric bootstrap procedure. From our computational approach we obtain a Bayesian model of predictive value which provides important insight into the progression of the disease, including estimates of the effective reproduction number, the infection fatality rate, and the regional-level immunity. We successfully validate our posterior model against several different sources, including outputs from extensive screening programs. Since our required data in comparison is easy and non-sensitive to collect, we argue that our approach is particularly promising as a tool to support monitoring and decisions within public health. Significance: Using public data from Swedish patient registries we develop a national-scale computational model of COVID-19. The parametrized model produces valuable weekly predictions of healthcare demands at the regional level and validates well against several different sources. We also obtain critical epidemiological insights into the disease progression, including, e.g., reproduction number, immunity and disease fatality estimates. The success of the model hinges on our novel use of filtering techniques which allows us to design an accurate data-driven procedure using data exclusively from healthcare demands, i.e., our approach does not rely on public testing and is therefore very cost-effective.

An epidemiological modeling framework to inform institutional-level response to infectious disease outbreaks: A Covid-19 case study

Institutions have an enhanced ability to implement tailored mitigation measures during infectious disease outbreaks. However, macro-level predictive models are inefficient for guiding institutional decision-making due to uncertainty in local-level model input parameters. We present an institutional-level modeling toolkit used to inform prediction, resource procurement and allocation, and policy implementation at Clemson University throughout the Covid-19 pandemic. Through incorporating real-time estimation of disease surveillance and epidemiological measures based on institutional data, we argue this approach helps minimize uncertainties in input parameters presented in the broader literature and increases prediction accuracy. We demonstrate this through case studies at Clemson and other university settings during the Omicron BA.1 and BA.4/BA.5 variant surges. The input parameters of our toolkit are easily adaptable to other institutional settings during future health emergencies. This methodological approach has potential to improve public health response through increasing the capability of institutions to make data-informed decisions that better prioritize the health and safety of their communities while minimizing operational disruptions.

Dynamic calibration with approximate Bayesian computation for a microsimulation of disease spread

The global COVID-19 pandemic brought considerable public and policy attention to the field of infectious disease modelling. A major hurdle that modellers must overcome, particularly when models are used to develop policy, is quantifying the uncertainty in a model's predictions. By including the most recent available data in a model, the quality of its predictions can be improved and uncertainties reduced. This paper adapts an existing, large-scale, individual-based COVID-19 model to explore the benefits of updating the model in pseudo-real time. We use Approximate Bayesian Computation (ABC) to dynamically recalibrate the model's parameter values as new data emerge. ABC offers advantages over alternative calibration methods by providing information about the uncertainty associated with particular parameter values and the resulting COVID-19 predictions through posterior distributions. Analysing such distributions is crucial in fully understanding a model and its outputs. We find that forecasts of future disease infection rates are improved substantially by incorporating up-to-date observations and that the uncertainty in forecasts drops considerably in later simulation windows (as the model is provided with additional data). This is an important outcome because the uncertainty in model predictions is often overlooked when models are used in policy.

Inaccuracies of deterministic finite-element models of human middle ear revealed by stochastic modelling

For over 40 years, finite-element models of the mechanics of the middle ear have been mostly deterministic in nature. Deterministic models do not take into account the effects of inter-individual variabilities on middle-ear parameters. We present a stochastic finite-element model of the human middle ear that uses variability in the model parameters to investigate the uncertainty in the model outputs (umbo, stapes, and tympanic-membrane displacements). We demonstrate: (1) uncertainties in the model parameters can be magnified by more than three times in the umbo and stapes footplate responses at frequencies above 2 kHz; (2) middle-ear models are biased and they distort the output distributions; and (3) with increased frequency, the highly-uncertain regions spatially spread out on the tympanic membrane surface. Our results assert that we should be mindful when using deterministic finite-element middle-ear models for critical tasks such as novel device developments and diagnosis.

Complex challenges should be approached by a multitude of theories and models

The ongoing pandemic may be regarded as a wicked problem. Therefore, it should be analyzed by a multitude of theories and models. Approaching the complex set of challenges posed to individuals and society by singular methods, can lead to suboptimal decisions. Good decisions must take into account the large set of uncertainties we are facing, by using well established procedures, as for example health technology assessment (HTA) and a nuanced ethical framework.

Insights into the quantification and reporting of model-related uncertainty across different disciplines

Quantifying uncertainty associated with our models is the only way we can express how much we know about any phenomenon. Incomplete consideration of model-based uncertainties can lead to overstated conclusions with real-world impacts in diverse spheres, including conservation, epidemiology, climate science, and policy. Despite these potentially damaging consequences, we still know little about how different fields quantify and report uncertainty. We introduce the "sources of uncertainty" framework, using it to conduct a systematic audit of model-related uncertainty quantification from seven scientific fields, spanning the biological, physical, and political sciences. Our interdisciplinary audit shows no field fully considers all possible sources of uncertainty, but each has its own best practices alongside shared outstanding challenges. We make ten easy-to-implement recommendations to improve the consistency, completeness, and clarity of reporting on model-related uncertainty. These recommendations serve as a guide to best practices across scientific fields and expand our toolbox for high-quality research.

The Royal Society RAMP modelling initiative

Normally, science proceeds following a well-established set of principles. Studies are done with an emphasis on correctness, are submitted to a journal editor who evaluates their relevance, and then undergo anonymous peer review by experts before publication in a journal and acceptance by the scientific community via the open literature. This process is slow, but its accuracy has served all fields of science well. In an emergency situation, different priorities come to the fore. Research and review need to be conducted quickly, and the target audience consists of policymakers. Scientists must jostle for the attention of non-specialists without sacrificing rigour, and must deal not only with peer assessment but also with media scrutiny by journalists who may have agendas other than ensuring scientific correctness. Here, we describe how the Royal Society coordinated efforts of diverse scientists to help model the coronavirus epidemic. This article is part of the theme issue 'Technical challenges of modelling real-life epidemics and examples of overcoming these'.

Understanding and managing uncertainty and variability for wastewater monitoring beyond the pandemic: Lessons learned from the United Kingdom national COVID-19 surveillance programmes

The COVID-19 pandemic has put unprecedented pressure on public health resources around the world. From adversity, opportunities have arisen to measure the state and dynamics of human disease at a scale not seen before. In the United Kingdom, the evidence that wastewater could be used to monitor the SARS-CoV-2 virus prompted the development of National wastewater surveillance programmes. The scale and pace of this work has proven to be unique in monitoring of virus dynamics at a national level, demonstrating the importance of wastewater-based epidemiology (WBE) for public health protection. Beyond COVID-19, it can provide additional value for monitoring and informing on a range of biological and chemical markers of human health. A discussion of measurement uncertainty associated with surveillance of wastewater, focusing on lessons-learned from the UK programmes monitoring COVID-19 is presented, showing that sources of uncertainty impacting measurement quality and interpretation of data for public health decision-making, are varied and complex. While some factors remain poorly understood, we present approaches taken by the UK programmes to manage and mitigate the more tractable sources of uncertainty. This work provides a platform to integrate uncertainty management into WBE activities as part of global One Health initiatives beyond the pandemic.

When Do We Need Massive Computations to Perform Detailed COVID-19 Simulations?

The COVID-19 pandemic has infected over 250 million people worldwide and killed more than 5 million as of November 2021. Many intervention strategies are utilized (e.g., masks, social distancing, vaccinations), but officials making decisions have a limited time to act. Computer simulations can aid them by predicting future disease outcomes, but they also require significant processing power or time. It is examined whether a machine learning model can be trained on a small subset of simulation runs to inexpensively predict future disease trajectories resembling the original simulation results. Using four previously published agent-based models (ABMs) for COVID-19, a decision tree regression for each ABM is built and its predictions are compared to the corresponding ABM. Accurate machine learning meta-models are generated from ABMs without strong interventions (e.g., vaccines, lockdowns) using small amounts of simulation data: the root-mean-square error (RMSE) with 25% of the data is close to the RMSE for the full dataset (0.15 vs 0.14 in one model; 0.07 vs 0.06 in another). However, meta-models for ABMs employing strong interventions require much more training data (at least 60%) to achieve a similar accuracy. In conclusion, machine learning meta-models can be used in some scenarios to assist in faster decision-making.

Purposes and duties in scientific modelling

More people than ever are paying attention to philosophical questions about epidemiological models, including their susceptibility to the influence of social and ethical values, sufficiency to inform policy decisions under certain conditions, and even their fundamental nature. One important question pertains to the purposes of epidemiological models, for example, are COVID-19 models for 'prediction' or 'projection'? Are they adequate for making causal inferences? Is one of their goals, or virtues, to change individual responses to the pandemic? In this essay, we offer our perspective on these questions and place them in the context of other recent philosophical arguments about epidemiological models. We argue that clarifying the intended purpose of a model, and assessing its adequacy for that purpose, are moral-epistemic duties, responsibilities which pertain to knowledge but have moral significance nonetheless. This moral significance, we argue, stems from the inherent value-ladenness of models, along with the potential for models to be used in political decision making in ways that conflict with liberal values and which could lead to downstream harms. Increasing conversation about the moral significance of modelling, we argue, could help us to resist further eroding our standards of democratic scrutiny in the COVID-19 era.

Real-Time Infectious Disease Modeling to Inform Emergency Public Health Decision Making

Infectious disease transmission is a nonlinear process with complex, sometimes unintuitive dynamics. Modeling can transform information about a disease process and its parameters into quantitative projections that help decision makers compare public health response options. However, modelers face methodologic challenges, data challenges, and communication challenges, which are exacerbated under the time constraints of a public health emergency. We review methods, applications, challenges and opportunities for real-time infectious disease modeling during public health emergencies, with examples drawn from the two deadliest pandemics in recent history: HIV/AIDS and coronavirus disease 2019 (COVID-19).

Expected final online publication date for the Annual Review of Public Health, Volume 43 is April 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Identifiability and predictability of integer- and fractional-order epidemiological models using physics-informed neural networks

We analyze a plurality of epidemiological models through the lens of physics-informed neural networks (PINNs) that enable us to identify time-dependent parameters and data-driven fractional differential operators. In particular, we consider several variations of the classical susceptible-infectious-removed (SIR) model by introducing more compartments and fractional-order and time-delay models. We report the results for the spread of COVID-19 in New York City, Rhode Island and Michigan states and Italy, by simultaneously inferring the unknown parameters and the unobserved dynamics. For integer-order and time-delay models, we fit the available data by identifying time-dependent parameters, which are represented by neural networks. In contrast, for fractional differential models, we fit the data by determining different time-dependent derivative orders for each compartment, which we represent by neural networks. We investigate the structural and practical identifiability of these unknown functions for different datasets, and quantify the uncertainty associated with neural networks and with control measures in forecasting the pandemic.

Ensembles Are Required to Handle Aleatoric and Parametric Uncertainty in Molecular Dynamics Simulation

Classical molecular dynamics is a computer simulation technique that is in widespread use across many areas of science, from physics and chemistry to materials, biology, and medicine. The method continues to attract criticism due its oft-reported lack of reproducibility which is in part due to a failure to submit it to reliable uncertainty quantification (UQ). Here we show that the uncertainty arises from a combination of (i) the input parameters and (ii) the intrinsic stochasticity of the method controlled by the random seeds. To illustrate the situation, we make a systematic UQ analysis of a widely used molecular dynamics code (NAMD), applied to estimate binding free energy of a ligand-bound to a protein. In particular, we replace the usually fixed input parameters with random variables, systematically distributed about their mean values, and study the resulting distribution of the simulation output. We also perform a sensitivity analysis, which reveals that, out of a total of 175 parameters, just six dominate the variance in the code output. Furthermore, we show that binding energy calculations dampen the input uncertainty, in the sense that the variation around the mean output free energy is less than the variation around the mean of the assumed input distributions, if the output is ensemble-averaged over the random seeds. Without such ensemble averaging, the predicted free energy is five times more uncertain. The distribution of the predicted properties is thus strongly dependent upon the random seed. Owing to this substantial uncertainty, robust statistical measures of uncertainty in molecular dynamics simulation require the use of ensembles in all contexts.

A forecast for large-scale, predictive biology: Lessons from meteorology

Quantitative systems biology, in which predictive mathematical models are constructed to guide the design of experiments and predict experimental outcomes, is at an exciting transition point, where the foundational scientific principles are becoming established, but the impact is not yet global. The next steps necessary for mathematical modeling to transform biological research and applications, in the same way it has already transformed other fields, is not completely clear. The purpose of this perspective is to forecast possible answers to this question-what needs to happen next-by drawing on the experience gained in another field, specifically meteorology. We review here a number of lessons learned in weather prediction that are directly relevant to biological systems modeling, and that we believe can enable the same kinds of global impact in our field as atmospheric modeling makes today.