Complex model calibration through emulation, a worked example for a stochastic epidemic model

Uncertainty quantification is a formal paradigm of statistical estimation that aims to account for all uncertainties inherent in the modelling process of real-world complex systems. The methods are directly applicable to stochastic models in epidemiology, however they have thus far not been widely used in this context. In this paper, we provide a tutorial on uncertainty quantification of stochastic epidemic models, aiming to facilitate the use of the uncertainty quantification paradigm for practitioners with other complex stochastic simulators of applied systems. We provide a formal workflow including the important decisions and considerations that need to be taken, and illustrate the methods over a simple stochastic epidemic model of UK SARS-CoV-2 transmission and patient outcome. We also present new approaches to visualisation of outputs from sensitivity analyses and uncertainty quantification more generally in high input and/or output dimensions.

Challenges on the interaction of models and policy for pandemic control

The COVID-19 pandemic has seen infectious disease modelling at the forefront of government decision-making. Models have been widely used throughout the pandemic to estimate pathogen spread and explore the potential impact of different intervention strategies. Infectious disease modellers and policymakers have worked effectively together, but there are many avenues for progress on this interface. In this paper, we identify and discuss seven broad challenges on the interaction of models and policy for pandemic control. We then conclude with suggestions and recommendations for the future.

The importance of uncertainty quantification in model reproducibility

Many computer models possess high-dimensional input spaces and substantial computational time to produce a single model evaluation. Although such models are often 'deterministic', these models suffer from a wide range of uncertainties. We argue that uncertainty quantification is crucial for computer model validation and reproducibility. We present a statistical framework, termed history matching, for performing global parameter search by comparing model output to the observed data. We employ Gaussian process (GP) emulators to produce fast predictions about model behaviour at the arbitrary input parameter settings allowing output uncertainty distributions to be calculated. History matching identifies sets of input parameters that give rise to acceptable matches between observed data and model output given our representation of uncertainties. Modellers could proceed by simulating computer models' outputs of interest at these identified parameter settings and producing a range of predictions. The variability in model results is crucial for inter-model comparison as well as model development. We illustrate the performance of emulation and history matching on a simple one-dimensional toy model and in application to a climate model. This article is part of the theme issue 'Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico'.

Key questions for modelling COVID-19 exit strategies

Combinations of intense non-pharmaceutical interventions (lockdowns) were introduced worldwide to reduce SARS-CoV-2 transmission. Many governments have begun to implement exit strategies that relax restrictions while attempting to control the risk of a surge in cases. Mathematical modelling has played a central role in guiding interventions, but the challenge of designing optimal exit strategies in the face of ongoing transmission is unprecedented. Here, we report discussions from the Isaac Newton Institute 'Models for an exit strategy' workshop (11-15 May 2020). A diverse community of modellers who are providing evidence to governments worldwide were asked to identify the main questions that, if answered, would allow for more accurate predictions of the effects of different exit strategies. Based on these questions, we propose a roadmap to facilitate the development of reliable models to guide exit strategies. This roadmap requires a global collaborative effort from the scientific community and policymakers, and has three parts: (i) improve estimation of key epidemiological parameters; (ii) understand sources of heterogeneity in populations; and (iii) focus on requirements for data collection, particularly in low-to-middle-income countries. This will provide important information for planning exit strategies that balance socio-economic benefits with public health.

Uncertainty and variability in computational and mathematical models of cardiac physiology

KEY POINTS

Mathematical and computational models of cardiac physiology have been an integral component of cardiac electrophysiology since its inception, and are collectively known as the Cardiac Physiome. We identify and classify the numerous sources of variability and uncertainty in model formulation, parameters and other inputs that arise from both natural variation in experimental data and lack of knowledge. The impact of uncertainty on the outputs of Cardiac Physiome models is not well understood, and this limits their utility as clinical tools. We argue that incorporating variability and uncertainty should be a high priority for the future of the Cardiac Physiome. We suggest investigating the adoption of approaches developed in other areas of science and engineering while recognising unique challenges for the Cardiac Physiome; it is likely that novel methods will be necessary that require engagement with the mathematics and statistics community.

ABSTRACT

The Cardiac Physiome effort is one of the most mature and successful applications of mathematical and computational modelling for describing and advancing the understanding of physiology. After five decades of development, physiological cardiac models are poised to realise the promise of translational research via clinical applications such as drug development and patient-specific approaches as well as ablation, cardiac resynchronisation and contractility modulation therapies. For models to be included as a vital component of the decision process in safety-critical applications, rigorous assessment of model credibility will be required. This White Paper describes one aspect of this process by identifying and classifying sources of variability and uncertainty in models as well as their implications for the application and development of cardiac models. We stress the need to understand and quantify the sources of variability and uncertainty in model inputs, and the impact of model structure and complexity and their consequences for predictive model outputs. We propose that the future of the Cardiac Physiome should include a probabilistic approach to quantify the relationship of variability and uncertainty of model inputs and outputs.

Risk analysis for autonomous underwater vehicle operations in extreme environments

Autonomous underwater vehicles (AUVs) are used increasingly to explore hazardous marine environments. Risk assessment for such complex systems is based on subjective judgment and expert knowledge as much as on hard statistics. Here, we describe the use of a risk management process tailored to AUV operations, the implementation of which requires the elicitation of expert judgment. We conducted a formal judgment elicitation process where eight world experts in AUV design and operation were asked to assign a probability of AUV loss given the emergence of each fault or incident from the vehicle's life history of 63 faults and incidents. After discussing methods of aggregation and analysis, we show how the aggregated risk estimates obtained from the expert judgments were used to create a risk model. To estimate AUV survival with mission distance, we adopted a statistical survival function based on the nonparametric Kaplan-Meier estimator. We present theoretical formulations for the estimator, its variance, and confidence limits. We also present a numerical example where the approach is applied to estimate the probability that the Autosub3 AUV would survive a set of missions under Pine Island Glacier, Antarctica in January-March 2009.

Towards a vulnerability assessment of the UK and northern European coasts: the role of regional climate variability

Within the framework of a Tyndall Centre research project, sea level and wave changes around the UK and in the North Sea have been analysed. This paper integrates the results of this project. Many aspects of the contribution of the North Atlantic Oscillation (NAO) to sea level and wave height have been resolved. The NAO is a major forcing parameter for sea-level variability. Strong positive response to increasing NAO was observed in the shallow parts of the North Sea, while slightly negative response was found in the southwest part of the UK. The cause of the strong positive response is mainly the increased westerly winds. The NAO increase during the last decades has affected both the mean sea level and the extreme sea levels in the North Sea. The derived spatial distribution of the NAO-related variability of sea level allows the development of scenarios for future sea level and wave height in the region. Because the response of sea level to the NAO is found to be variable in time across all frequency bands, there is some inherent uncertainty in the use of the empirical relationships to develop scenarios of future sea level. Nevertheless, as it remains uncertain whether the multi-decadal NAO variability is related to climate change, the use of the empirical relationships in developing scenarios is justified. The resulting scenarios demonstrate: (i) that the use of regional estimates of sea level increase the projected range of sea-level change by 50% and (ii) that the contribution of the NAO to winter sea-level variability increases the range of uncertainty by a further 10-20cm. On the assumption that the general circulation models have some skill in simulating the future NAO change, then the NAO contribution to sea-level change around the UK is expected to be very small (<4cm) by 2080. Wave heights are also sensitive to the NAO changes, especially in the western coasts of the UK. Under the same scenarios for future NAO changes, the projected significant wave-height changes in the northeast Atlantic will exceed 0.4m. In addition, wave-direction changes of around 20 degrees per unit NAO index have been documented for one location. Such changes raise the possibility of consequential alteration of coastal erosion.

Assimilating satellite ocean-colour observations into oceanic ecosystem models

The effectiveness of ocean-colour data assimilation in providing robust biological-parameter estimates for basin-scale ecosystem models is investigated for a phytoplankton-zooplankton-nutrient model using North Atlantic satellite chlorophyll data. The model is forced by annual cycles of mixed-layer depth, day length, photosynthetically available radiation and a temperature-dependent phytoplankton maximum growth rate. Although ocean-colour data are potentially limited in their ability to constrain model parameters because they provide information about the phytoplankton component only, this limitation is offset by the volume of data available covering the range of possible biogeochemical responses to similar and widely varying physical conditions. The results are improved by applying wintertime nutrient estimates based on in situ observations as an additional constraint. Repeatability of parameter estimates obtained from independent samples is examined. Results obtained using regional and basin-wide sampling strategies for obtaining the optimization dataset are compared and the geographic applicability of the calibrated models is assessed.