Development of a critical appraisal tool for models predicting the impact of 'test, trace, and protect' programmes on COVID-19 transmission

OBJECTIVES

To develop a critical appraisal tool for non-computational-specialist public health professionals to assess the quality and relevance of modelling studies about Test and Trace (and Protect - TTP) programmes' impact on COVID-19 transmission.

STUDY DESIGN

Decision-making tool development.

METHODS

Using Tugwell et al.'s 1985 Health Care Effectiveness equation as a conceptual framework, combined with a purposive search of the relevant early modeling literature, we developed six critical appraisal questions for the rapid assessment of modeling studies related to the evaluation of TTP programmes' effectiveness.

RESULTS

By applying the critical appraisal tool to selected recent COVID-19 modeling studies, we demonstrate how models can be evaluated using the six questions to evaluate internal and external validity and relevance.

CONCLUSIONS

These six critical appraisal questions are able to discriminate between modeling studies of higher and lower quality and relevance to evaluating TTP programmes' impact. However, these questions require independent validation in a larger and systematic sample of relevant modeling studies which have appeared in later stages of the pandemic.

Endemic foot and mouth disease: pastoral in-herd disease dynamics in sub-Saharan Africa

Foot and mouth disease (FMD) burden disproportionally affects Africa where it is considered endemic. Smallholder livestock keepers experience significant losses due to disease, but the dynamics and mechanisms underlying persistence at the herd-level and beyond remain poorly understood. We address this knowledge gap using stochastic, compartmental modelling to explore FMD virus (FMDV) persistence, outbreak dynamics and disease burden in individual cattle herds within an endemic setting. Our analysis suggests repeated introduction of virus from outside the herd is required for long-term viral persistence, irrespective of carrier presence. Risk of new disease exposures resulting in significant secondary outbreaks is reduced by the presence of immune individuals giving rise to a period of reduced risk, the predicted duration of which suggests that multiple strains of FMDV are responsible for observed yearly herd-level outbreaks. Our analysis suggests management of population turnover could potentially reduce disease burden and deliberate infection of cattle, practiced by local livestock keepers in parts of Africa, has little effect on the duration of the reduced risk period but increases disease burden. This work suggests that FMD control should be implemented beyond individual herds but, in the interim, herd management may be used to reduced FMD impact to livestock keepers.

Posterior-based proposals for speeding up Markov chain Monte Carlo

Markov chain Monte Carlo (MCMC) is widely used for Bayesian inference in models of complex systems. Performance, however, is often unsatisfactory in models with many latent variables due to so-called poor mixing, necessitating the development of application-specific implementations. This paper introduces 'posterior-based proposals' (PBPs), a new type of MCMC update applicable to a huge class of statistical models (whose conditional dependence structures are represented by directed acyclic graphs). PBPs generate large joint updates in parameter and latent variable space, while retaining good acceptance rates (typically 33%). Evaluation against other approaches (from standard Gibbs/random walk updates to state-of-the-art Hamiltonian and particle MCMC methods) was carried out for widely varying model types: an individual-based model for disease diagnostic test data, a financial stochastic volatility model, a mixed model used in statistical genetics and a population model used in ecology. While different methods worked better or worse in different scenarios, PBPs were found to be either near to the fastest or significantly faster than the next best approach (by up to a factor of 10). PBPs, therefore, represent an additional general purpose technique that can be usefully applied in a wide variety of contexts.

Bayesian model evidence as a practical alternative to deviance information criterion

While model evidence is considered by Bayesian statisticians as a gold standard for model selection (the ratio in model evidence between two models giving the Bayes factor), its calculation is often viewed as too computationally demanding for many applications. By contrast, the widely used deviance information criterion (DIC), a different measure that balances model accuracy against complexity, is commonly considered a much faster alternative. However, recent advances in computational tools for efficient multi-temperature Markov chain Monte Carlo algorithms, such as steppingstone sampling (SS) and thermodynamic integration schemes, enable efficient calculation of the Bayesian model evidence. This paper compares both the capability (i.e. ability to select the true model) and speed (i.e. CPU time to achieve a given accuracy) of DIC with model evidence calculated using SS. Three important model classes are considered: linear regression models, mixed models and compartmental models widely used in epidemiology. While DIC was found to correctly identify the true model when applied to linear regression models, it led to incorrect model choice in the other two cases. On the other hand, model evidence led to correct model choice in all cases considered. Importantly, and perhaps surprisingly, DIC and model evidence were found to run at similar computational speeds, a result reinforced by analytically derived expressions.

Using model-based proposals for fast parameter inference on discrete state space, continuous-time Markov processes

Bayesian statistics provides a framework for the integration of dynamic models with incomplete data to enable inference of model parameters and unobserved aspects of the system under study. An important class of dynamic models is discrete state space, continuous-time Markov processes (DCTMPs). Simulated via the Doob-Gillespie algorithm, these have been used to model systems ranging from chemistry to ecology to epidemiology. A new type of proposal, termed 'model-based proposal' (MBP), is developed for the efficient implementation of Bayesian inference in DCTMPs using Markov chain Monte Carlo (MCMC). This new method, which in principle can be applied to any DCTMP, is compared (using simple epidemiological SIS and SIR models as easy to follow exemplars) to a standard MCMC approach and a recently proposed particle MCMC (PMCMC) technique. When measurements are made on a single-state variable (e.g. the number of infected individuals in a population during an epidemic), model-based proposal MCMC (MBP-MCMC) is marginally faster than PMCMC (by a factor of 2-8 for the tests performed), and significantly faster than the standard MCMC scheme (by a factor of 400 at least). However, when model complexity increases and measurements are made on more than one state variable (e.g. simultaneously on the number of infected individuals in spatially separated subpopulations), MBP-MCMC is significantly faster than PMCMC (more than 100-fold for just four subpopulations) and this difference becomes increasingly large.

Stochastic modelling of environmental variation for biological populations

We examine stochastic effects, in particular environmental variability, in population models of biological systems. Some simple models of environmental stochasticity are suggested, and we demonstrate a number of analytic approximations and simulation-based approaches that can usefully be applied to them. Initially, these techniques, including moment-closure approximations and local linearization, are explored in the context of a simple and relatively tractable process. Our presentation seeks to introduce these techniques to a broad-based audience of applied modellers. Therefore, as a test case, we study a natural stochastic formulation of a non-linear deterministic model for nematode infections in ruminants, proposed by Roberts and Grenfell (1991). This system is particularly suitable for our purposes, since it captures the essence of more complicated formulations of parasite demography and herd immunity found in the literature. We explore two modes of behaviour. In the endemic regime the stochastic dynamic fluctuates widely around the non-zero fixed points of the deterministic model. Enhancement of these fluctuations in the presence of environmental stochasticity can lead to extinction events. Using a simple model of environmental fluctuations we show that the magnitude of this system response reflects not only the variance of environmental noise, but also its autocorrelation structure. In the managed regime host-replacement is modelled via periodic perturbation of the population variables. In the absence of environmental variation stochastic effects are negligible, and we examine the system response to a realistic environmental perturbation based on the effect of micro-climatic fluctuations on the contact rate. The resultant stochastic effects and the relevance of analytic approximations based on simple models of environmental stochasticity are discussed.

Stochastic effects in a model of nematode infection in ruminants

We illustrate the importance of stochastic effects in population models of biological systems and demonstrate a number of analytic and simulation-based approaches that can usefully be applied to such models. In so doing, we compare the stochastic approach to the more usual deterministic one. The model studied represents the gastrointestinal infection of ruminants by nematodes when the hosts maintain a fixed density. The incorporation of a feedback mechanism, which accounts for the immune response of the infected animals, results in a highly nonlinear model; similar forms of nonlinearity are a feature of many plausible models in population biology. In the absence of an analytic solution to the full stochastic model we explore a number of approximations and compare them to simulations of the full stochastic process. We explore three modes of behaviour of the system. In the endemic regime the stochastic system fluctuates widely around the non-zero fixed points of the deterministic model. In the managed regime, where the system is subject to external periodic perturbation, stochastic effects are negligible. Finally, we find that in a regime in which the deterministic model predicts the long-term persistence of oscillations the stochastic model shows that extinction can occur. Of the approximation procedures we consider, the Normal approximation to the full stochastic process is the most generally applicable, and it is also the most accurate in the light of simulation results. Local linearization provides reasonably accurate prediction of the variance-covariance structure, and a transfer function approach allows calculation of the time-lagged auto- and cross-correlations in the endemic regime. Linearization of the stochastic updates themselves results in poor prediction of the population variances.

Prevalence of islet cell antibodies, insulin antibodies and hyperglycaemia in 2291 schoolchildren

Islet cell antibodies, insulin antibodies and hyperglycaemia were investigated in 2291 healthy schoolchildren. Eight of the subjects had islet cell antibodies and eight had insulin antibodies. However, no schoolchild who was positive for islet cell antibodies also had insulin antibodies present. Hyperglycaemia was observed in five children but neither islet cell antibodies nor insulin antibodies could be detected in the sera from these particular subjects.

The detection of abscesses with diffusible tracers

In six patients with suspected infection, scintigraphy with diffusible tracers outlined the margins and central portion of an abscess. An animal model was developed to study this process quantitatively. Small inflammatory lesions yielding volumes of pus of 0.1-0.2 mL were shown to have increased blood volume of 0.7 +/- 0.5 mL, increased blood flow by a factor of three and increased extracellular fluid volume of 10 +/- 6 mL. This supports the patient data and indicates that the pathophysiologic features characterising the clinical studies are hyperemia and oedema associated with an abscess. The slow, central area of tracer diffusion corresponded to the presence of pus. Scintigraphy with 99mTc-DTPA is a convenient way of detecting suspected inflammatory lesions and localising collections of pus.

The oxygen consumption of some tissues from hypophysectomized goldfish, Carassius auratus L

Hypophysectomy has no effect on the O2 consumptios reduction in liver O2 consumption is attributed to the increased glycogen content that follows hypophysectomy which has the effect of increasing the nonmetabolizing dry weight component of the cells.