An exact approach to calibrating infectious disease models to surveillance data: The case of HIV and HSV-2

Calibration of individual-based models to epidemiological data: A systematic review

Individual-based models (IBMs) informing public health policy should be calibrated to data and provide estimates of uncertainty. Two main components of model-calibration methods are the parameter-search strategy and the goodness-of-fit (GOF) measure; many options exist for each of these. This review provides an overview of calibration methods used in IBMs modelling infectious disease spread. We identified articles on PubMed employing simulation-based methods to calibrate IBMs informing public health policy in HIV, tuberculosis, and malaria epidemiology published between 1 January 2013 and 31 December 2018. Articles were included if models stored individual-specific information, and calibration involved comparing model output to population-level targets. We extracted information on parameter-search strategies, GOF measures, and model validation. The PubMed search identified 653 candidate articles, of which 84 met the review criteria. Of the included articles, 40 (48%) combined a quantitative GOF measure with an algorithmic parameter-search strategy–either an optimisation algorithm (14/40) or a sampling algorithm (26/40). These 40 articles varied widely in their choices of parameter-search strategies and GOF measures. For the remaining 44 (52%) articles, the parameter-search strategy could either not be identified (32/44) or was described as an informal, non-reproducible method (12/44). Of these 44 articles, the majority (25/44) were unclear about the GOF measure used; of the rest, only five quantitatively evaluated GOF. Only a minority of the included articles, 14 (17%) provided a rationale for their choice of model-calibration method. Model validation was reported in 31 (37%) articles. Reporting on calibration methods is far from optimal in epidemiological modelling studies of HIV, malaria and TB transmission dynamics. The adoption of better documented, algorithmic calibration methods could improve both reproducibility and the quality of inference in model-based epidemiology. There is a need for research comparing the performance of calibration methods to inform decisions about the parameter-search strategies and GOF measures.

Calibration—that is, “fitting” the model to data—is a crucial part of using mathematical models to better forecast and control the population-level spread of infectious diseases. Evidence that the mathematical model is well-calibrated improves confidence that the model provides a realistic picture of the consequences of health policy decisions. To make informed decisions, Policymakers need information about uncertainty: i.e., what is the range of likely outcomes (rather than just a single prediction). Thus, modellers should also strive to provide accurate measurements of uncertainty, both for their model parameters and for their predictions. This systematic review provides an overview of the methods used to calibrate individual-based models (IBMs) of the spread of HIV, malaria, and tuberculosis. We found that less than half of the reviewed articles used reproducible, non-subjective calibration methods. For the remaining articles, the method could either not be identified or was described as an informal, non-reproducible method. Only one-third of the articles obtained estimates of parameter uncertainty. We conclude that the adoption of better-documented, algorithmic calibration methods could improve both reproducibility and the quality of inference in model-based epidemiology.

Systematic selection between age and household structure for models aimed at emerging epidemic predictions

Numerous epidemic models have been developed to capture aspects of human contact patterns, making model selection challenging when they fit (often-scarce) early epidemic data equally well but differ in predictions. Here we consider the invasion of a novel directly transmissible infection and perform an extensive, systematic and transparent comparison of models with explicit age and/or household structure, to determine the accuracy loss in predictions in the absence of interventions when ignoring either or both social components. We conclude that, with heterogeneous and assortative contact patterns relevant to respiratory infections, the model’s age stratification is crucial for accurate predictions. Conversely, the household structure is only needed if transmission is highly concentrated in households, as suggested by an empirical but robust rule of thumb based on household secondary attack rate. This work serves as a template to guide the simplicity/accuracy trade-off in designing models aimed at initial, rapid assessment of potential epidemic severity.

Models of emerging epidemics can be exceedingly helpful in planning the response, but early on model selection is a difficult task. Here, the authors explore the joint contribution of age stratification and household structure on epidemic spread, and provides a rule of thumb to guide model choice.