Calibration of individual-based models to epidemiological data: A systematic review

Calibration and Validation of the Colorectal Cancer and Adenoma Incidence and Mortality (CRC-AIM) Microsimulation Model Using Deep Neural Networks

OBJECTIVES

Machine learning (ML)-based emulators improve the calibration of decision-analytical models, but their performance in complex microsimulation models is yet to be determined.

METHODS

We demonstrated the use of an ML-based emulator with the Colorectal Cancer (CRC)-Adenoma Incidence and Mortality (CRC-AIM) model, which includes 23 unknown natural history input parameters to replicate the CRC epidemiology in the United States. We first generated 15,000 input combinations and ran the CRC-AIM model to evaluate CRC incidence, adenoma size distribution, and the percentage of small adenoma detected by colonoscopy. We then used this data set to train several ML algorithms, including deep neural network (DNN), random forest, and several gradient boosting variants (i.e., XGBoost, LightGBM, CatBoost) and compared their performance. We evaluated 10 million potential input combinations using the selected emulator and examined input combinations that best estimated observed calibration targets. Furthermore, we cross-validated outcomes generated by the CRC-AIM model with those made by CISNET models. The calibrated CRC-AIM model was externally validated using the United Kingdom Flexible Sigmoidoscopy Screening Trial (UKFSST).

RESULTS

The DNN with proper preprocessing outperformed other tested ML algorithms and successfully predicted all 8 outcomes for different input combinations. It took 473 s for the trained DNN to predict outcomes for 10 million inputs, which would have required 190 CPU-years without our DNN. The overall calibration process took 104 CPU-days, which included building the data set, training, selecting, and hyperparameter tuning of the ML algorithms. While 7 input combinations had acceptable fit to the targets, a combination that best fits all outcomes was selected as the best vector. Almost all of the predictions made by the best vector laid within those from the CISNET models, demonstrating CRC-AIM's cross-model validity. Similarly, CRC-AIM accurately predicted the hazard ratios of CRC incidence and mortality as reported by UKFSST, demonstrating its external validity. Examination of the impact of calibration targets suggested that the selection of the calibration target had a substantial impact on model outcomes in terms of life-year gains with screening.

CONCLUSIONS

Emulators such as a DNN that is meticulously selected and trained can substantially reduce the computational burden of calibrating complex microsimulation models.

HIGHLIGHTS

Calibrating a microsimulation model, a process to find unobservable parameters so that the model fits observed data, is computationally complex.We used a deep neural network model, a popular machine learning algorithm, to calibrate the Colorectal Cancer Adenoma Incidence and Mortality (CRC-AIM) model.We demonstrated that our approach provides an efficient and accurate method to significantly speed up calibration in microsimulation models.The calibration process successfully provided cross-model validation of CRC-AIM against 3 established CISNET models and also externally validated against a randomized controlled trial.

Guidance on the use of complex systems models for economic evaluations of public health interventions

To help health economic modelers respond to demands for greater use of complex systems models in public health. To propose identifiable features of such models and support researchers to plan public health modeling projects using these models. A working group of experts in complex systems modeling and economic evaluation was brought together to develop and jointly write guidance for the use of complex systems models for health economic analysis. The content of workshops was informed by a scoping review. A public health complex systems model for economic evaluation is defined as a quantitative, dynamic, non-linear model that incorporates feedback and interactions among model elements, in order to capture emergent outcomes and estimate health, economic and potentially other consequences to inform public policies. The guidance covers: when complex systems modeling is needed; principles for designing a complex systems model; and how to choose an appropriate modeling technique. This paper provides a definition to identify and characterize complex systems models for economic evaluations and proposes guidance on key aspects of the process for health economics analysis. This document will support the development of complex systems models, with impact on public health systems policy and decision making.

Machine learning model for predicting 1-year and 3-year all-cause mortality in ischemic heart failure patients

OBJECTIVE

Machine learning (ML) model has not been developed specifically for ischemic heart failure (HF) patients. Whether the performance of ML model is better than the MAGGIC risk score and NT-proBNP is unknown. The current study was to apply ML algorithm to build risk model for predicting 1-year and 3-year all-cause mortality in ischemic HF patient and to compare the performance of ML model with the MAGGIC risk score and NT-proBNP.

METHOD

Three ML algorithms without and with feature selection were used for model exploration, and the performance was determined based on the area under the curve (AUC) in five-fold cross-validation. The best performing ML model was selected and compared with the MAGGIC risk score and NT-proBNP. The calibration of ML model was assessed by the Brier score.

RESULTS

Random forest with feature selection had the highest AUC (0.742 and 95% CI: 0.697-0.787) for predicting 1-year all-cause mortality, and support vector machine without feature selection had the highest AUC (0.732 and 95% CI: 0.694-0.707) for predicting 3-year all-cause mortality. When compared to the MAGGIC risk score and NT-proBNP, ML model had a comparable AUC for predicting 1-year (0.742 vs 0.714 vs 0.694) and 3-year all-cause mortality (0.732 vs 0.712 vs 0.682). The Brier score for predicting 1-year and 3-year all-cause mortality were 0.068 and 0.174, respectively.

CONCLUSION

ML models predicted prognosis in ischemic HF with good discrimination and well calibration. These models may be used by clinicians as a decision-making tool to estimate the prognosis of ischemic HF patients.

A Tale of Three Recent Pandemics: Influenza, HIV and SARS-CoV-2

Emerging infectious diseases are one of the main threats to public health, with the potential to cause a pandemic when the infectious agent manages to spread globally. The first major pandemic to appear in the 20th century was the influenza pandemic of 1918, caused by the influenza A H1N1 strain that is characterized by a high fatality rate. Another major pandemic was caused by the human immunodeficiency virus (HIV), that started early in the 20th century and remained undetected until 1981. The ongoing HIV pandemic demonstrated a high mortality and morbidity rate, with discrepant impacts in different regions around the globe. The most recent major pandemic event, is the ongoing pandemic of COVID-19, caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which has caused over 5.7 million deaths since its emergence, 2 years ago. The aim of this work is to highlight the main determinants of the emergence, epidemic response and available countermeasures of these three pandemics, as we argue that such knowledge is paramount to prepare for the next pandemic. We analyse these pandemics’ historical and epidemiological contexts and the determinants of their emergence. Furthermore, we compare pharmaceutical and non-pharmaceutical interventions that have been used to slow down these three pandemics and zoom in on the technological advances that were made in the progress. Finally, we discuss the evolution of epidemiological modelling, that has become an essential tool to support public health policy making and discuss it in the context of these three pandemics. While these pandemics are caused by distinct viruses, that ignited in different time periods and in different regions of the globe, our work shows that many of the determinants of their emergence and countermeasures used to halt transmission were common. Therefore, it is important to further improve and optimize such approaches and adapt it to future threatening emerging infectious diseases.

Agent-based modeling of COVID-19 outbreaks for New York state and UK: Parameter identification algorithm

This paper uses Covasim, an agent-based model (ABM) of COVID-19, to evaluate and scenarios of epidemic spread in New York State (USA) and the UK. Epidemiological parameters such as contagiousness (virus transmission rate), initial number of infected people, and probability of being tested depend on the region's demographic and geographical features, the containment measures introduced; they are calibrated to data about COVID-19 spread in the region of interest. At the first stage of our study, epidemiological data (numbers of people tested, diagnoses, critical cases, hospitalizations, and deaths) for each of the mentioned regions were analyzed. The data were characterized in terms of seasonality, stationarity, and dependency spaces, and were extrapolated using machine learning techniques to specify unknown epidemiological parameters of the model. At the second stage, the Optuna optimizer based on the tree Parzen estimation method for objective function minimization was applied to determine the model's unknown parameters. The model was validated with the historical data of 2020. The modeled results of COVID-19 spread in New York State and the UK have demonstrated that if the level of testing and containment measures is preserved, the number of positive cases in New York State remain the same during March of 2021, while in the UK it will reduce.

Real-Time Infectious Disease Modeling to Inform Emergency Public Health Decision Making

Infectious disease transmission is a nonlinear process with complex, sometimes unintuitive dynamics. Modeling can transform information about a disease process and its parameters into quantitative projections that help decision makers compare public health response options. However, modelers face methodologic challenges, data challenges, and communication challenges, which are exacerbated under the time constraints of a public health emergency. We review methods, applications, challenges and opportunities for real-time infectious disease modeling during public health emergencies, with examples drawn from the two deadliest pandemics in recent history: HIV/AIDS and coronavirus disease 2019 (COVID-19).

Expected final online publication date for the Annual Review of Public Health, Volume 43 is April 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Emulator-based Bayesian optimization for efficient multi-objective calibration of an individual-based model of malaria

Individual-based models have become important tools in the global battle against infectious diseases, yet model complexity can make calibration to biological and epidemiological data challenging. We propose using a Bayesian optimization framework employing Gaussian process or machine learning emulator functions to calibrate a complex malaria transmission simulator. We demonstrate our approach by optimizing over a high-dimensional parameter space with respect to a portfolio of multiple fitting objectives built from datasets capturing the natural history of malaria transmission and disease progression. Our approach quickly outperforms previous calibrations, yielding an improved final goodness of fit. Per-objective parameter importance and sensitivity diagnostics provided by our approach offer epidemiological insights and enhance trust in predictions through greater interpretability.

Where Do We Go From Here? A Framework for Using Susceptible-Infectious-Recovered Models for Policy Making in Emerging Infectious Diseases

OBJECTIVES

Throughout the coronavirus disease 2019 pandemic, susceptible-infectious-recovered (SIR) modeling has been the preeminent modeling method to inform policy making worldwide. Nevertheless, the usefulness of such models has been subject to controversy. An evolution in the epidemiological modeling field is urgently needed, beginning with an agreed-upon set of modeling standards for policy recommendations. The objective of this article is to propose a set of modeling standards to support policy decision making.

METHODS

We identify and describe 5 broad standards: transparency, heterogeneity, calibration and validation, cost-benefit analysis, and model obsolescence and recalibration. We give methodological recommendations and provide examples in the literature that employ these standards well. We also develop and demonstrate a modeling practices checklist using existing coronavirus disease 2019 literature that can be employed by readers, authors, and reviewers to evaluate and compare policy modeling literature along our formulated standards.

RESULTS

We graded 16 articles using our checklist. On average, the articles met 6.81 of our 19 categories (36.7%). No articles contained any cost-benefit analyses and few were adequately transparent.

CONCLUSIONS

There is significant room for improvement in modeling pandemic policy. Issues often arise from a lack of transparency, poor modeling assumptions, lack of a system-wide perspective in modeling, and lack of flexibility in the academic system to rapidly iterate modeling as new information becomes available. In anticipation of future challenges, we encourage the modeling community at large to contribute toward the refinement and consensus of a shared set of standards for infectious disease policy modeling.

Cross-Entropy Method in Application to the SIRC Model

The study considers the usage of a probabilistic optimization method called Cross-Entropy (CE). This is the version of the Monte Carlo method created by Reuven Rubinstein (1997). It was developed in the context of determining rare events. Here we will present the way in which the CE method can be used for problems of optimization of epidemiological models, and more specifically the optimization of the Susceptible–Infectious–Recovered–Cross-immune (SIRC) model based on the functions supervising the care of specific groups in the model. With the help of weighted sampling, an attempt was made to find the fastest and most accurate version of the algorithm.