Using Active Learning for Speeding up Calibration in Simulation Models

Cost-effectiveness Analysis of Colorectal Cancer Screening Strategies Using Active Learning and Monte Carlo Simulation

INTRODUCTION

Detection of colorectal cancer (CRC) in the early stages through available screening tests increases the patient's survival chances. Multimodal screening policies can benefit patients by providing more diverse screening options and balancing the risks and benefits of screening tests. We investigate the cost-effectiveness of a wide variety of multimodal CRC screening policies.

METHODS

We developed a Monte Carlo simulation framework to model CRC dynamics. We proposed an innovative calibration process using machine learning models to estimate age- and size-specific adenomatous polyps' progression and regression rates. The proposed approach significantly expedites the model parameter space search.

RESULTS

Two multimodal proposed policies (i.e., 1] colonoscopy at 50 y and fecal occult blood test annually between 60 and 75 y and 2] colonoscopy at 50 and 60 y and fecal immunochemical test annually between 70 and 75 y) are identified as efficient frontier policies. Both policies are cost-effective at a willingness to pay of $50,000. Sensitivity analyses were performed to assess the sensitivity of results to a change in screening test costs as well as adherence behavior. The sensitivity analysis results suggest that the proposed policies are mostly robust to the considered changes in screening test costs, as there is a significant overlap between the efficient frontier policies of the baseline and the sensitivity analysis cases. However, the efficient frontier policies were more sensitive to changes in adherence behavior.

CONCLUSION

Generally, combining stool-based tests with visual tests will benefit patients with higher life expectancy and a lower expected cost compared with unimodal screening policies. Colonoscopy at younger ages (when the colonoscopy complication risk is lower) and stool-based tests at older ages are shown to be more effective.

HIGHLIGHTS

We propose a detailed Markov model to capture the colorectal cancer (CRC) dynamics. The proposed Markov model presents the detailed dynamics of adenomas progression to CRC.We use more than 44,000 colonoscopy reports and available data in the literature to calibrate the proposed Markov model using an innovative approach that leverages machine learning models to expedite the calibration process.We investigate the cost-effectiveness of a wide variety of multimodal CRC screening policies and compare their performances with the current in-practice policies.

Modeling Thyroid Cancer Epidemiology in the United States Using Papillary Thyroid Carcinoma Microsimulation Model

OBJECTIVES

Thyroid cancer incidence increased over 200% from 1992 to 2018, whereas mortality rates had not increased proportionately. The increased incidence has been attributed primarily to the detection of subclinical disease, raising important questions related to thyroid cancer control. We developed the Papillary Thyroid Carcinoma Microsimulation model (PATCAM) to answer them, including the impact of overdiagnosis on thyroid cancer incidence.

METHODS

PATCAM simulates individuals from age 15 until death in birth cohorts starting from 1975 using 4 inter-related components, including natural history, detection, post-diagnosis, and other-cause mortality. PATCAM was built using high-quality data and calibrated against observed age-, sex-, and stage-specific incidence in the United States as reported by the Surveillance, Epidemiology, and End Results database. PATCAM was validated against US thyroid cancer mortality and 3 active surveillance studies, including the largest and longest running thyroid cancer active surveillance cohort in the world (from Japan) and 2 from the United States.

RESULTS

PATCAM successfully replicated age- and stage-specific papillary thyroid cancers (PTC) incidence and mean tumor size at diagnosis and PTC mortality in the United States between 1975 and 2015. PATCAM accurately predicted the proportion of tumors that grew more than 3 mm and 5 mm in 5 years and 10 years, aligning with the 95% confidence intervals of the reported rates from active surveillance studies in most cases.

CONCLUSIONS

PATCAM successfully reproduced observed US thyroid cancer incidence and mortality over time and was externally validated. PATCAM can be used to identify factors that influence the detection of subclinical PTCs.

Calibration and Validation of the Colorectal Cancer and Adenoma Incidence and Mortality (CRC-AIM) Microsimulation Model Using Deep Neural Networks

OBJECTIVES

Machine learning (ML)-based emulators improve the calibration of decision-analytical models, but their performance in complex microsimulation models is yet to be determined.

METHODS

We demonstrated the use of an ML-based emulator with the Colorectal Cancer (CRC)-Adenoma Incidence and Mortality (CRC-AIM) model, which includes 23 unknown natural history input parameters to replicate the CRC epidemiology in the United States. We first generated 15,000 input combinations and ran the CRC-AIM model to evaluate CRC incidence, adenoma size distribution, and the percentage of small adenoma detected by colonoscopy. We then used this data set to train several ML algorithms, including deep neural network (DNN), random forest, and several gradient boosting variants (i.e., XGBoost, LightGBM, CatBoost) and compared their performance. We evaluated 10 million potential input combinations using the selected emulator and examined input combinations that best estimated observed calibration targets. Furthermore, we cross-validated outcomes generated by the CRC-AIM model with those made by CISNET models. The calibrated CRC-AIM model was externally validated using the United Kingdom Flexible Sigmoidoscopy Screening Trial (UKFSST).

RESULTS

The DNN with proper preprocessing outperformed other tested ML algorithms and successfully predicted all 8 outcomes for different input combinations. It took 473 s for the trained DNN to predict outcomes for 10 million inputs, which would have required 190 CPU-years without our DNN. The overall calibration process took 104 CPU-days, which included building the data set, training, selecting, and hyperparameter tuning of the ML algorithms. While 7 input combinations had acceptable fit to the targets, a combination that best fits all outcomes was selected as the best vector. Almost all of the predictions made by the best vector laid within those from the CISNET models, demonstrating CRC-AIM's cross-model validity. Similarly, CRC-AIM accurately predicted the hazard ratios of CRC incidence and mortality as reported by UKFSST, demonstrating its external validity. Examination of the impact of calibration targets suggested that the selection of the calibration target had a substantial impact on model outcomes in terms of life-year gains with screening.

CONCLUSIONS

Emulators such as a DNN that is meticulously selected and trained can substantially reduce the computational burden of calibrating complex microsimulation models.

HIGHLIGHTS

Calibrating a microsimulation model, a process to find unobservable parameters so that the model fits observed data, is computationally complex.We used a deep neural network model, a popular machine learning algorithm, to calibrate the Colorectal Cancer Adenoma Incidence and Mortality (CRC-AIM) model.We demonstrated that our approach provides an efficient and accurate method to significantly speed up calibration in microsimulation models.The calibration process successfully provided cross-model validation of CRC-AIM against 3 established CISNET models and also externally validated against a randomized controlled trial.

Active Learning for Multi-way Sensitivity Analysis with Application to Disease Screening Modeling

Sensitivity analysis is an important aspect of model development as it can be used to assess the level of confidence that is associated with the outcomes of a study. In many practical problems, sensitivity analysis involves evaluating a large number of parameter combinations which may require an extensive amount of time and resources. However, such a computational burden can be avoided by identifying smaller subsets of parameter combinations that can be later used to generate the desired outcomes for other parameter combinations. In this study, we investigate machine learning-based approaches for speeding up the sensitivity analysis. Furthermore, we apply feature selection methods to identify the relative importance of quantitative model parameters in terms of their predictive ability on the outcomes. Finally, we highlight the effectiveness of active learning strategies in improving the sensitivity analysis processes by reducing the total number of quantitative model runs required to construct a high-performance prediction model. Our experiments on two datasets obtained from the sensitivity analysis performed for two disease screening modeling studies indicate that ensemble methods such as Random Forests and XGBoost consistently outperform other machine learning algorithms in the prediction task of the associated sensitivity analysis. In addition, we note that active learning can lead to significant speed-ups in sensitivity analysis by enabling the selection of more useful parameter combinations (i.e., instances) to be used for prediction models.

The impact of vaccination to control COVID-19 burden in the United States: A simulation modeling approach

INTRODUCTION

Vaccination programs aim to control the COVID-19 pandemic. However, the relative impacts of vaccine coverage, effectiveness, and capacity in the context of nonpharmaceutical interventions such as mask use and physical distancing on the spread of SARS-CoV-2 are unclear. Our objective was to examine the impact of vaccination on the control of SARS-CoV-2 using our previously developed agent-based simulation model.

METHODS

We applied our agent-based model to replicate COVID-19-related events in 1) Dane County, Wisconsin; 2) Milwaukee metropolitan area, Wisconsin; 3) New York City (NYC). We evaluated the impact of vaccination considering the proportion of the population vaccinated, probability that a vaccinated individual gains immunity, vaccination capacity, and adherence to nonpharmaceutical interventions. We estimated the timing of pandemic control, defined as the date after which only a small number of new cases occur.

RESULTS

The timing of pandemic control depends highly on vaccination coverage, effectiveness, and adherence to nonpharmaceutical interventions. In Dane County and Milwaukee, if 50% of the population is vaccinated with a daily vaccination capacity of 0.25% of the population, vaccine effectiveness of 90%, and the adherence to nonpharmaceutical interventions is 60%, controlled spread could be achieved by June 2021 versus October 2021 in Dane County and November 2021 in Milwaukee without vaccine.

DISCUSSION

In controlling the spread of SARS-CoV-2, the impact of vaccination varies widely depending not only on effectiveness and coverage, but also concurrent adherence to nonpharmaceutical interventions.

Choosing a Metamodel of a Simulation Model for Uncertainty Quantification

BACKGROUND

Metamodeling may substantially reduce the computational expense of individual-level state transition simulation models (IL-STM) for calibration, uncertainty quantification, and health policy evaluation. However, because of the lack of guidance and readily available computer code, metamodels are still not widely used in health economics and public health. In this study, we provide guidance on how to choose a metamodel for uncertainty quantification.

METHODS

We built a simulation study to evaluate the prediction accuracy and computational expense of metamodels for uncertainty quantification using life-years gained (LYG) by treatment as the IL-STM outcome. We analyzed how metamodel accuracy changes with the characteristics of the simulation model using a linear model (LM), Gaussian process regression (GP), generalized additive models (GAMs), and artificial neural networks (ANNs). Finally, we tested these metamodels in a case study consisting of a probabilistic analysis of a lung cancer IL-STM.

RESULTS

In a scenario with low uncertainty in model parameters (i.e., small confidence interval), sufficient numbers of simulated life histories, and simulation model runs, commonly used metamodels (LM, ANNs, GAMs, and GP) have similar, good accuracy, with errors smaller than 1% for predicting LYG. With a higher level of uncertainty in model parameters, the prediction accuracy of GP and ANN is superior to LM. In the case study, we found that in the worst case, the best metamodel had an error of about 2.1%.

CONCLUSION

To obtain good prediction accuracy, in an efficient way, we recommend starting with LM, and if the resulting accuracy is insufficient, we recommend trying ANNs and eventually also GP regression.

The Impact of Vaccination to Control COVID-19 Burden in the United States: A Simulation Modeling Approach

Introduction

Vaccination programs aim to control the COVID-19 pandemic. However, the relative impacts of vaccine coverage, effectiveness, and capacity in the context of nonpharmaceutical interventions such as mask use and physical distancing on the spread of SARS-CoV-2 are unclear. Our objective was to examine the impact of vaccination on the control of SARS-CoV-2 using our previously developed agent-based simulation model.

Methods

We applied our agent-based model to replicate COVID-19-related events in 1) Dane County, Wisconsin; 2) Milwaukee metropolitan area, Wisconsin; 3) New York City (NYC). We evaluated the impact of vaccination considering the proportion of the population vaccinated, probability that a vaccinated individual gains immunity, vaccination capacity, and adherence to nonpharmaceutical interventions. The primary outcomes were the number of confirmed COVID-19 cases and the timing of pandemic control, defined as the date after which only a small number of new cases occur. We also estimated the number of cases without vaccination.

Results

The timing of pandemic control depends highly on vaccination coverage, effectiveness, and adherence to nonpharmaceutical interventions. In Dane County and Milwaukee, if 50% of the population is vaccinated with a daily vaccination capacity of 0.1% of the population, vaccine effectiveness of 90%, and the adherence to nonpharmaceutical interventions is 65%, controlled spread could be achieved by July 2021 and August 2021, respectively versus in March 2022 in both regions without vaccine. If adherence to nonpharmaceutical interventions increases to 70%, controlled spread could be achieved by May 2021 and April 2021 in Dane County and Milwaukee, respectively.

Discussion

In controlling the spread of SARS-CoV-2, the impact of vaccination varies widely depending not only on effectiveness and coverage, but also concurrent adherence to nonpharmaceutical interventions. The effect of SARS-CoV-2 variants was not considered.

Primary Funding Source

National Institute of Allergy and Infectious Diseases.

Novel microsimulation model of tobacco use behaviours and outcomes: calibration and validation in a US population

BACKGROUND AND OBJECTIVE

Simulation models can project effects of tobacco use and cessation and inform tobacco control policies. Most existing tobacco models do not explicitly include relapse, a key component of the natural history of tobacco use. Our objective was to develop, calibrate and validate a novel individual-level microsimulation model that would explicitly include smoking relapse and project cigarette smoking behaviours and associated mortality risks.

METHODS

We developed the Simulation of Tobacco and Nicotine Outcomes and Policy (STOP) model, in which individuals transition monthly between tobacco use states (current/former/never) depending on rates of initiation, cessation and relapse. Simulated individuals face tobacco use-stratified mortality risks. For US women and men, we conducted cross-validation with a Cancer Intervention and Surveillance Modeling Network (CISNET) model. We then incorporated smoking relapse and calibrated cessation rates to reflect the difference between a transient quit attempt and sustained abstinence. We performed external validation with the National Health Interview Survey (NHIS) and the linked National Death Index. Comparisons were based on root-mean-square error (RMSE).

RESULTS

In cross-validation, STOP-generated projections of current/former/never smoking prevalence fit CISNET-projected data well (coefficient of variation (CV)-RMSE≤15%). After incorporating smoking relapse, multiplying the CISNET-reported cessation rates for women/men by 7.75/7.25, to reflect the ratio of quit attempts to sustained abstinence, resulted in the best approximation to CISNET-reported smoking prevalence (CV-RMSE 2%/3%). In external validation using these new multipliers, STOP-generated cumulative mortality curves for 20-year-old current smokers and never smokers each had CV-RMSE ≤1% compared with NHIS. In simulating those surveyed by NHIS in 1997, the STOP-projected prevalence of current/former/never smokers annually (1998-2009) was similar to that reported by NHIS (CV-RMSE 12%).

CONCLUSIONS

The STOP model, with relapse included, performed well when validated to US smoking prevalence and mortality. STOP provides a flexible framework for policy-relevant analysis of tobacco and nicotine product use.

Multiobjective Calibration of Disease Simulation Models Using Gaussian Processes

Background. Developing efficient procedures of model calibration, which entails matching model predictions to observed outcomes, has gained increasing attention. With faithful but complex simulation models established for cancer diseases, key parameters of cancer natural history can be investigated for possible fits, which can subsequently inform optimal prevention and treatment strategies. When multiple calibration targets exist, one approach to identifying optimal parameters relies on the Pareto frontier. However, computational burdens associated with higher-dimensional parameter spaces require a metamodeling approach. The goal of this work is to explore multiobjective calibration using Gaussian process regression (GPR) with an eye toward how multiple goodness-of-fit (GOF) criteria identify Pareto-optimal parameters. Methods. We applied GPR, a metamodeling technique, to estimate colorectal cancer (CRC)-related prevalence rates simulated from a microsimulation model of CRC natural history, known as the Colon Modeling Open Source Tool (CMOST). We embedded GPR metamodels within a Pareto optimization framework to identify best-fitting parameters for age-, adenoma-, and adenoma staging-dependent transition probabilities and risk factors. The Pareto frontier approach is demonstrated using genetic algorithms with both sum-of-squared errors (SSEs) and Poisson deviance GOF criteria. Results. The GPR metamodel is able to approximate CMOST outputs accurately on 2 separate parameter sets. Both GOF criteria are able to identify different best-fitting parameter sets on the Pareto frontier. The SSE criterion emphasizes the importance of age-specific adenoma progression parameters, while the Poisson criterion prioritizes adenoma-specific progression parameters. Conclusion. Different GOF criteria assert different components of the CRC natural history. The combination of multiobjective optimization and nonparametric regression, along with diverse GOF criteria, can advance the calibration process by identifying optimal regions of the underlying parameter landscape.

High-throughput cancer hypothesis testing with an integrated PhysiCell-EMEWS workflow

BACKGROUND

Cancer is a complex, multiscale dynamical system, with interactions between tumor cells and non-cancerous host systems. Therapies act on this combined cancer-host system, sometimes with unexpected results. Systematic investigation of mechanistic computational models can augment traditional laboratory and clinical studies, helping identify the factors driving a treatment's success or failure. However, given the uncertainties regarding the underlying biology, these multiscale computational models can take many potential forms, in addition to encompassing high-dimensional parameter spaces. Therefore, the exploration of these models is computationally challenging. We propose that integrating two existing technologies-one to aid the construction of multiscale agent-based models, the other developed to enhance model exploration and optimization-can provide a computational means for high-throughput hypothesis testing, and eventually, optimization.

RESULTS

In this paper, we introduce a high throughput computing (HTC) framework that integrates a mechanistic 3-D multicellular simulator (PhysiCell) with an extreme-scale model exploration platform (EMEWS) to investigate high-dimensional parameter spaces. We show early results in applying PhysiCell-EMEWS to 3-D cancer immunotherapy and show insights on therapeutic failure. We describe a generalized PhysiCell-EMEWS workflow for high-throughput cancer hypothesis testing, where hundreds or thousands of mechanistic simulations are compared against data-driven error metrics to perform hypothesis optimization.

CONCLUSIONS

While key notational and computational challenges remain, mechanistic agent-based models and high-throughput model exploration environments can be combined to systematically and rapidly explore key problems in cancer. These high-throughput computational experiments can improve our understanding of the underlying biology, drive future experiments, and ultimately inform clinical practice.

Extreme-scale Dynamic Exploration of a Distributed Agent-based Model with the EMEWS Framework

Agent-based models (ABMs) integrate multiple scales of behavior and data to produce higher-order dynamic phenomena and are increasingly used in the study of important social complex systems in biomedicine, socio-economics and ecology/resource management. However, the development, validation and use of ABMs is hampered by the need to execute very large numbers of simulations in order to identify their behavioral properties, a challenge accentuated by the computational cost of running realistic, large-scale, potentially distributed ABM simulations. In this paper we describe the Extreme-scale Model Exploration with Swift (EMEWS) framework, which is capable of efficiently composing and executing large ensembles of simulations and other "black box" scientific applications while integrating model exploration (ME) algorithms developed with the use of widely available 3rd-party libraries written in popular languages such as R and Python. EMEWS combines novel stateful tasks with traditional run-to-completion many task computing (MTC) and solves many problems relevant to high-performance workflows, including scaling to very large numbers (millions) of tasks, maintaining state and locality information, and enabling effective multiple-language problem solving. We present the high-level programming model of the EMEWS framework and demonstrate how it is used to integrate an active learning ME algorithm to dynamically and efficiently characterize the parameter space of a large and complex, distributed Message Passing Interface (MPI) agent-based infectious disease model.

Modeling and comparison of alternative approaches for sector duration optimization in a dedicated radiosurgery system

Stereotactic radiosurgery (SRS) is an effective technique to treat brain metastasis for which several inverse planning methods may be appropriate. We compare three different optimization models for segment duration optimization in SRS using Leksell Gamma Knife^® Icon^TM (Elekta, Stockholm, Sweden). We investigate (1) a linear programming approach, (2) a piecewise quadratic penalty approach, and (3) an unconstrained convex moment-based penalty approach. We examine the performances of these approaches using anonymized data from 14 previously treated cases. In addition, we investigate the important modeling question of selecting weights for the objective functions where we use a simulated annealing algorithm to determine these weights for each model. The inverse plans obtained via optimization models are compared against each other and against the clinical plans. The three inverse planning models can all yield optimal treatment plans in a reasonable amount of time and the treatment plans obtained by these models meet or exceed clinical guidelines while displaying high conformity.

The University of Wisconsin Breast Cancer Epidemiology Simulation Model: An Update

The University of Wisconsin Breast Cancer Epidemiology Simulation Model (UWBCS), also referred to as Model W, is a discrete-event microsimulation model that uses a systems engineering approach to replicate breast cancer epidemiology in the US over time. This population-based model simulates the lifetimes of individual women through 4 main model components: breast cancer natural history, detection, treatment, and mortality. A key feature of the UWBCS is that, in addition to specifying a population distribution in tumor growth rates, the model allows for heterogeneity in tumor behavior, with some tumors having limited malignant potential (i.e., would never become fatal in a woman's lifetime if left untreated) and some tumors being very aggressive based on metastatic spread early in their onset. The model is calibrated to Surveillance, Epidemiology, and End Results (SEER) breast cancer incidence and mortality data from 1975 to 2010, and cross-validated against data from the Wisconsin cancer reporting system. The UWBCS model generates detailed outputs including underlying disease states and observed clinical outcomes by age and calendar year, as well as costs, resource usage, and quality of life associated with screening and treatment. The UWBCS has been recently updated to account for differences in breast cancer detection, treatment, and survival by molecular subtypes (defined by ER/HER2 status), to reflect the recent advances in screening and treatment, and to consider a range of breast cancer risk factors, including breast density, race, body-mass-index, and the use of postmenopausal hormone therapy. Therefore, the model can evaluate novel screening strategies, such as risk-based screening, and can assess breast cancer outcomes by breast cancer molecular subtype. In this article, we describe the most up-to-date version of the UWBCS.

Bayesian Methods for Calibrating Health Policy Models: A Tutorial

Mathematical simulation models are commonly used to inform health policy decisions. These health policy models represent the social and biological mechanisms that determine health and economic outcomes, combine multiple sources of evidence about how policy alternatives will impact those outcomes, and synthesize outcomes into summary measures salient for the policy decision. Calibrating these health policy models to fit empirical data can provide face validity and improve the quality of model predictions. Bayesian methods provide powerful tools for model calibration. These methods summarize information relevant to a particular policy decision into (1) prior distributions for model parameters, (2) structural assumptions of the model, and (3) a likelihood function created from the calibration data, combining these different sources of evidence via Bayes' theorem. This article provides a tutorial on Bayesian approaches for model calibration, describing the theoretical basis for Bayesian calibration approaches as well as pragmatic considerations that arise in the tasks of creating calibration targets, estimating the posterior distribution, and obtaining results to inform the policy decision. These considerations, as well as the specific steps for implementing the calibration, are described in the context of an extended worked example about the policy choice to provide (or not provide) treatment for a hypothetical infectious disease. Given the many simplifications and subjective decisions required to create prior distributions, model structure, and likelihood, calibration should be considered an exercise in creating a reasonable model that produces valid evidence for policy, rather than as a technique for identifying a unique theoretically optimal summary of the evidence.