Calibration and Validation of the Colorectal Cancer and Adenoma Incidence and Mortality (CRC-AIM) Microsimulation Model Using Deep Neural Networks

OBJECTIVES

Machine learning (ML)-based emulators improve the calibration of decision-analytical models, but their performance in complex microsimulation models is yet to be determined.

METHODS

We demonstrated the use of an ML-based emulator with the Colorectal Cancer (CRC)-Adenoma Incidence and Mortality (CRC-AIM) model, which includes 23 unknown natural history input parameters to replicate the CRC epidemiology in the United States. We first generated 15,000 input combinations and ran the CRC-AIM model to evaluate CRC incidence, adenoma size distribution, and the percentage of small adenoma detected by colonoscopy. We then used this data set to train several ML algorithms, including deep neural network (DNN), random forest, and several gradient boosting variants (i.e., XGBoost, LightGBM, CatBoost) and compared their performance. We evaluated 10 million potential input combinations using the selected emulator and examined input combinations that best estimated observed calibration targets. Furthermore, we cross-validated outcomes generated by the CRC-AIM model with those made by CISNET models. The calibrated CRC-AIM model was externally validated using the United Kingdom Flexible Sigmoidoscopy Screening Trial (UKFSST).

RESULTS

The DNN with proper preprocessing outperformed other tested ML algorithms and successfully predicted all 8 outcomes for different input combinations. It took 473 s for the trained DNN to predict outcomes for 10 million inputs, which would have required 190 CPU-years without our DNN. The overall calibration process took 104 CPU-days, which included building the data set, training, selecting, and hyperparameter tuning of the ML algorithms. While 7 input combinations had acceptable fit to the targets, a combination that best fits all outcomes was selected as the best vector. Almost all of the predictions made by the best vector laid within those from the CISNET models, demonstrating CRC-AIM's cross-model validity. Similarly, CRC-AIM accurately predicted the hazard ratios of CRC incidence and mortality as reported by UKFSST, demonstrating its external validity. Examination of the impact of calibration targets suggested that the selection of the calibration target had a substantial impact on model outcomes in terms of life-year gains with screening.

CONCLUSIONS

Emulators such as a DNN that is meticulously selected and trained can substantially reduce the computational burden of calibrating complex microsimulation models.

HIGHLIGHTS

Calibrating a microsimulation model, a process to find unobservable parameters so that the model fits observed data, is computationally complex.We used a deep neural network model, a popular machine learning algorithm, to calibrate the Colorectal Cancer Adenoma Incidence and Mortality (CRC-AIM) model.We demonstrated that our approach provides an efficient and accurate method to significantly speed up calibration in microsimulation models.The calibration process successfully provided cross-model validation of CRC-AIM against 3 established CISNET models and also externally validated against a randomized controlled trial.

Life-years gained resulting from screening colonoscopy compared with follow-up colonoscopy after a positive stool-based colorectal screening test

Screening colonoscopies for colorectal cancer (CRC) are typically covered without patient cost-sharing, whereas follow-up colonoscopies for positive stool-based screening tests often incur patient costs. The objective of this analysis was to estimate and compare the life-years gained (LYG) per average-risk screening colonoscopy and follow-up colonoscopy after a positive stool-based test to better inform CRC coverage policy and reimbursement decisions. CRC outcomes from screening and follow-up colonoscopies versus no screening were estimated using CRC-AIM in a simulated population of average-risk individuals screened between ages 45-75 years. The LYG/colonoscopy per 1000 individuals was 0.09 for screening colonoscopy and 0.29 for follow-up colonoscopy. 0.01 and 0.04 CRC cases and 0.01 and 0.02 CRC deaths were averted per screening and follow-up colonoscopies, respectively. Coverage policies should be revised to encourage individuals to complete recommended screening processes.

Lowering the colorectal cancer screening age improves predicted outcomes in a microsimulation model

AIMS

While most guidelines still recommend colorectal cancer (CRC) screening initiation at age 50 years in average-risk individuals, guideline-creating bodies are starting to lower the recommended age of initiation to 45 years to mitigate the trend of increasing CRC rates in younger populations. Using CRC-AIM, we modeled the impact of lowering the CRC screening initiation age, incorporating theoretical and reported adherence rates, for triennial multi-target stool DNA (mt-sDNA) or annual fecal immunochemical test (FIT) screening.

METHODS AND MATERIALS

Screening strategies were simulated for individuals without CRC at age 40 and screened from ages 50 to 75 or 45 to 75 years. Outcomes included CRC incidence, CRC mortality, and life-years gained (LYG) per 1000 individuals screened (compared with no screening). Models used theoretically perfect (100%) and previously reported (71% mt-sDNA; 43% FIT) adherence rates.

RESULTS

With perfect adherence, mt-sDNA and FIT resulted in 22.2 and 23.4 more predicted LYG, respectively, with screening initiation at age 45 versus 50 years; reported adherence resulted in 23.9 and 24.4 more LYG, respectively. With perfect adherence, screening initiation at age 45 versus 50 years resulted in 26.1 and 28.6 CRC cases, respectively, with mt-sDNA and 22.8 and 25.5 cases with FIT; with reported real-world adherence there were 28.5 and 31.2 cases, respectively, with mt-sDNA and 37.1 and 40.2 cases with FIT. Similar patterns were observed for CRC deaths. With screening initiation at age 45 and reported adherence, mt-sDNA averted 8.6 more CRC cases and 3.3 more deaths per 1000 individuals than FIT.

CONCLUSIONS

Estimated CRC screening outcomes improved by lowering the initiation age from 50 to 45 years. Incorporating reported adherence rates yields greater benefits from triennial mt-sDNA versus annual FIT screening.

Impact of Patient Adherence to Stool-Based Colorectal Cancer Screening and Colonoscopy Following a Positive Test on Clinical Outcomes

Colorectal cancer-screening models commonly assume 100% adherence, which is inconsistent with real-world experience. The influence of adherence to initial stool-based screening [fecal immunochemical test (FIT), multitarget stool DNA (mt-sDNA)] and follow-up colonoscopy (after a positive stool test) on colorectal cancer outcomes was modeled using the Colorectal Cancer and Adenoma Incidence and Mortality Microsimulation Model. Average-risk individuals without diagnosed colorectal cancer at age 40 undergoing annual FIT or triennial mt-sDNA screening from ages 50 to 75 were simulated. Primary analyses incorporated published mt-sDNA (71%) or FIT (43%) screening adherence, with follow-up colonoscopy adherence ranging from 40% to 100%. Secondary analyses simulated 100% adherence for stool-based screening and colonoscopy follow-up (S1), published adherence for stool-based screening with 100% adherence to colonoscopy follow-up (S2), and published adherence for both stool-based screening and colonoscopy follow-up after positive mt-sDNA (73%) or FIT (47%; S3). Outcomes were life-years gained (LYG) and colorectal cancer incidence and mortality reductions (per 1,000 individuals) versus no screening. Adherence to colonoscopy follow-up after FIT had to be 4%-13% higher than mt-sDNA to reach equivalent LYG. The theoretical S1 favored FIT versus mt-sDNA (LYG 316 vs. 297; colorectal cancer incidence reduction 68% vs. 64%; colorectal cancer mortality reduction 76% vs. 72%). The more realistic S2 and S3 favored mt-sDNA versus FIT (S2: LYG 284 vs. 245, colorectal cancer incidence reduction 61% vs. 50%, colorectal cancer mortality reduction 69% vs. 59%; S3: LYG 203 vs. 113, colorectal cancer incidence reduction 43% vs. 23%, colorectal cancer mortality reduction 49% vs. 27%, respectively). Incorporating realistic adherence rates for colorectal cancer screening influences modeled outcomes and should be considered when assessing comparative effectiveness. PREVENTION RELEVANCE: Adherence rates for initial colorectal cancer screening by FIT or mt-sDNA and for colonoscopy follow-up of a positive initial test influence the comparative effectiveness of these screening strategies. Using adherence rates based on published data for stool-based testing and colonoscopy follow-up yielded superior outcomes with an mt-sDNA versus FIT-screening strategy.

Impact of the sessile serrated polyp pathway on predicted colorectal cancer outcomes in the CRC-AIM model

10545

Background: Approximately 20–30% of colorectal cancers (CRC) arise from the serrated polyp pathway. The multitarget stool DNA (mt-sDNA) test has greater sensitivity to detect sessile serrated polyps (SSPs) than a leading fecal immunochemical test (FIT). However, most modeling analyses do not account for the contribution of the SSP pathway to risk of CRC. We used the CRC-AIM model to assess the impact of the SSP pathway on predicted CRC outcomes with mt-sDNA or FIT screening. Methods: A simulated cohort of average-risk US patients underwent triennial mt-sDNA or annual FIT screening from ages 50–75. The percentage of CRCs arising from the SSP pathway were modeled at 0% (base case), 20%, and 30%, with stool screening adherence based on theoretical (100%) or previously reported (mt-sDNA 71%; FIT 43%) rates. Published SSP sensitivities for mt-sDNA and FIT were used. All other model inputs were identical to CISNET models. Sensitivity analyses used screening adherence rates of 40–70%. Key outcomes were life-years gained (LYG), CRC incidence and CRC mortality per 1000 patients. Results: Including SSPs in the model demonstrated a greater loss of LYG with FIT than mt-sDNA (Table). At 100% adherence, compared with base case, modeling 20% or 30% SSP pathway CRCs resulted in a decrease of 9–15 LYG with FIT and 2–4 LYG with mt-sDNA, a decrease in CRC incidence reduction of 3.9–6.1% with FIT and 0.7–1.1% with mt-sDNA, and a decrease in CRC mortality reduction of 2.6–4.0% with FIT and 0.4–0.8% with mt-sDNA. Using previously reported adherence, compared with base case, modeling 20% or 30% SSP pathway CRCs resulted in a decrease of 13–20 LYG with FIT and 2–5 LYG with mt-sDNA, a decrease in CRC incidence reduction of 4.4–6.9% with FIT and 0.6–1.1% with mt-sDNA, and a decrease in CRC mortality reduction of 3.5–5.4% with FIT and 0.4–0.9% with mt-sDNA. Assuming reported adherence and 30% SSP pathway CRCs, mt-sDNA had 48 more LYG, 14.6% greater CRC incidence reduction, and 12.4% greater CRC mortality reduction than FIT. Assuming 30% SSP pathway CRCs, outcomes favored mt-sDNA vs FIT even after modeling equivalent adherence rates ranging from 40–70%. Conclusions: After incorporating the SSP pathway into the model, outcomes with mt-sDNA neared those of FIT at 100% screening adherence rates and surpassed FIT at more realistic reported screening adherence rates.[Table: see text]

Impact of screening and follow-up colonoscopy adenoma sensitivity on colorectal cancer screening outcomes in the CRC-AIM microsimulation model

BACKGROUND

Real-world data for patients with positive colorectal cancer (CRC) screening stool-tests demonstrate that adenoma detection rates are lower when endoscopists are blinded to the stool-test results. This suggests adenoma sensitivity may be lower for screening colonoscopy than for follow-up to a known positive stool-based test. Previous CRC microsimulation models assume identical sensitivities between screening and follow-up colonoscopies after positive stool-tests. The Colorectal Cancer and Adenoma Incidence and Mortality Microsimulation Model (CRC-AIM) was used to explore the impact on screening outcomes when assuming different adenoma sensitivity between screening and combined follow-up/surveillance colonoscopies.

METHODS

Modeled screening strategies included colonoscopy every 10 years, triennial multitarget stool DNA (mt-sDNA), or annual fecal immunochemical test (FIT) from 50 to 75 years. Outcomes were reported per 1000 individuals without diagnosed CRC at age 40. Base-case adenoma sensitivity values were identical for screening and follow-up/surveillance colonoscopies. Ranges of adenoma sensitivity values for colonoscopy performance were developed using different slopes of odds ratio adjustments and were designated as small, medium, or large impact scenarios.

RESULTS

As the differences in adenoma sensitivity for screening versus follow-up/surveillance colonoscopies became greater, life-years gained (LYG) and reductions in CRC-related incidence and mortality versus no screening increased for mt-sDNA and FIT and decreased for screening colonoscopy. The LYG relative to screening colonoscopy reached >90% with FIT in the base-case scenario and with mt-sDNA in a "medium impact" scenario.

CONCLUSIONS

Assuming identical adenoma sensitivities for screening and follow-up/surveillance colonoscopies underestimate the potential benefits of stool-based screening strategies.

Estimating the impact of differential adherence on the comparative effectiveness of stool-based colorectal cancer screening using the CRC-AIM microsimulation model

BACKGROUND

Real-world adherence to colorectal cancer (CRC) screening strategies is imperfect. The CRC-AIM microsimulation model was used to estimate the impact of imperfect adherence on the relative benefits and burdens of guideline-endorsed, stool-based screening strategies.

METHODS

Predicted outcomes of multi-target stool DNA (mt-sDNA), fecal immunochemical tests (FIT), and high-sensitivity guaiac-based fecal occult blood tests (HSgFOBT) were simulated for 40-year-olds free of diagnosed CRC. For robustness, imperfect adherence was incorporated in multiple ways and with extensive sensitivity analysis. Analysis 1 assumed adherence from 0%-100%, in 10% increments. Analysis 2 longitudinally applied real-world first-round differential adherence rates (base-case imperfect rates = 40% annual FIT vs 34% annual HSgFOBT vs 70% triennial mt-sDNA). Analysis 3 randomly assigned individuals to receive 1, 5, or 9 lifetime (9 = 100% adherence) mt-sDNA tests and 1, 5, or 9 to 26 (26 = 100% adherence) FIT tests. Outcomes are reported per 1000 individuals compared with no screening.

RESULTS

Each screening strategy decreased CRC incidence and mortality versus no screening. In individuals screened between ages 50-75 and adherence ranging from 10%a-100%, the life-years gained (LYG) for triennial mt-sDNA ranged from 133.1-300.0, for annual FIT from 96.3-318.1, and for annual HSgFOBT from 99.8-320.6. At base-case imperfect adherence rates, mt-sDNA resulted in 19.1% more LYG versus FIT, 25.4% more LYG versus HSgFOBT, and generally had preferable efficiency ratios while offering the most LYG. Completion of at least 21 FIT tests is needed to reach approximately the same LYG achieved with 9 mt-sDNA tests.

CONCLUSIONS

Adherence assumptions affect the conclusions of CRC screening microsimulations that are used to inform CRC screening guidelines. LYG from FIT and HSgFOBT are more sensitive to changes in adherence assumptions than mt-sDNA because they require more tests be completed for equivalent benefit. At imperfect adherence rates, mt-sDNA provides more LYG than FIT or HSgFOBT at an acceptable tradeoff in screening burden.