Calibrating Natural History of Cancer Models in the Presence of Data Incompatibility: Problems and Solutions

Challenges and Opportunities in Interdisciplinary Research and Real-World Data for Treatment Sequences in Health Technology Assessments

With an ever-increasing number of treatment options, the assessment of treatment sequences has become crucial in health technology assessment (HTA). This review systematically explores the multifaceted challenges inherent in evaluating sequences, delving into their interplay and nuances that go beyond economic model structures. We synthesised a 'roadmap' of literature from key methodological studies, highlighting the evolution of recent advances and emerging research themes. These insights were compared against HTA guidelines to identify potential avenues for future research. Our findings reveal a spectrum of challenges in sequence evaluation, encompassing selecting appropriate decision-analytic modelling approaches and comparators, deriving appropriate clinical effectiveness evidence in the face of data scarcity, scrutinising effectiveness assumptions and statistical adjustments, considering treatment displacement, and optimising model computations. Integrating methodologies from diverse disciplines-statistics, epidemiology, causal inference, operational research and computer science-has demonstrated promise in addressing these challenges. An updated review of application studies is warranted to provide detailed insights into the extent and manner in which these methodologies have been implemented. Data scarcity on the effectiveness of treatment sequences emerged as a dominant concern, especially because treatment sequences are rarely compared in clinical trials. Real-world data (RWD) provide an alternative means for capturing evidence on effectiveness and future research should prioritise harnessing causal inference methods, particularly Target Trial Emulation, to evaluate treatment sequence effectiveness using RWD. This approach is also adaptable for analysing trials harbouring sequencing information and adjusting indirect comparisons when collating evidence from heterogeneous sources. Such investigative efforts could lend support to reviews of HTA recommendations and contribute to synthesising external control arms involving treatment sequences.

Calibration and Validation of the Colorectal Cancer and Adenoma Incidence and Mortality (CRC-AIM) Microsimulation Model Using Deep Neural Networks

OBJECTIVES

Machine learning (ML)-based emulators improve the calibration of decision-analytical models, but their performance in complex microsimulation models is yet to be determined.

METHODS

We demonstrated the use of an ML-based emulator with the Colorectal Cancer (CRC)-Adenoma Incidence and Mortality (CRC-AIM) model, which includes 23 unknown natural history input parameters to replicate the CRC epidemiology in the United States. We first generated 15,000 input combinations and ran the CRC-AIM model to evaluate CRC incidence, adenoma size distribution, and the percentage of small adenoma detected by colonoscopy. We then used this data set to train several ML algorithms, including deep neural network (DNN), random forest, and several gradient boosting variants (i.e., XGBoost, LightGBM, CatBoost) and compared their performance. We evaluated 10 million potential input combinations using the selected emulator and examined input combinations that best estimated observed calibration targets. Furthermore, we cross-validated outcomes generated by the CRC-AIM model with those made by CISNET models. The calibrated CRC-AIM model was externally validated using the United Kingdom Flexible Sigmoidoscopy Screening Trial (UKFSST).

RESULTS

The DNN with proper preprocessing outperformed other tested ML algorithms and successfully predicted all 8 outcomes for different input combinations. It took 473 s for the trained DNN to predict outcomes for 10 million inputs, which would have required 190 CPU-years without our DNN. The overall calibration process took 104 CPU-days, which included building the data set, training, selecting, and hyperparameter tuning of the ML algorithms. While 7 input combinations had acceptable fit to the targets, a combination that best fits all outcomes was selected as the best vector. Almost all of the predictions made by the best vector laid within those from the CISNET models, demonstrating CRC-AIM's cross-model validity. Similarly, CRC-AIM accurately predicted the hazard ratios of CRC incidence and mortality as reported by UKFSST, demonstrating its external validity. Examination of the impact of calibration targets suggested that the selection of the calibration target had a substantial impact on model outcomes in terms of life-year gains with screening.

CONCLUSIONS

Emulators such as a DNN that is meticulously selected and trained can substantially reduce the computational burden of calibrating complex microsimulation models.

HIGHLIGHTS

Calibrating a microsimulation model, a process to find unobservable parameters so that the model fits observed data, is computationally complex.We used a deep neural network model, a popular machine learning algorithm, to calibrate the Colorectal Cancer Adenoma Incidence and Mortality (CRC-AIM) model.We demonstrated that our approach provides an efficient and accurate method to significantly speed up calibration in microsimulation models.The calibration process successfully provided cross-model validation of CRC-AIM against 3 established CISNET models and also externally validated against a randomized controlled trial.

Belimumab for Treating Active Autoantibody-Positive Systemic Lupus Erythematosus: An Evidence Review Group Perspective of a NICE Single Technology Appraisal

As part of its Single Technology Appraisal (STA) process, the National Institute for Health and Care Excellence (NICE) invited the manufacturer (GlaxoSmithKline [GSK]) of Benlysta (belimumab) to submit evidence regarding its clinical and cost effectiveness, for the review and possible extension of a previously conditionally approved intravenous formulation of belimumab for the treatment of active autoantibody-positive systemic lupus erythematosus (SLE). Kleijnen Systematic Reviews Ltd, in collaboration with Maastricht University Medical Centre+, was commissioned to act as the independent Evidence Review Group (ERG). This paper summarises the company submission (CS), presents the ERG's critical review of the clinical and cost-effectiveness evidence in the CS, highlights the key methodological considerations, and describes the development of the NICE guidance by the NICE Appraisal Committee.This appraisal is different to the previous appraisal in three ways: (1). This appraisal expands its definition of 'high disease activity'. (2). In TA397, belimumab was approved, with a managed access arrangement (MAA), for adults only. This appraisal includes subjects aged 5 years or older. (3). The original appraisal included an intravenous formulation only, but the current appraisal also includes a new subcutaneous formulation in the form of a prefilled pen.The company was required to collect real-world data from the British Isles Lupus Assessment Group Biologics Register (BILAG-BR), including data on the efficacy, safety, and effect on health-related quality of life of belimumab versus rituximab. This appraisal considers these data as well as additional clinical trial evidence presented in the company's updated submission to address uncertainties identified during the original appraisal. The ERG identified three major concerns with the evidence presented on the clinical effectiveness in the current submission; namely, short follow-up in the main comparative trials (BLISS-SC, BLISS-52 and BLISS-76); using the propensity score-matching (PSM) analysis in calibrating the cost-effectiveness model can severely bias the results in favour of belimumab; and BILAG-BR data are not suitable for a comparison of belimumab with rituximab.The main issue in the economic analysis was the uncertainty about long-term disease activity progression and resulting organ damage. The company's approach of calibrating modelled organ damage to longer-term data analysed using the PSM analysis was methodologically inappropriate. The final analysis comparing belimumab with standard treatment for the intravenous formulation resulted in an incremental cost-effectiveness ratio of £12,335 per quality-adjusted life-year (QALY) gained and £30,278 per QALY gained in the company's and ERG's base-case analyses, respectively. For the subcutaneous formulation, the final analysis resulted in £8480 per QALY gained and £29,313 per QALY gained in the company's and ERG's base-case analyses, respectively. NICE recommended belimumab in both intravenous and subcutaneous formulations as an add-on treatment option for active autoantibody-positive SLE in the HDA-2 subgroup.

Modelling the impact of the coronavirus pandemic on bowel cancer screening outcomes in England: A decision analysis to prepare for future screening disruption

The English Bowel Cancer Screening Programme invites people between the ages of 60 and 74 to take a Faecal Immunochemical Test every two years. This programme was interrupted during the coronavirus pandemic. The research aimed: (1) to estimate the impact of colorectal cancer (CRC) Faecal Immunochemical Test screening pauses of different lengths and the actual coronavirus-related screening pause in England, and (2) to analyse the most effective and cost-effective strategies to re-start CRC screening to prepare for future disruptions. The analysis used the validated Microsimulation Model in Cancer of the Bowel built in the R programming language. The model simulated the life course of a representative English screening population from 2019, by age, sex, socio-economic deprivation, and prior screening history. The modelling scenarios were based on assumptions and data from screening centres in England. Pausing bowel screening in England due to coronavirus pandemic is predicted to increase CRC deaths by 0.73% within 10 years and 0.13% over the population's lifetime, with excess deaths due to peak in 2023. More deaths are expected in men and people aged over 70. Pausing screening for longer would result in greater additional CRC cases and deaths. Postponing screening for everyone would be the most cost-effective strategy to minimise the impact of screening disruption without any additional endoscopy capacity. If endoscopy capacity can be increased, temporarily raising the Faecal Immunochemical Test threshold to 190 μg/g may help to minimise CRC deaths, particularly if screening programmes start from age 50 in the future.