A Review of Survival Analysis Methods Used in NICE Technology Appraisals of Cancer Treatments: Consistency, Limitations, and Areas for Improvement

Assessing the Performance of Alternative Methods for Estimating Long-Term Survival Benefit of Immuno-oncology Therapies

OBJECTIVES

This study aimed to determine the accuracy and consistency of established methods of extrapolating mean survival for immuno-oncology (IO) therapies, the extent of any systematic biases in estimating long-term clinical benefit, what influences the magnitude of any bias, and the potential implications for health technology assessment.

METHODS

A targeted literature search was conducted to identify published long-term follow-up from clinical trials of immune-checkpoint inhibitors. Earlier published results were identified and Kaplan-Meier estimates for short- and long-term follow-up were digitized and converted to pseudo-individual patient data using an established algorithm. Six standard parametric, 5 flexible parametric, and 2 mixture-cure models (MCMs) were used to extrapolate long-term survival. Mean and restricted mean survival time (RMST) were estimated and compared between short- and long-term follow-up.

RESULTS

Predicted RMST from extrapolation of early data underestimated observed RMST in long-term follow-up for 184 of 271 extrapolations. All models except the MCMs frequently underestimated observed RMST. Mean survival estimates increased with longer follow-up in 196 of 270 extrapolations. The increase exceeded 20% in 122 extrapolations. Log-logistic and log-normal models showed the smallest change with additional follow-up. MCM performance varied substantially with functional form.

CONCLUSIONS

Standard and flexible parametric models frequently underestimate mean survival for IO treatments. Log-logistic and log-normal models may be the most pragmatic and parsimonious solutions for estimating IO mean survival from immature data. Flexible parametric models may be preferred when the data used in health technology assessment are more mature. MCMs fitted to immature data produce unreliable results and are not recommended.

Challenges and Opportunities in Interdisciplinary Research and Real-World Data for Treatment Sequences in Health Technology Assessments

With an ever-increasing number of treatment options, the assessment of treatment sequences has become crucial in health technology assessment (HTA). This review systematically explores the multifaceted challenges inherent in evaluating sequences, delving into their interplay and nuances that go beyond economic model structures. We synthesised a 'roadmap' of literature from key methodological studies, highlighting the evolution of recent advances and emerging research themes. These insights were compared against HTA guidelines to identify potential avenues for future research. Our findings reveal a spectrum of challenges in sequence evaluation, encompassing selecting appropriate decision-analytic modelling approaches and comparators, deriving appropriate clinical effectiveness evidence in the face of data scarcity, scrutinising effectiveness assumptions and statistical adjustments, considering treatment displacement, and optimising model computations. Integrating methodologies from diverse disciplines-statistics, epidemiology, causal inference, operational research and computer science-has demonstrated promise in addressing these challenges. An updated review of application studies is warranted to provide detailed insights into the extent and manner in which these methodologies have been implemented. Data scarcity on the effectiveness of treatment sequences emerged as a dominant concern, especially because treatment sequences are rarely compared in clinical trials. Real-world data (RWD) provide an alternative means for capturing evidence on effectiveness and future research should prioritise harnessing causal inference methods, particularly Target Trial Emulation, to evaluate treatment sequence effectiveness using RWD. This approach is also adaptable for analysing trials harbouring sequencing information and adjusting indirect comparisons when collating evidence from heterogeneous sources. Such investigative efforts could lend support to reviews of HTA recommendations and contribute to synthesising external control arms involving treatment sequences.

Treatment Effect Waning Assumptions: A Review of National Institute of Health and Care Excellence Technology Appraisals

OBJECTIVES

This study aims to review the National Institute of Health and Care Excellence (NICE) technology assessments to gain insights into the implementation of treatment effect (TE) waning, whereby the hazard or survival in an assessed technology converges to that of the comparator. This analysis aims to contribute to inform future guidance in this area.

METHODS

Technology appraisals published October 20, 2021 to September 20, 2023 were reviewed and data extracted on TE waning circumstances, methods, and rationale to compile a database based on 3 research questions: When are TE waning assumptions used? What methods are used? Why have the company/Evidence Assessment Group/committee preferred these methods?

RESULTS

Both the evidence assessment group/company and the committee included TE waning assumptions in 28 appraisals. There was no pattern of waning assumptions between shorter (<20 years) and longer (>20 years) time horizons. The most prominent time point for applying waning assumptions was at 5 years, with 30 out of 59 (50.8%) of the methods applied used 5 years. Stopping rules were used in 21 out of 30 (70.1%) of the appraisals for which the committee included waning, and waning assumptions were used more in oncology. The most common reason given for including TE waning assumptions was precedent from prior appraisals.

CONCLUSIONS

Considerable heterogeneity existed in both the methods used and justifications given for TE waning assumptions. This variability poses a risk of inconsistent decision making. Reliance on past appraisals emphasizes the necessity to advocate for evidence-driven approaches and underscores the demand for guidance on suitable methods for incorporating assumptions.

SurvInt: a simple tool to obtain precise parametric survival extrapolations

BACKGROUND

Economic evaluation of emerging health technologies is mandated by agencies such as the National Institute for Health and Care Excellence (NICE) to ensure their cost is proportional to their benefit. To avoid bias, NICE stipulate that the benefit of a treatment is assessed across the lifetime of the patient population, which can be many decades. Unfortunately, follow-up from a clinical trial will not usually cover the required period and the observed follow-up will require extrapolation. For survival data this is often done by selecting a preferred model from a set of candidate parametric models. This approach is limited in that the choice of model is restricted to those originally fitted. What if none of the models are consistent with clinical prediction or external data?

METHOD/RESULTS

This paper introduces SurvInt, a tool that estimates the parameters of common parametric survival models which interpolate key survival time co-ordinates specified by the user, which could come from external trials, real world data or expert clinical opinion. This is achieved by solving simultaneous equations based on the survival functions of the parametric models. The application of SurvInt is shown through two examples where traditional parametric modelling did not produce models that were consistent with external data or clinical opinion. Additional features include model averaging, mixture cure models, background mortality, piecewise modelling, restricted mean survival time estimation and probabilistic sensitivity analysis.

CONCLUSIONS

SurvInt allows precise parametric survival models to be estimated and carried forward into economic models. It provides access to extrapolations that are consistent with multiple data sources such as observed data and clinical predictions, opening the door to precise exploration of regions of uncertainty/disagreement. SurvInt could avoid the need for post-hoc adjustments for complications such as treatment switching, which are often applied to obtain a plausible survival model but at the cost of introducing additional uncertainty. Phase III clinical trials are not designed with extrapolation in mind, and so it is sensible to consider alternative approaches to predict future survival that incorporate external information.

Extending Beyond Bagust and Beale: Fully Parametric Piecewise Exponential Models for Extrapolation of Survival Outcomes in Health Technology Assessment

OBJECTIVES

When extrapolating time-to-event data the Bagust and Beale (B&B) approach uses the Kaplan-Meier survival function until a manually chosen time point, after which a constant hazard is assumed. This study demonstrates an objective statistical approach to estimate this time point.

METHODS

We estimate piecewise exponential models (PEMs), whereby the hazard function is partitioned into segments each with constant hazards. The boundaries of these segments are known as change points. Our approach determines the location and number of change points in PEMs from which the hazard in the final segment is used to model long-term survival. We reviewed previous applications of the B&B approach in National Institute for Health and Care Excellence Technology Appraisals (TAs) completed between July 2011 and June 2017. The time points after which constant hazards were assumed were compared between PEMs and the B&B approaches. When further survival data were published following the original TA, we compared these updated estimates to predicted survival from the PEM and other parametric models adjusted for general population mortality.

RESULTS

Six of the 59 TAs in this review considered the B&B approach. There was general agreement between the location of time points identified through the PEM and the B&B approaches. In 2 of the identified TAs the best fitting model to the data was a no-change-point model. Of the 3 TAs for which further survival data became available, PEM provided the closest prediction for survival outcomes in 2 TAs.

CONCLUSIONS

PEMs are useful for survival extrapolation when a long-term constant hazard trend for the disease is clinically plausible.

Survival Extrapolation Incorporating General Population Mortality Using Excess Hazard and Cure Models: A Tutorial

BACKGROUND

Different parametric survival models can lead to widely discordant extrapolations and decision uncertainty in cost-effectiveness analyses. The use of excess hazard (EH) methods, which incorporate general population mortality data, has the potential to reduce model uncertainty. This review highlights key practical considerations of EH methods for estimating long-term survival.

METHODS

Demonstration of methods used a case study of 686 patients from the German Breast Cancer Study Group, followed for a maximum of 7.3 y and divided into low (1/2) and high (3) grade cancers. Seven standard parametric survival models were fit to each group separately. The same 7 distributions were then used in an EH framework, which incorporated general population mortality rates, and fitted both with and without a cure parameter. Survival extrapolations, restricted mean survival time (RMST), and difference in RMST between high and low grades were compared up to 30 years along with Akaike information criterion goodness-of-fit and cure fraction estimates. The sensitivity of the EH models to lifetable misspecification was investigated.

RESULTS

In our case study, variability in survival extrapolations was extensive across the standard models, with 30-y RMST ranging from 7.5 to 14.3 y. Incorporation of general population mortality rates using EH cure methods substantially reduced model uncertainty, whereas EH models without cure had less of an effect. Long-term treatment effects approached the null for most models but at varying rates. Lifetable misspecification had minimal effect on RMST differences.

CONCLUSIONS

EH methods may be useful for survival extrapolation, and in cancer, EHs may decrease over time and be easier to extrapolate than all-cause hazards. EH cure models may be helpful when cure is plausible and likely to result in less extrapolation variability.

HIGHLIGHTS

In health economic modeling, to help anchor long-term survival extrapolation, it has been recommended that survival models incorporate background mortality rates using excess hazard (EH) methods.We present a thorough description of EH methods with and without the assumption of cure and demonstrate user-friendly software to aid researchers wishing to use these methods.EH models are applied to a case study, and we demonstrate that EHs are easier to extrapolate and that the use of the EH cure model, when cure is plausible, can reduce extrapolation variability.EH methods are relatively robust to lifetable misspecification.

Deep Learning Techniques with Genomic Data in Cancer Prognosis: A Comprehensive Review of the 2021-2023 Literature

Deep learning has brought about a significant transformation in machine learning, leading to an array of novel methodologies and consequently broadening its influence. The application of deep learning in various sectors, especially biomedical data analysis, has initiated a period filled with noteworthy scientific developments. This trend has majorly influenced cancer prognosis, where the interpretation of genomic data for survival analysis has become a central research focus. The capacity of deep learning to decode intricate patterns embedded within high-dimensional genomic data has provoked a paradigm shift in our understanding of cancer survival. Given the swift progression in this field, there is an urgent need for a comprehensive review that focuses on the most influential studies from 2021 to 2023. This review, through its careful selection and thorough exploration of dominant trends and methodologies, strives to fulfill this need. The paper aims to enhance our existing understanding of applications of deep learning in cancer survival analysis, while also highlighting promising directions for future research. This paper undertakes aims to enrich our existing grasp of the application of deep learning in cancer survival analysis, while concurrently shedding light on promising directions for future research in this vibrant and rapidly proliferating field.

An Investigation into the Relationship Between Choice of Model Structure and How to Adjust for Subsequent Therapies Using a Case Study in Oncology

BACKGROUND

A common challenge in health technology assessments (HTAs) of cancer treatments is how subsequent therapy use within the trial follow-up may influence cost-effectiveness model outcomes. Although overall survival (OS) is often a key driver of model results, there are no guidelines to advise how to adjust for this potential confounding, with different approaches available dependent on the model structure.

OBJECTIVE

We compared a partitioned survival analysis (PartSA) with a semi-Markov multi-state model (MSM) structure, with and without attempts to adjust for the impact of subsequent therapies on OS using a case study describing outcomes for people with relapsed/refractory multiple myeloma.

METHODS

Both model structures included three health states: pre-progression, progressed disease and death. Three traditional crossover methods were considered within the context of the PartSA, whereas for the MSM, the probability of post-progression death was pooled across arms. Impacts on the model incremental cost-effectiveness ratio (ICER) were recorded.

RESULTS

The unadjusted PartSA produced an ICER of £623,563, and after adjustment yielded an ICER range of £381,340-£386,907. The unadjusted MSM produced an ICER of £1,283,780. Adjusting OS in the MSM resulted in an ICER of £345,486.

CONCLUSIONS

The simplicity of the PartSA is lost when the decision problem becomes more complex (for example, when OS data are confounded by subsequent therapies). In this setting, the MSM structure may be considered more flexible, with fewer and less restrictive assumptions required versus the PartSA. Researchers should consider important study design features that may influence the generalisability of data when undertaking model conceptualisation.

A Systematic Review of Methods to Incorporate External Evidence into Trial-Based Survival Extrapolations for Health Technology Assessment

BACKGROUND

External evidence is commonly used to inform survival modeling for health technology assessment (HTA). While there are a range of methodological approaches that have been proposed, it is unclear which methods could be used and how they compare.

PURPOSE

This review aims to identify, describe, and categorize established methods to incorporate external evidence into survival extrapolation for HTA.

DATA SOURCES

Embase, MEDLINE, EconLit, and Web of Science databases were searched to identify published methodological studies, supplemented by hand searching and citation tracking.

STUDY SELECTION

Eligible studies were required to present a novel extrapolation approach incorporating external evidence (i.e., data or information) within survival model estimation.

DATA EXTRACTION

Studies were classified according to how the external evidence was integrated as a part of model fitting. Information was extracted concerning the model-fitting process, key requirements, assumptions, software, application contexts, and presentation of comparisons with, or validation against, other methods.

DATA SYNTHESIS

Across 18 methods identified from 22 studies, themes included use of informative prior(s) (n = 5), piecewise (n = 7), and general population adjustment (n = 9), plus a variety of "other" (n = 8) approaches. Most methods were applied in cancer populations (n = 13). No studies compared or validated their method against another method that also incorporated external evidence.

LIMITATIONS

As only studies with a specific methodological objective were included, methods proposed as part of another study type (e.g., an economic evaluation) were excluded from this review.

CONCLUSIONS

Several methods were identified in this review, with common themes based on typical data sources and analytical approaches. Of note, no evidence was found comparing the identified methods to one another, and so an assessment of different methods would be a useful area for further research.HighlightsThis review aims to identify methods that have been used to incorporate external evidence into survival extrapolations, focusing on those that may be used to inform health technology assessment.We found a range of different approaches, including piecewise methods, Bayesian methods using informative priors, and general population adjustment methods, as well as a variety of "other" approaches.No studies attempted to compare the performance of alternative methods for incorporating external evidence with respect to the accuracy of survival predictions. Further research investigating this would be valuable.

Integration of Chemoinformatics and Multi-Omics Analysis Defines ECT2 as a Potential Target for Cancer Drug Therapy

Epithelial cell transforming 2 (ECT2) is a potential oncogene and a number of recent studies have correlated it with the progression of several human cancers. Despite this elevated attention for ECT2 in oncology-related reports, there is no collective study to combine and integrate the expression and oncogenic behavior of ECT2 in a panel of human cancers. The current study started with a differential expression analysis of ECT2 in cancerous versus normal tissue. Following that, the study asked for the correlation between ECT2 upregulation and tumor stage, grade, and metastasis, along with its effect on patient survival. Moreover, the methylation and phosphorylation status of ECT2 in tumor versus normal tissue was assessed, in addition to the investigation of the ECT2 effect on the immune cell infiltration in the tumor microenvironment. The current study revealed that ECT2 was upregulated as mRNA and protein levels in a list of human tumors, a feature that allowed for the increased filtration of myeloid-derived suppressor cells (MDSC) and decreased the level of natural killer T (NKT) cells, which ultimately led to a poor prognosis survival. Lastly, we screened for several drugs that could inhibit ECT2 and act as antitumor agents. Collectively, this study nominated ECT2 as a prognostic and immunological biomarker, with reported inhibitors that represent potential antitumor drugs.

Blended Survival Curves: A New Approach to Extrapolation for Time-to-Event Outcomes from Clinical Trials in Health Technology Assessment

BACKGROUND

Survival extrapolation is essential in cost-effectiveness analysis to quantify the lifetime survival benefit associated with a new intervention, due to the restricted duration of randomized controlled trials (RCTs). Current approaches of extrapolation often assume that the treatment effect observed in the trial can continue indefinitely, which is unrealistic and may have a huge impact on decisions for resource allocation.

OBJECTIVE

We introduce a novel methodology as a possible solution to alleviate the problem of survival extrapolation with heavily censored data from clinical trials.

METHOD

The main idea is to mix a flexible model (e.g., Cox semiparametric) to fit as well as possible the observed data and a parametric model encoding assumptions on the expected behavior of underlying long-term survival. The two are "blended" into a single survival curve that is identical with the Cox model over the range of observed times and gradually approaching the parametric model over the extrapolation period based on a weight function. The weight function regulates the way two survival curves are blended, determining how the internal and external sources contribute to the estimated survival over time.

RESULTS

A 4-y follow-up RCT of rituximab in combination with fludarabine and cyclophosphamide versus fludarabine and cyclophosphamide alone for the first-line treatment of chronic lymphocytic leukemia is used to illustrate the method.

CONCLUSION

Long-term extrapolation from immature trial data may lead to significantly different estimates with various modelling assumptions. The blending approach provides sufficient flexibility, allowing a wide range of plausible scenarios to be considered as well as the inclusion of external information, based, for example, on hard data or expert opinion. Both internal and external validity can be carefully examined.

HIGHLIGHTS

Interim analyses of trials with limited follow-up are often subject to high degrees of administrative censoring, which may result in implausible long-term extrapolations using standard approaches.In this article, we present an innovative methodology based on "blending" survival curves to relax the traditional proportional hazard assumption and simultaneously incorporate external information to guide the extrapolation.The blended method provides a simple and powerful framework to allow a careful consideration of a wide range of plausible scenarios, accounting for model fit to the short-term data as well as the plausibility of long-term extrapolations.

Synthetic biology-inspired cell engineering in diagnosis, treatment, and drug development

The fast-developing synthetic biology (SB) has provided many genetic tools to reprogram and engineer cells for improved performance, novel functions, and diverse applications. Such cell engineering resources can play a critical role in the research and development of novel therapeutics. However, there are certain limitations and challenges in applying genetically engineered cells in clinical practice. This literature review updates the recent advances in biomedical applications, including diagnosis, treatment, and drug development, of SB-inspired cell engineering. It describes technologies and relevant examples in a clinical and experimental setup that may significantly impact the biomedicine field. At last, this review concludes the results with future directions to optimize the performances of synthetic gene circuits to regulate the therapeutic activities of cell-based tools in specific diseases.

Clinical effectiveness reporting of novel cancer drugs in the context of non-proportional hazards: a review of nice single technology appraisals

OBJECTIVES

The hazard ratio (HR) is a commonly used summary statistic when comparing time to event (TTE) data between trial arms, but assumes the presence of proportional hazards (PH). Non-proportional hazards (NPH) are increasingly common in NICE technology appraisals (TAs) due to an abundance of novel cancer treatments, which have differing mechanisms of action compared with traditional chemotherapies. The goal of this study is to understand how pharmaceutical companies, evidence review groups (ERGs) and appraisal committees (ACs) test for PH and report clinical effectiveness in the context of NPH.

METHODS

A thematic analysis of NICE TAs concerning novel cancer treatments published between 1 January 2020 and 31 December 2021 was undertaken. Data on PH testing and clinical effectiveness reporting for overall survival (OS) and progression-free survival (PFS) were obtained from company submissions, ERG reports, and final appraisal determinations (FADs).

RESULTS

NPH were present for OS or PFS in 28/40 appraisals, with log-cumulative hazard plots the most common testing methodology (40/40), supplemented by Schoenfeld residuals (20/40) and/or other statistical methods (6/40). In the context of NPH, the HR was ubiquitously reported by companies, inconsistently critiqued by ERGs (10/28), and commonly reported in FADs (23/28).

CONCLUSIONS

There is inconsistency in PH testing methodology used in TAs. ERGs are inconsistent in critiquing use of the HR in the context of NPH, and even when critiqued it remains a commonly reported outcome measure in FADs. Other measures of clinical effectiveness should be considered, along with guidance on clinical effectiveness reporting when NPH are present.

Economic evaluation of using polygenic risk score to guide risk screening and interventions for the prevention of type 2 diabetes in individuals with high overall baseline risk

Type 2 diabetes (T2D) with increasing prevalence is a significant global public health challenge. Obesity, unhealthy diet, and low physical activity are one of the major determinants of the rise in T2D prevalence. In addition, family history and genetic risk of diabetes also play a role in the process of developing T2D. Therefore, solutions for the early identification of individuals at high risk for T2D for early targeted detection of T2D, prevention, and intervention are highly preferred. Recently, novel genomic-based polygenic risk scores (PRSs) have been suggested to improve the accuracy of risk prediction supporting the targeting of preventive interventions to those at highest risk for T2D. Therefore, the aim of the present study was to assess the cost-utility of an additional PRS testing information (as a part of overall risk assessment) followed by a lifestyle intervention and an additional medical therapy when estimated 10-year overall risk for T2D exceeded 20% among Finnish individuals screened as at the high-risk category (i.e., 10%–20% 10-year overall risk of T2D) based on traditional risk factors only. For a cost-utility analysis, an individual-level state-transition model with probabilistic sensitivity analysis was constructed. A 1-year cycle length and a lifetime time horizon were applied in the base-case. A 3% discount rate was used for costs and QALYs. Cost-effectiveness acceptability curve (CEAC) and estimates for the expected value of perfect information (EVPI) were calculated to assist decision makers. The use of the targeted PRS strategy reclassified 12.4 percentage points of individuals to be very high-risk individuals who would have been originally classified as high risk using the usual strategy only. Over a lifetime horizon, the targeted PRS was a dominant strategy (i.e., less costly, more effective). One-way and scenario sensitivity analyses showed that results remained dominant in almost all simulations. However, there is uncertainty, since the probability (EVPI) of cost-effectiveness at a WTP of 0€/QALY was 63.0% (243€) indicating the probability that the PRS strategy is a dominant option. In conclusion, the results demonstrated that the PRS provides moderate additional value in Finnish population in risk screening leading to potential cost savings and better quality of life when compared with the current screening methods for T2D risk.

Informed Bayesian survival analysis

Background

We provide an overview of Bayesian estimation, hypothesis testing, and model-averaging and illustrate how they benefit parametric survival analysis. We contrast the Bayesian framework to the currently dominant frequentist approach and highlight advantages, such as seamless incorporation of historical data, continuous monitoring of evidence, and incorporating uncertainty about the true data generating process.

Methods

We illustrate the application of the outlined Bayesian approaches on an example data set, retrospective re-analyzing a colon cancer trial. We assess the performance of Bayesian parametric survival analysis and maximum likelihood survival models with AIC/BIC model selection in fixed-n and sequential designs with a simulation study.

Results

In the retrospective re-analysis of the example data set, the Bayesian framework provided evidence for the absence of a positive treatment effect of adding Cetuximab to FOLFOX6 regimen on disease-free survival in patients with resected stage III colon cancer. Furthermore, the Bayesian sequential analysis would have terminated the trial 10.3 months earlier than the standard frequentist analysis. In a simulation study with sequential designs, the Bayesian framework on average reached a decision in almost half the time required by the frequentist counterparts, while maintaining the same power, and an appropriate false-positive rate. Under model misspecification, the Bayesian framework resulted in higher false-negative rate compared to the frequentist counterparts, which resulted in a higher proportion of undecided trials. In fixed-n designs, the Bayesian framework showed slightly higher power, slightly elevated error rates, and lower bias and RMSE when estimating treatment effects in small samples. We found no noticeable differences for survival predictions. We have made the analytic approach readily available to other researchers in the RoBSA R package.

Conclusions

The outlined Bayesian framework provides several benefits when applied to parametric survival analyses. It uses data more efficiently, is capable of considerably shortening the length of clinical trials, and provides a richer set of inferences.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01676-9.

Dynamic and Flexible Survival Models for Extrapolation of Relative Survival: A Case Study and Simulation Study

BACKGROUND

Extrapolation of survival data is a key task in health technology assessments (HTAs), which may be improved by incorporating general population mortality data via relative survival models. Dynamic survival models are a promising method for extrapolation that may be expanded to dynamic relative survival models (DRSMs), a novel development presented here. There are currently neither examples of dynamic models in HTA nor comparisons of DRSMs with other relative survival models when used for survival extrapolation.

METHODS

An existing appraisal, for which there had been disagreement over the approach to survival extrapolation, was chosen and the health economic model recreated. The sensitivity of estimates of cost-effectiveness to different model choices (standard survival models, DSMs, and DRSMs) and specifications was examined. The appraisal informed a simulation study to evaluate DRSMs with relative survival models based on both standard and spline-based (flexible) models.

RESULTS

Dynamic models provided insight into the behavior of the trend in the hazard function and how it may vary during the extrapolated phase. DRSMs led to extrapolations with improved plausibility for which model choice may be based on clinical input. In the simulation study, the flexible and dynamic relative survival models performed similarly and provided highly variable extrapolations.

LIMITATIONS

Further experience with these models is required to identify settings when they are most useful, and they provide sufficiently accurate extrapolations.

CONCLUSIONS

Dynamic models provide a flexible and attractive method for extrapolating survival data and facilitate the use of clinical input for model choice. Flexible and dynamic relative survival models make few structural assumptions and can improve extrapolation plausibility, but further research is required into methods for reducing the variability in extrapolations.

A Review of Economic Models Submitted to NICE's Technology Appraisal Programme, for Treatments of T1DM & T2DM

Background: In the UK, 4.7 million people are currently living with diabetes. This is projected to increase to 5 million by 2025. The direct and indirect costs of T1DM and T2DM are rising, and direct costs already account for approximately 10% of the National Health Service (NHS) budget.

Objective: The aim of this review is to assess the economic models used in the context of NICE’s Technology Appraisals (TA) Programme of T1DM and T2DM treatments, as well as to examine their compliance with the American Diabetes Association’s (ADA) guidelines on computer modelling.

Methods: A review of the economic models used in NICE’s TA programme of T1DM and T2DM treatments was undertaken. Relevant TAs were identified through searching the NICE website for published appraisals completed up to April 2021. The review also examined the associated Evidence Review Group (ERG) reports and Final Appraisal Documents (FAD), which are publicly accessible. ERG reports were scrutinised to identify major issues pertaining to the economic modelling. The FAD documents were then examined to assess how these issues reflected on NICE recommendations.

Results: Overall, 10 TAs pertaining to treatments of T1DM and T2DM were identified. Two TAs were excluded as they did not use economic models. Seven of the 8 included TAs related to a novel class of oral antidiabetic drugs (OADs), gliflozins, and one to continuous subcutaneous insulin infusion (CSII) devices. There is a lack of recent, robust data informing risk equations to enable the derivation of transition probabilities. Despite uncertainty surrounding its clinical relevance, bodyweight/BMI is a key driver in many T2DM-models. HbA1c’s reliability as a predictor of hard outcomes is uncertain, chiefly for macrovascular complications. The external validity of T1DM is even less clear. There is an inevitable trade-off between the sophistication of models’ design, their transparency and practicality.

Conclusion: Economic models are essential tools to support decision-making in relation to market access and ascertain diabetes technologies’ cost effectiveness. However, key structural and methodological issues exist. Models’ shortcomings should be acknowledged and contextualised within the framework of technology appraisals. Diabetes medications and other technologies should also be subject to regular and consistent re-appraisal to inform disinvestment decisions. Artificial intelligence could potentially enhance models’ transparency and practicality.

Predicting Survival for Chimeric Antigen Receptor T-Cell Therapy: A Validation of Survival Models Using Follow-Up Data From ZUMA-1

OBJECTIVES

Survival extrapolation for chimeric antigen receptor T-cell therapies is challenging, owing to their unique mechanistic properties that translate to complex hazard functions. Axicabtagene ciloleucel is indicated for the treatment of relapse or refractory diffuse large B-cell lymphoma after 2 or more lines of therapy based on the ZUMA-1 trial. Four data snapshots are available, with minimum follow-up of 12, 24, 36, and 48 months. This analysis explores how survival extrapolations for axicabtagene ciloleucel using ZUMA-1 data can be validated and compared.

METHODS

Three different parametric modeling approaches were applied: standard parametric, spline-based, and cure-based models. Models were compared using a range of metrics, across the 4 data snapshot, including visual fit, plausibility of long-term estimates, statistical goodness of fit, inspection of hazard plots, point-estimate accuracy, and conditional survival estimates.

RESULTS

Standard and spline-based parametric extrapolations were generally incapable of fitting the ZUMA-1 data well. Cure-based models provided the best fit based on the earliest data snapshot, with extrapolations remaining consistent as data matured. At 48 months, the maximum survival overestimate was 8.3% (Gompertz mixture-cure model) versus the maximum underestimate of 33.5% (Weibull standard parametric model).

CONCLUSIONS

Where a plateau in the survival curve is clinically plausible, cure-based models may be helpful in making accurate predictions based on immature data. The ability to reliably extrapolate from maturing data may reduce delays in patient access to potentially lifesaving treatments. Additional research is required to understand how models compare in broader contexts, including different treatments and therapeutic areas.

Biased Survival Predictions When Appraising Health Technologies in Heterogeneous Populations

INTRODUCTION

Time-to-event data from clinical trials are routinely extrapolated using parametric models to estimate the cost effectiveness of novel therapies, but how this approach performs in the presence of heterogeneous populations remains unknown.

METHODS

We performed a simulation study of seven scenarios with varying exponential distributions modelling treatment and prognostic effects across subgroup and complement populations, with follow-up typical of clinical trials used to appraise the cost effectiveness of therapies by agencies such as the UK National Institute for Health and Care Excellence (NICE). We compared established and emerging methods of estimating population life-years (LYs) using parametric models. We also proved analytically that an exponential model fitted to censored heterogeneous survival times sampled from two distinct exponential distributions will produce a biased estimate of the hazard rate and LYs.

RESULTS

LYs are underestimated by the methods in the presence of heterogeneity, resulting in either under- or overestimation of the incremental benefit. In scenarios where the overestimation of benefit is likely, which is of interest to the healthcare provider, the method of taking the average LYs from all plausible models has the least bias. LY estimates from complete Kaplan-Meier curves have high variation, suggesting mature data may not be a reliable solution. We explore the effect of increasing trial sample size and accounting for detected treatment-subgroup interactions.

CONCLUSIONS

The bias associated with heterogeneous populations suggests that NICE may need to be more cautious when appraising therapies and to consider model averaging or the separate modelling of subgroups when heterogeneity is suspected or detected.

Impact of limited sample size and follow-up on single event survival extrapolation for health technology assessment: a simulation study

INTRODUCTION

Extrapolation of time-to-event data from clinical trials is commonly used in decision models for health technology assessment (HTA). The objective of this study was to assess performance of standard parametric survival analysis techniques for extrapolation of time-to-event data for a single event from clinical trials with limited data due to small samples or short follow-up.

METHODS

Simulated populations with 50,000 individuals were generated with an exponential hazard rate for the event of interest. A scenario consisted of 5000 repetitions with six sample size groups (30-500 patients) artificially censored after every 10% of events observed. Goodness-of-fit statistics (AIC, BIC) were used to determine the best-fitting among standard parametric distributions (exponential, Weibull, log-normal, log-logistic, generalized gamma, Gompertz). Median survival, one-year survival probability, time horizon (1% survival time, or 99th percentile of survival distribution) and restricted mean survival time (RMST) were compared to population values to assess coverage and error (e.g., mean absolute percentage error).

RESULTS

The true exponential distribution was correctly identified using goodness-of-fit according to BIC more frequently compared to AIC (average 92% vs 68%). Under-coverage and large errors were observed for all outcomes when distributions were specified by AIC and for time horizon and RMST with BIC. Error in point estimates were found to be strongly associated with sample size and completeness of follow-up. Small samples produced larger average error, even with complete follow-up, than large samples with short follow-up. Correctly specifying the event distribution reduced magnitude of error in larger samples but not in smaller samples.

CONCLUSIONS

Limited clinical data from small samples, or short follow-up of large samples, produce large error in estimates relevant to HTA regardless of whether the correct distribution is specified. The associated uncertainty in estimated parameters may not capture the true population values. Decision models that base lifetime time horizon on the model's extrapolated output are not likely to reliably estimate mean survival or its uncertainty. For data with an exponential event distribution, BIC more reliably identified the true distribution than AIC. These findings have important implications for health decision modelling and HTA of novel therapies seeking approval with limited evidence.

The Extrapolation Performance of Survival Models for Data With a Cure Fraction: A Simulation Study

OBJECTIVES

Curative treatments can result in complex hazard functions. The use of standard survival models may result in poor extrapolations. Several models for data which may have a cure fraction are available, but comparisons of their extrapolation performance are lacking. A simulation study was performed to assess the performance of models with and without a cure fraction when fit to data with a cure fraction.

METHODS

Data were simulated from a Weibull cure model, with 9 scenarios corresponding to different lengths of follow-up and sample sizes. Cure and noncure versions of standard parametric, Royston-Parmar, and dynamic survival models were considered along with noncure fractional polynomial and generalized additive models. The mean-squared error and bias in estimates of the hazard function were estimated.

RESULTS

With the shortest follow-up, none of the cure models provided good extrapolations. Performance improved with increasing follow-up, except for the misspecified standard parametric cure model (lognormal). The performance of the flexible cure models was similar to that of the correctly specified cure model. Accurate estimates of the cured fraction were not necessary for accurate hazard estimates. Models without a cure fraction provided markedly worse extrapolations.

CONCLUSIONS

For curative treatments, failure to model the cured fraction can lead to very poor extrapolations. Cure models provide improved extrapolations, but with immature data there may be insufficient evidence to choose between cure and noncure models, emphasizing the importance of clinical knowledge for model choice. Dynamic cure fraction models were robust to model misspecification, but standard parametric cure models were not.

Finite Mixture Models, a Flexible Alternative to Standard Modeling Techniques for Extrapolated Mean Survival Times Needed for Cost-Effectiveness Analyses

OBJECTIVES

To compare finite mixture models with common survival models with respect to how well they fit heterogenous data used to estimate mean survival times required for cost-effectiveness analysis.

METHODS

Publicly available overall survival (OS) and progression-free survival (PFS) curves were digitized to produce nonproprietary data. Regression models based on the following distributions were fit to the data: Weibull, lognormal, log-logistic, generalized F, generalized gamma, Gompertz, mixture of 2 Weibulls, and mixture of 3 Weibulls. A second set of analyses was performed based on data in which patients who had not experienced an event by 30 months were censored. Model performance was compared based on the Akaike information criterion (AIC).

RESULTS

For PFS, the 3-Weibull mixture (AIC = 479.94) and 2-Weibull mixture (AIC = 488.24) models outperformed other models by more than 40 points and produced the most accurate estimates of mean survival times. For OS, the AIC values for all models were similar (all within 4 points). The means for the mixture 3-Weibulls mixture model (17.60 months) and the 2-Weibull mixture model (17.59 months) were the closest to the Kaplan-Meier mean estimate of (17.58 months). The results and conclusions from the censored analysis of PFS were similar to the uncensored PFS analysis. On the basis of extrapolated mean OS, all models produced estimates within 10% of the Kaplan-Meier mean survival time.

CONCLUSIONS

Finite mixture models offer a flexible modeling approach that has benefits over standard parametric models when analyzing heterogenous data for estimating survival times needed for cost-effectiveness analysis.

Onwards and Upwards: A Systematic Survey of Economic Evaluation Methods in Oncology

INTRODUCTION

The type of methods used in economic evaluations of health technology can lead to results that may influence decisions. Despite the potential impact on decision making, there is very little documentation of methods used in economic evaluation in oncology pertaining to key assumptions and extrapolation methods of survival benefits, especially in terms of survival analysis techniques and methods for extrapolation.

OBJECTIVES

The primary objectives of this study were to identify, examine, and describe the methods used in economic evaluations in oncology over a 10-year period, while secondary objectives included examining the use of identified methods across different geographic regions.

METHODS

A systematic search of the published oncology literature was conducted to identify economic evaluations of advanced or metastatic cancers published between 2010 and 2019 using the PUBMED, Ovid MEDLINE, and EMBASE databases. A random sample was taken, and information on type of study, data source, modeling techniques, and survival analysis methods were abstracted and descriptively summarized.

RESULTS

A total of 8481 abstracts were identified and 76 economic evaluations were abstracted and assessed. Most identified studies were from North America (38%), East Asia (21%), continental Europe (18%), or the UK (16%), and most commonly focused on lung cancer (18%), colorectal cancer (16%), or breast cancer (13%). A large majority of studies were based on data from randomized controlled trials (82%), utilized a cost-utility approach (82%), and took a public healthcare system perspective (83%). Common model structures included Markov (49%) and partitioned survival (17%). Fitted parametric curves were the most commonly used extrapolation method (89%) for overall survival and most often utilized the Weibull distribution (64%). Secondary assessments showed modest regional variation in the use of identified methods, including the use of fitted parametric curves, testing of the proportional hazards assumption, and validation of results.

CONCLUSION

A majority of papers in the study sample reported basic characteristics of study type, data source used, modeling techniques, and utilization of survival analysis methods. However, greater detail in reporting extrapolation methods, statistical analyses, and validation of results could be potential improvements, especially across regions, in order to support greater consistency in decision making. Future research could document the diffusion of novel modeling techniques into economic evaluation.

Extrapolating Parametric Survival Models in Health Technology Assessment Using Model Averaging: A Simulation Study

Previous work examined the suitability of relying on routine methods of model selection when extrapolating survival data in a health technology appraisal setting. Here we explore solutions to improve reliability of restricted mean survival time (RMST) estimates from trial data by assessing model plausibility and implementing model averaging. We compare our previous methods of selecting a model for extrapolation using the Akaike information criterion (AIC) and Bayesian information criterion (BIC). Our methods of model averaging include using equal weighting across models falling within established threshold ranges for AIC and BIC and using BIC-based weighted averages. We apply our plausibility assessment and implement model averaging to the output of our previous simulations, where 10,000 runs of 12 trial-based scenarios were examined. We demonstrate that removing implausible models from consideration reduces the mean squared error associated with the restricted mean survival time (RMST) estimate from each selection method and increases the percentage of RMST estimates that were within 10% of the RMST from the parameters of the sampling distribution. The methods of averaging were superior to selecting a single optimal extrapolation, aside from some of the exponential scenarios where BIC already selected the exponential model. The averaging methods with wide criterion-based thresholds outperformed BIC-weighted averaging in the majority of scenarios. We conclude that model averaging approaches should feature more widely in the appraisal of health technologies where extrapolation is influential and considerable uncertainty is present. Where data demonstrate complicated underlying hazard rates, funders should account for the additional uncertainty associated with these extrapolations in their decision making. Extended follow-up from trials should be encouraged and used to review prices of therapies to ensure a fair price is paid.

Extrapolating Parametric Survival Models in Health Technology Assessment: A Simulation Study

Extrapolations of parametric survival models fitted to censored data are routinely used in the assessment of health technologies to estimate mean survival, particularly in diseases that potentially reduce the life expectancy of patients. Akaike’s information criterion (AIC) and Bayesian information criterion (BIC) are commonly used in health technology assessment alongside an assessment of plausibility to determine which statistical model best fits the data and should be used for prediction of long-term treatment effects. We compare fit and estimates of restricted mean survival time (RMST) from 8 parametric models and contrast models preferred in terms of AIC, BIC, and log-likelihood, without considering model plausibility. We assess the methods’ suitability for selecting a parametric model through simulation of data replicating the follow-up of intervention arms for various time-to-event outcomes from 4 clinical trials. Follow-up was replicated through the consideration of recruitment duration and minimum and maximum follow-up times. Ten thousand simulations of each scenario were performed. We demonstrate that the different methods can result in disagreement over the best model and that it is inappropriate to base model selection solely on goodness-of-fit statistics without consideration of hazard behavior and plausibility of extrapolations. We show that typical trial follow-up can be unsuitable for extrapolation, resulting in unreliable estimation of multiple parameter models, and infer that selecting survival models based only on goodness-of-fit statistics is unsuitable due to the high level of uncertainty in a cost-effectiveness analysis. This article demonstrates the potential problems of overreliance on goodness-of-fit statistics when selecting a model for extrapolation. When follow-up is more mature, BIC appears superior to the other selection methods, selecting models with the most accurate and least biased estimates of RMST.

Partitioned Survival and State Transition Models for Healthcare Decision Making in Oncology: Where Are We Now?

OBJECTIVES

Partitioned survival models (PSMs) are routinely used to inform reimbursement decisions for oncology drugs. We discuss the appropriateness of PSMs compared to the most common alternative, state transition models (STMs).

METHODS

In 2017, we published a National Institute for Health and Care Excellence (NICE) Technical Support Document (TSD 19) describing and critically reviewing PSMs. This article summarizes findings from TSD 19, reviews new evidence comparing PSMs and STMs, and reviews recent NICE appraisals to understand current practice.

RESULTS

PSMs evaluate state membership differently from STMs and do not include a structural link between intermediate clinical endpoints (eg, disease progression) and survival. PSMs directly consider clinical trial endpoints and can be developed without access to individual patient data, but limit the scope for sensitivity analyses to explore clinical uncertainties in the extrapolation period. STMs facilitate these sensitivity analyses but require development of robust survival models for individual health-state transitions. Recent work has shown PSMs and STMs can produce substantively different survival extrapolations and that extrapolations from STMs are heavily influenced by specification of the underlying survival models. Recent NICE appraisals have not generally included both model types, reviewed individual clinical event data, or scrutinized life-years accrued in individual health states.

CONCLUSIONS

The credibility of survival predictions from PSMs and STMs, including life-years accrued in individual health states, should be assessed using trial data on individual clinical events, external data, and expert opinion. STMs should be used alongside PSMs to support assessment of clinical uncertainties in the extrapolation period, such as uncertainty in post-progression survival.