Using causal forests to assess heterogeneity in cost-effectiveness analysis

Understanding the impact of radiotherapy fractionation on overall survival in a large head and neck squamous cell carcinoma dataset: a comprehensive approach combining mechanistic and machine learning models

Introduction

Treating head and neck squamous cell carcinomas (HNSCC), especially human papillomavirus negative (HPV-) and locally advanced cases, remains difficult. Our previous analyses of radiotherapy-only HNSCC clinical trials data using mechanistically-motivated models of tumor repopulation and killing by radiotherapy predicted that hyperfractionation with twice-daily fractions, or hypofractionation involving increased doses/fraction and reduced treatment durations, both improve tumor control and reduce late normal tissue toxicity, compared with standard protocols using 35×2 Gy. Here we further investigated the validity of these conclusions by analyzing a large modern dataset on 3,346 HNSCC radiotherapy patients from the University Health Network in Toronto, Canada, where 42.5% of patients were also treated with chemotherapy.

Methods

We used a two-step approach that combines mechanistic modeling concepts with state-of-the-art machine learning, beginning with Random Survival Forests (RSF) for an exploratory analysis and followed by Causal Survival Forests (CSF) for a focused causal analysis. The mechanistic concept of biologically effective dose (BED) was implemented for the standard dose-independent (DI) tumor repopulation model, our alternative dose-dependent (DD) repopulation model, and a simple model with no repopulation (BEDsimp). These BED variants were included in the RSF model, along with age, stage, HPV status and other relevant variables, to predict patient overall survival (OS) and cause-specific mortality (deaths from the index cancer, other cancers or other causes).

Results

Model interpretation using Shapley Additive Explanations (SHAP) values and correlation matrices showed that high values of BEDDD or BEDDI, but not BEDsimp, were associated with decreased patient mortality. Targeted causal inference analyses were then performed using CSF to estimate the causal effect of each BED variant on OS. They revealed that high BEDDD (>61.8 Gy) or BEDDI (>57.6 Gy), but not BEDsimp, increased patient restricted mean survival time (RMST) by 0.5-1.0 years and increased survival probability (SP) by 5-15% several years after treatment. In addition to population-level averages, CSF generated individual-level causal effect estimates for each patient, facilitating personalized medicine.

Discussion

These findings are generally consistent with those of our previous mechanistic modeling, implying the potential benefits of altered radiotherapy fractionation schemes (e.g. 25×2.4 Gy, 20×2.75 Gy, 18×3.0 Gy) which increase BEDDD and BEDDI and counteract tumor repopulation more effectively than standard fractionation. Such regimens may represent potentially useful hypofractionated options for treating HNSCC.

Integrating decision modeling and machine learning to inform treatment stratification

There is increasing interest in moving away from "one size fits all (OSFA)" approaches toward stratifying treatment decisions. Understanding how expected effectiveness and cost-effectiveness varies with patient covariates is a key aspect of stratified decision making. Recently proposed machine learning (ML) methods can learn heterogeneity in outcomes without pre-specifying subgroups or functional forms, enabling the construction of decision rules ('policies') that map individual covariates into a treatment decision. However, these methods do not yet integrate ML estimates into a decision modeling framework in order to reflect long-term policy-relevant outcomes and synthesize information from multiple sources. In this paper, we propose a method to integrate ML and decision modeling, when individual patient data is available to estimate treatment-specific survival time. We also propose a novel implementation of policy tree algorithms to define subgroups using decision model output. We demonstrate these methods using the SPRINT (Systolic Blood Pressure Intervention Trial), comparing outcomes for "standard" and "intensive" blood pressure targets. We find that including ML into a decision model can impact the estimate of incremental net health benefit (INHB) for OSFA policies. We also find evidence that stratifying treatment using subgroups defined by a tree-based algorithm can increase the estimates of the INHB.

Exploring Heterogeneity in Cost-Effectiveness Using Machine Learning Methods: A Case Study Using the FIRST-ABC Trial

OBJECTIVE

The aim of this study was to explore heterogeneity in the cost-effectiveness of high-flow nasal cannula (HFNC) therapy compared with continuous positive airway pressure (CPAP) in children following extubation.

DESIGN

Using data from the FIRST-line support for Assistance in Breathing in Children (FIRST-ABC) trial, we explore heterogeneity at the individual and subgroup levels using a causal forest approach, alongside a seemingly unrelated regression (SUR) approach for comparison.

SETTINGS

FIRST-ABC is a noninferiority randomized controlled trial (ISRCTN60048867) including children in UK paediatric intensive care units, which compared HFNC with CPAP as the first-line mode of noninvasive respiratory support.

PATIENTS

In the step-down FIRST-ABC, 600 children clinically assessed to require noninvasive respiratory support were randomly assigned to HFNC and CPAP groups with 1:1 treatment allocation ratio. In this analysis, 118 patients were excluded because they did not consent to accessing their medical records, did not consent to follow-up questionnaire or did not receive respiratory support.

MEASUREMENTS AND MAIN RESULTS

The primary outcome of this study is the incremental net monetary benefit (INB) of HFNC compared with CPAP using a willingness-to-pay threshold of £20,000 per QALY gain. INB is calculated based on total costs and quality adjusted life years (QALYs) at 6 months. The findings suggest modest heterogeneity in cost-effectiveness of HFNC compared with CPAP at the subgroup level, while greater heterogeneity is detected at the individual level.

CONCLUSIONS

The estimated overall INB of HFNC is smaller than the INB for patients with better baseline status suggesting that HFNC can be more cost-effective among less severely ill patients.

Advances in Addressing Patient Heterogeneity in Economic Evaluation: A Review of the Methods Literature

Cost-effectiveness analyses commonly use population or sample averages, which can mask key differences across subgroups and may lead to suboptimal resource allocation. Despite there being several new methods developed over the last decade, there is no recent summary of what methods are available to researchers. This review sought to identify advances in methods for addressing patient heterogeneity in economic evaluations and to provide an overview of these methods. A literature search was conducted using the Econlit, Embase and MEDLINE databases to identify studies published after 2011 (date of a previous review on this topic). Eligible studies needed to have an explicit methodological focus, related to how patient heterogeneity can be accounted for within a full economic evaluation. Sixteen studies were included in the review. Methodologies were varied and included regression techniques, model design and value of information analysis. Recent publications have applied methodologies more commonly used in other fields, such as machine learning and causal forests. Commonly noted challenges associated with considering patient heterogeneity included data availability (e.g., sample size), statistical issues (e.g., risk of false positives) and practical factors (e.g., computation time). A range of methods are available to address patient heterogeneity in economic evaluation, with relevant methods differing according to research question, scope of the economic evaluation and data availability. Researchers need to be aware of the challenges associated with addressing patient heterogeneity (e.g., data availability) to ensure findings are meaningful and robust. Future research is needed to assess whether and how methods are being applied in practice.

Are there patients with an intracapsular fracture of the hip who may benefit from an uncemented hemiarthroplasty?

Aims

Cemented hemiarthroplasty is an effective form of treatment for most patients with an intracapsular fracture of the hip. However, it remains unclear whether there are subgroups of patients who may benefit from the alternative operation of a modern uncemented hemiarthroplasty - the aim of this study was to investigate this issue. Knowledge about the heterogeneity of treatment effects is important for surgeons in order to target operations towards specific subgroups who would benefit the most.

Methods

We used causal forest analysis to compare subgroup- and individual-level treatment effects between cemented and modern uncemented hemiarthroplasty in patients aged > 60 years with an intracapsular fracture of the hip, using data from the World Hip Trauma Evaluation 5 (WHiTE 5) multicentre randomized clinical trial. EuroQol five-dimension index scores were used to measure health-related quality of life at one, four, and 12 months postoperatively.

Results

Our analysis revealed a complex landscape of responses to the use of a cemented hemiarthroplasty in the 12 months after surgery. There was heterogeneity of effects with regard to baseline characteristics, including age, pre-injury health status, and lifestyle factors such as alcohol consumption. This heterogeneity was greater at the one-month mark than at subsequent follow-up timepoints, with particular regard to subgroups based on age. However, for all subgroups, the effect estimates for quality of life lay within the confidence intervals derived from the analysis of all patients.

Conclusion

The use of a cemented hemiarthroplasty is expected to increase health-related quality of life compared with modern uncemented hemiarthroplasty for all subgroups of patients aged > 60 years with a displaced intracapsular fracture of the hip.

An economic evaluation of chronic obstructive pulmonary disease clinical pathway in Saskatchewan, Canada: Data-driven techniques to identify cost-effectiveness among patient subgroups

BACKGROUND

Saskatchewan has implemented care pathways for several common health conditions. To date, there has not been any cost-effectiveness evaluation of care pathways in the province. The objective of this study was to evaluate the real-world cost-effectiveness of a chronic obstructive pulmonary disease (COPD) care pathway program in Saskatchewan.

METHODS

Using patient-level administrative health data, we identified adults (35+ years) with COPD diagnosis recruited into the care pathway program in Regina between April 1, 2018, and March 31, 2019 (N = 759). The control group comprised adults (35+ years) with COPD who lived in Saskatoon during the same period (N = 759). The control group was matched to the intervention group using propensity scores. Costs were calculated at the patient level. The outcome measure was the number of days patients remained without experiencing COPD exacerbation within 1-year follow-up. Both manual and data-driven policy learning approaches were used to assess heterogeneity in the cost-effectiveness by patient demographic and disease characteristics. Bootstrapping was used to quantify uncertainty in the results.

RESULTS

In the overall sample, the estimates indicate that the COPD care pathway was not cost-effective using the willingness to pay (WTP) threshold values in the range of $1,000 and $5,000/exacerbation day averted. The manual subgroup analyses show the COPD care pathway was dominant among patients with comorbidities and among patients aged 65 years or younger at the WTP threshold of $2000/exacerbation day averted. Although similar profiles as those identified in the manual subgroup analyses were confirmed, the data-driven policy learning approach suggests more nuanced demographic and disease profiles that the care pathway would be most appropriate for.

CONCLUSIONS

Both manual subgroup analysis and data-driven policy learning approach showed that the COPD care pathway consistently produced cost savings and better health outcomes among patients with comorbidities or among those relatively younger. The care pathway was not cost-effective in the entire sample.

Exploring treatment effect heterogeneity of a PROMs alert intervention in knee and hip arthroplasty patients: A causal forest application

Patient reported outcome measures (PROMs) experience an uptake in use for hip (HA) and knee arthroplasty (KA) patients. As they may be used for patient monitoring interventions, it remains unclear whether their use in HA/KA patients is effective, and which patient groups benefit the most. Nonetheless, knowledge about treatment effect heterogeneity is crucial for decision makers to target interventions towards specific subgroups that benefit to a greater extend. Therefore, we evaluate the treatment effect heterogeneity of a remote PROM monitoring intervention that includes ∼8000 HA/KA patients from a randomized controlled trial conducted in nine German hospitals. The study setting gave us the unique opportunity to apply a causal forest, a recently developed machine learning method, to explore treatment effect heterogeneity of the intervention. We found that among both HA and KA patients, the intervention was especially effective for patients that were female, >65 years of age, had a blood pressure disease, were not working, reported no backpain and were adherent. When transferring the study design into standard care, policy makers should make use of the knowledge obtained in this study and allocate the treatment towards subgroups for which the treatment is especially effective.

Analysis of causal effects of 137Cs deposition on 137Cs concentrations in trees after the Fukushima accident using machine learning

Radioactive contamination of forests by long-lived radionuclides from nuclear accidents such as Chernobyl and Fukushima continues to be studied and quantitatively modeled. Whereas traditional statistical and machine learning (ML) techniques generate predictions by focusing on correlations between variables, quantification of causal effects of radioactivity deposition levels on contamination of plant tissues represents a more fundamental and relevant research goal. Modeling of cause-and-effect relationships is advantageous over standard predictive modeling, particularly by improving the generalizability of results to other situations, where the distributions of variables, including potential confounders, differ from those in the training data. Here we used the state-of-the-art causal forest (CF) algorithm to quantify the causal effect of ¹³⁷Cs land contamination after the Fukushima accident on ¹³⁷Cs activity concentrations in the wood of four common Japanese forest tree species: Hinoki cypress (Chamaecyparis obtusa), konara oak (Quercus serrata), red pine (Pinus densiflora), and Sugi cedar (Cryptomeria japonica). We estimated the average causal effect for the population, quantified how it was influenced by other environmental variables, and produced effect estimates at the individual level. The estimated causal effect was quite robust to various refutation methods, and was negatively influenced by high mean annual precipitation, elevation, and time after the accident. Wood subtype (e.g. sapwood, heartwood) and tree species made smaller contributions to the causal effect. We believe that causal ML techniques have promising potential in radiation ecology and can usefully expand the toolkit of modeling approaches available to researchers in this field.

Machine learning clinical decision support for interdisciplinary multimodal chronic musculoskeletal pain treatment

Introduction

Chronic musculoskeletal pain is a prevalent condition impacting around 20% of people globally; resulting in patients living with pain, fatigue, restricted social and employment capacity, and reduced quality of life. Interdisciplinary multimodal pain treatment programs have been shown to provide positive outcomes by supporting patients modify their behavior and improve pain management through focusing attention on specific patient valued goals rather than fighting pain.

Methods

Given the complex nature of chronic pain there is no single clinical measure to assess outcomes from multimodal pain programs. Using Centre for Integral Rehabilitation data from 2019-2021 (n = 2,364), we developed a multidimensional machine learning framework of 13 outcome measures across 5 clinically relevant domains including activity/disability, pain, fatigue, coping and quality of life. Machine learning models for each endpoint were separately trained using the most important 30 of 55 demographic and baseline variables based on minimum redundancy maximum relevance feature selection. Five-fold cross validation identified best performing algorithms which were rerun on deidentified source data to verify prognostic accuracy.

Results

Individual algorithm performance ranged from 0.49 to 0.65 AUC reflecting characteristic outcome variation across patients, and unbalanced training data with high positive proportions of up to 86% for some measures. As expected, no single outcome provided a reliable indicator, however the complete set of algorithms established a stratified prognostic patient profile. Patient level validation achieved consistent prognostic assessment of outcomes for 75.3% of the study group (n = 1,953). Clinician review of a sample of predicted negative patients (n = 81) independently confirmed algorithm accuracy and suggests the prognostic profile is potentially valuable for patient selection and goal setting.

Discussion

These results indicate that although no single algorithm was individually conclusive, the complete stratified profile consistently identified patient outcomes. Our predictive profile provides promising positive contribution for clinicians and patients to assist with personalized assessment and goal setting, program engagement and improved patient outcomes.

Assessing the ability of an instrumental variable causal forest algorithm to personalize treatment evidence using observational data: the case of early surgery for shoulder fracture

Background

Comparative effectiveness research (CER) using observational databases has been suggested to obtain personalized evidence of treatment effectiveness. Inferential difficulties remain using traditional CER approaches especially related to designating patients to reference classes a priori. A novel Instrumental Variable Causal Forest Algorithm (IV-CFA) has the potential to provide personalized evidence using observational data without designating reference classes a priori, but the consistency of the evidence when varying key algorithm parameters remains unclear. We investigated the consistency of IV-CFA estimates through application to a database of Medicare beneficiaries with proximal humerus fractures (PHFs) that previously revealed heterogeneity in the effects of early surgery using instrumental variable estimators.

Methods

IV-CFA was used to estimate patient-specific early surgery effects on both beneficial and detrimental outcomes using different combinations of algorithm parameters and estimate variation was assessed for a population of 72,751 fee-for-service Medicare beneficiaries with PHFs in 2011. Classification and regression trees (CART) were applied to these estimates to create ex-post reference classes and the consistency of these classes were assessed. Two-stage least squares (2SLS) estimators were applied to representative ex-post reference classes to scrutinize the estimates relative to known 2SLS properties.

Results

IV-CFA uncovered substantial early surgery effect heterogeneity across PHF patients, but estimates for individual patients varied with algorithm parameters. CART applied to these estimates revealed ex-post reference classes consistent across algorithm parameters. 2SLS estimates showed that ex-post reference classes containing older, frailer patients with more comorbidities, and lower utilizers of healthcare were less likely to benefit and more likely to have detriments from higher rates of early surgery.

Conclusions

IV-CFA provides an illuminating method to uncover ex-post reference classes of patients based on treatment effects using observational data with a strong instrumental variable. Interpretation of treatment effect estimates within each ex-post reference class using traditional CER methods remains conditional on the extent of measured information in the data.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01663-0.