Generalizability of heterogeneous treatment effects based on causal forests applied to two randomized clinical trials of intensive glycemic control

Machine learning approaches to evaluate heterogeneous treatment effects in randomized controlled trials: a scoping review

BACKGROUND AND OBJECTIVES

Estimating heterogeneous treatment effects (HTEs) in randomized controlled trials (RCTs) has received substantial attention recently. This has led to the development of several statistical and machine learning (ML) algorithms to assess HTEs through identifying individualized treatment effects. However, a comprehensive review of these algorithms is lacking. We thus aimed to catalog and outline currently available statistical and ML methods for identifying HTEs via effect modeling using clinical RCT data and summarize how they have been applied in practice.

STUDY DESIGN AND SETTING

We performed a scoping review using prespecified search terms in MEDLINE and Embase, aiming to identify studies that assessed HTEs using advanced statistical and ML methods in RCT data published from 2010 to 2022.

RESULTS

Among a total of 32 studies identified in the review, 17 studies applied existing algorithms to RCT data, and 15 extended existing algorithms or proposed new algorithms. Applied algorithms included penalized regression, causal forest, Bayesian causal forest, and other metalearner frameworks. Of these methods, causal forest was the most frequently used (7 studies) followed by Bayesian causal forest (4 studies). Most applications were in cardiology (6 studies), followed by psychiatry (4 studies). We provide example R codes in simulated data to illustrate how to implement these algorithms.

CONCLUSION

This review identified and outlined various algorithms currently used to identify HTEs and individualized treatment effects in RCT data. Given the increasing availability of new algorithms, analysts should carefully select them after examining model performance and considering how the models will be used in practice.

A Novel Digital Twin Strategy to Examine the Implications of Randomized Clinical Trials for Real-World Populations

Randomized clinical trials (RCTs) are essential to guide medical practice; however, their generalizability to a given population is often uncertain. We developed a statistically informed Generative Adversarial Network (GAN) model, RCT-Twin-GAN, that leverages relationships between covariates and outcomes and generates a digital twin of an RCT (RCT-Twin) conditioned on covariate distributions from a second patient population. We used RCT-Twin-GAN to reproduce treatment effect outcomes of the Systolic Blood Pressure Intervention Trial (SPRINT) and the Action to Control Cardiovascular Risk in Diabetes (ACCORD) Blood Pressure Trial, which tested the same intervention but found different treatment effects. To demonstrate treatment effect estimates of each RCT conditioned on the other RCT's patient population, we evaluated the cardiovascular event-free survival of SPRINT digital twins conditioned on the ACCORD cohort and vice versa (ACCORD twins conditioned on SPRINT). The conditioned digital twins were balanced across intervention and control arms (mean absolute standardized mean difference (MASMD) of covariates between treatment arms 0.019 (SD 0.018), and the conditioned covariates of the SPRINT-Twin on ACCORD were more similar to ACCORD than SPRINT (MASMD 0.0082 SD 0.016 vs. 0.46 SD 0.20). Notably, across iterations, SPRINT conditioned ACCORD-Twin datasets reproduced the overall non-significant effect size seen in ACCORD (5-year cardiovascular outcome hazard ratio (95% confidence interval) of 0.88 (0.73-1.06) in ACCORD vs. median 0.87 (0.68-1.13) in the SPRINT conditioned ACCORD-Twin), while the ACCORD conditioned SPRINT-Twins reproduced the significant effect size seen in SPRINT (0.75 (0.64-0.89) vs. median 0.79 (0.72-0.86)) in the ACCORD conditioned SPRINT-Twin). Finally, we demonstrate the translation of this approach to real-world populations by conditioning the trials on an electronic health record population. Therefore, RCT-Twin-GAN simulates the direct translation of RCT-derived treatment effects across various patient populations.

Toward Precision in Critical Care Research: Methods for Observational and Interventional Studies

Critical care trials evaluate the effect of interventions in patients with diverse personal histories and causes of illness, often under the umbrella of heterogeneous clinical syndromes, such as sepsis or acute respiratory distress syndrome. Given this variation, it is reasonable to expect that the effect of treatment on outcomes may differ for individuals with variable characteristics. However, in randomized controlled trials, efficacy is typically assessed by the average treatment effect (ATE), which quantifies the average effect of the intervention on the outcome in the study population. Importantly, the ATE may hide variations of the treatment's effect on a clinical outcome across levels of patient characteristics, which may erroneously lead to the conclusion that an intervention does not work overall when it may in fact benefit certain patients. In this review, we describe methodological approaches for assessing heterogeneity of treatment effect (HTE), including expert-derived subgrouping, data-driven subgrouping, baseline risk modeling, treatment effect modeling, and individual treatment rule estimation. Next, we outline how insights from HTE analyses can be incorporated into the design of clinical trials. Finally, we propose a research agenda for advancing the field and bringing HTE approaches to the bedside.

A multimodal approach using fundus images and text meta-data in a machine learning classifier with embeddings to predict years with self-reported diabetes - An exploratory analysis

AIMS

Machine learning models can use image and text data to predict the number of years since diabetes diagnosis; such model can be applied to new patients to predict, approximately, how long the new patient may have lived with diabetes unknowingly. We aimed to develop a model to predict self-reported diabetes duration.

METHODS

We used the Brazilian Multilabel Ophthalmological Dataset. Unit of analysis was the fundus image and its meta-data, regardless of the patient. We included people 40 + years and fundus images without diabetic retinopathy. Fundus images and meta-data (sex, age, comorbidities and taking insulin) were passed to the MedCLIP model to extract the embedding representation. The embedding representation was passed to an Extra Tree Classifier to predict: 0-4, 5-9, 10-14 and 15 + years with self-reported diabetes.

RESULTS

There were 988 images from 563 people (mean age = 67 years; 64 % were women). Overall, the F1 score was 57 %. The group 15 + years of self-reported diabetes had the highest precision (64 %) and F1 score (63 %), while the highest recall (69 %) was observed in the group 0-4 years. The proportion of correctly classified observations was 55 % for the group 0-4 years, 51 % for 5-9 years, 58 % for 10-14 years, and 64 % for 15 + years with self-reported diabetes.

CONCLUSIONS

The machine learning model had acceptable accuracy and F1 score, and correctly classified more than half of the patients according to diabetes duration. Using large foundational models to extract image and text embeddings seems a feasible and efficient approach to predict years living with self-reported diabetes.

Heterogeneous treatment effects of Cerebrolysin as an early add-on to reperfusion therapy: post hoc analysis of the CEREHETIS trial

Background: There has been intensive research into enhancing the effects of reperfusion therapy to mitigate hemorrhagic transformation (HT) in stroke patients. Using neuroprotective agents alongside intravenous thrombolysis (IVT) appears a promising approach. Cerebrolysin is one of the candidates since it consists of neuropeptides mimicking the action of neurotrophic factors on brain protection and repair. Objectives: We looked at treatment effects of Cerebrolysin as an early add-on to IVT in stroke patients with varying HT risk. Methods: It was post hoc analysis of the CEREHETIS trial (ISRCTN87656744). Patients with middle cerebral artery infarction (n = 238) were selected from the intention-to-treat population. To stratify participants according to their HT risk, the DRAGON, SEDAN and HTI scores were computed for each eligible subject using on-admission data. The study endpoints were any and symptomatic HT, and functional outcome measured with the modified Rankin Scale (mRS) on day 90. Favorable functional outcome (FFO) was defined as an mRS ≤2. The performance of each stratification tool was estimated with regression approaches. Heterogeneous treatment effect analysis was conducted using techniques of meta-analysis and the matching-smoothing method. Results: The HTI score outperformed other tools in terms of HT risk stratification. Heterogeneity of Cerebrolysin treatment effects was moderate (I2, 35.8%-56.7%; H2, 1.56-2.31) and mild (I2, 10.9%; H2, 1.12) for symptomatic and any HT, respectively. A significant positive impact of Cerebrolysin on HT and functional outcome was observed in the moderate (HTI = 1) and high (HTI ≥2) HT risk patients, but it was neutral in those with the low (HTI = 0) risk. In particular, there was a steady decline in the rate of symptomatic (HTI = 0 vs. HTI = 4: by 4.3%, p = 0.077 vs. 21.1%, p < 0.001) and any HT (HTI = 0 vs. HTI = 4: by 1.2%, p = 0.737 vs. 32.7%, p < 0.001). Likewise, an mRS score reduction (HTI = 0 vs. HTI = 4: by 1.8%, p = 0.903 vs. 126%, p < 0.001) with a reciprocal increase of the fraction of FFO patients (HTI = 0 vs. HTI = 4: by 1.2% p = 0.757 vs. 35.5%, p < 0.001) was found. Conclusion: Clinically meaningful heterogeneity of Cerebrolysin treatment effects on HT and functional outcome was established in stroke patients. The beneficial effects were significant in those whose estimated on-admission HT risk was either moderate or high.

[Cerebrolysin as an early add-on to reperfusion therapy: heterogeneous treatment effect analysis in ischemic stroke patients with varying risk of hemorrhagic transformation]

OBJECTIVE

The study goal was the assessment of heterogeneous treatment effects of Cerebrolysin as an early add-on to reperfusion therapy in stroke patients with varying risk of hemorrhagic transformation (HT).

MATERIAL AND METHODS

It was post hoc analysis of the CEREHETIS trial (ISRCTN87656744). Patients with middle cerebral artery infarction (n=238) were stratified by HT risk with the HTI score. The study outcomes were symptomatic and any HT, and functional outcome measured with the modified Rankin Scale (mRS) on day 90. Favorable outcome was defined as an mRS score of ≤2. Heterogeneous treatment effect analysis was performed using techniques of meta-analysis and the matching-smoothing method.

RESULTS

Heterogeneity of Cerebrolysin treatment effects was moderate (I2=36.98-69.3%, H2=1.59-3.26) and mild (I2=18.33-32.39%, H2=1.22-1.48) for symptomatic and any HT, respectively. A positive impact of the Cerebrolysin treatment on HT and functional outcome was observed in patients with moderate (HTI=1) and high (HTI≥2) HT risk. However, the effect was neutral in those with low risk (HTI=0). In high HT risk patients, there was a steady decline in the rate of symptomatic (HTI=0 vs. HTI≥2: by 3.8%, p=0.120 vs. 14.3%, p<0.001) and any HT (HTI=0 vs. HTI≥2: by 0.6%, p=0.864 vs. 19.5%, p<0.001). Likewise, Cerebrolysin treatment resulted in an overall decrease in the mRS scores (HTI=0 vs. HTI≥2: by 2.1%, p=0.893 vs. 63%, p<0.001) with a reciprocal increase of the fraction with favorable outcome (HTI=0 vs. HTI≥2: by 2% p=0.634 vs. 19.2%, p<0.001).

CONCLUSION

Clinically meaningful heterogeneity of Cerebrolysin treatment effects on HT and functional outcome was established in stroke patients. The Cerebrolysin positive impact was significant in those whose estimated on-admission HT risk was either moderate or high.

Precision subclassification of type 2 diabetes: a systematic review

BACKGROUND

Heterogeneity in type 2 diabetes presentation and progression suggests that precision medicine interventions could improve clinical outcomes. We undertook a systematic review to determine whether strategies to subclassify type 2 diabetes were associated with high quality evidence, reproducible results and improved outcomes for patients.

METHODS

We searched PubMed and Embase for publications that used 'simple subclassification' approaches using simple categorisation of clinical characteristics, or 'complex subclassification' approaches which used machine learning or 'omics approaches in people with established type 2 diabetes. We excluded other diabetes subtypes and those predicting incident type 2 diabetes. We assessed quality, reproducibility and clinical relevance of extracted full-text articles and qualitatively synthesised a summary of subclassification approaches.

RESULTS

Here we show data from 51 studies that demonstrate many simple stratification approaches, but none have been replicated and many are not associated with meaningful clinical outcomes. Complex stratification was reviewed in 62 studies and produced reproducible subtypes of type 2 diabetes that are associated with outcomes. Both approaches require a higher grade of evidence but support the premise that type 2 diabetes can be subclassified into clinically meaningful subtypes.

CONCLUSION

Critical next steps toward clinical implementation are to test whether subtypes exist in more diverse ancestries and whether tailoring interventions to subtypes will improve outcomes.

Practical Guide to Honest Causal Forests for Identifying Heterogeneous Treatment Effects

"Heterogeneous treatment effects" is a term which refers to conditional average treatment effects (i.e., CATEs) that vary across population subgroups. Epidemiologists are often interested in estimating such effects because they can help detect populations that may particularly benefit from or be harmed by a treatment. However, standard regression approaches for estimating heterogeneous effects are limited by preexisting hypotheses, test a single effect modifier at a time, and are subject to the multiple-comparisons problem. In this article, we aim to offer a practical guide to honest causal forests, an ensemble tree-based learning method which can discover as well as estimate heterogeneous treatment effects using a data-driven approach. We discuss the fundamentals of tree-based methods, describe how honest causal forests can identify and estimate heterogeneous effects, and demonstrate an implementation of this method using simulated data. Our implementation highlights the steps required to simulate data sets, build honest causal forests, and assess model performance across a variety of simulation scenarios. Overall, this paper is intended for epidemiologists and other population health researchers who lack an extensive background in machine learning yet are interested in utilizing an emerging method for identifying and estimating heterogeneous treatment effects.

Comparison of causal forest and regression-based approaches to evaluate treatment effect heterogeneity: an application for type 2 diabetes precision medicine

OBJECTIVE

Precision medicine requires reliable identification of variation in patient-level outcomes with different available treatments, often termed treatment effect heterogeneity. We aimed to evaluate the comparative utility of individualized treatment selection strategies based on predicted individual-level treatment effects from a causal forest machine learning algorithm and a penalized regression model.

METHODS

Cohort study characterizing individual-level glucose-lowering response (6 month reduction in HbA1c) in people with type 2 diabetes initiating SGLT2-inhibitor or DPP4-inhibitor therapy. Model development set comprised 1,428 participants in the CANTATA-D and CANTATA-D2 randomised clinical trials of SGLT2-inhibitors versus DPP4-inhibitors. For external validation, calibration of observed versus predicted differences in HbA1c in patient strata defined by size of predicted HbA1c benefit was evaluated in 18,741 patients in UK primary care (Clinical Practice Research Datalink).

RESULTS

Heterogeneity in treatment effects was detected in clinical trial participants with both approaches (proportion predicted to have a benefit on SGLT2-inhibitor therapy over DPP4-inhibitor therapy: causal forest: 98.6%; penalized regression: 81.7%). In validation, calibration was good with penalized regression but sub-optimal with causal forest. A strata with an HbA1c benefit > 10 mmol/mol with SGLT2-inhibitors (3.7% of patients, observed benefit 11.0 mmol/mol [95%CI 8.0-14.0]) was identified using penalized regression but not causal forest, and a much larger strata with an HbA1c benefit 5-10 mmol with SGLT2-inhibitors was identified with penalized regression (regression: 20.9% of patients, observed benefit 7.8 mmol/mol (95%CI 6.7-8.9); causal forest 11.6%, observed benefit 8.7 mmol/mol (95%CI 7.4-10.1).

CONCLUSIONS

Consistent with recent results for outcome prediction with clinical data, when evaluating treatment effect heterogeneity researchers should not rely on causal forest or other similar machine learning algorithms alone, and must compare outputs with standard regression, which in this evaluation was superior.

Systematic review of precision subclassification of type 2 diabetes

Heterogeneity in type 2 diabetes presentation, progression and treatment has the potential for precision medicine interventions that can enhance care and outcomes for affected individuals. We undertook a systematic review to ascertain whether strategies to subclassify type 2 diabetes are associated with improved clinical outcomes, show reproducibility and have high quality evidence. We reviewed publications that deployed 'simple subclassification' using clinical features, biomarkers, imaging or other routinely available parameters or 'complex subclassification' approaches that used machine learning and/or genomic data. We found that simple stratification approaches, for example, stratification based on age, body mass index or lipid profiles, had been widely used, but no strategy had been replicated and many lacked association with meaningful outcomes. Complex stratification using clustering of simple clinical data with and without genetic data did show reproducible subtypes of diabetes that had been associated with outcomes such as cardiovascular disease and/or mortality. Both approaches require a higher grade of evidence but support the premise that type 2 diabetes can be subclassified into meaningful groups. More studies are needed to test these subclassifications in more diverse ancestries and prove that they are amenable to interventions.

A permutation procedure to detect heterogeneous treatment effects in randomized clinical trials while controlling the type I error rate

BACKGROUND/AIMS

Secondary analyses of randomized clinical trials often seek to identify subgroups with differential treatment effects. These discoveries can help guide individual treatment decisions based on patient characteristics and identify populations for which additional treatments are needed. Traditional analyses require researchers to pre-specify potential subgroups to reduce the risk of reporting spurious results. There is a need for methods that can detect such subgroups without a priori specification while allowing researchers to control the probability of falsely detecting heterogeneous subgroups when treatment effects are uniform across the study population.

METHODS

We propose a permutation procedure for tuning parameter selection that allows for type I error control when testing for heterogeneous treatment effects framed within the Virtual Twins procedure for subgroup identification. We verify that the type I error rate can be controlled at the nominal rate and investigate the power for detecting heterogeneous effects when present through extensive simulation studies. We apply our method to a secondary analysis of data from a randomized trial of very low nicotine content cigarettes.

RESULTS

In the absence of type I error control, the observed type I error rate for Virtual Twins was between 99% and 100%. In contrast, models tuned via the proposed permutation were able to control the type I error rate and detect heterogeneous effects when present. An application of our approach to a recently completed trial of very low nicotine content cigarettes identified several variables with potentially heterogeneous treatment effects.

CONCLUSIONS

The proposed permutation procedure allows researchers to engage in secondary analyses of clinical trials for treatment effect heterogeneity while maintaining the type I error rate without pre-specifying subgroups.

Heterogeneous treatment effects of intensive glycemic control on major adverse cardiovascular events in the ACCORD and VADT trials: a machine-learning analysis

BACKGROUND

Evidence to guide type 2 diabetes treatment individualization is limited. We evaluated heterogeneous treatment effects (HTE) of intensive glycemic control in type 2 diabetes patients on major adverse cardiovascular events (MACE) in the Action to Control Cardiovascular Risk in Diabetes Study (ACCORD) and the Veterans Affairs Diabetes Trial (VADT).

METHODS

Causal forests machine learning analysis was performed using pooled individual data from two randomized trials (n = 12,042) to identify HTE of intensive versus standard glycemic control on MACE in patients with type 2 diabetes. We used variable prioritization from causal forests to build a summary decision tree and examined the risk difference of MACE between treatment arms in the resulting subgroups.

RESULTS

A summary decision tree used five variables (hemoglobin glycation index, estimated glomerular filtration rate, fasting glucose, age, and body mass index) to define eight subgroups in which risk differences of MACE ranged from - 5.1% (95% CI - 8.7, - 1.5) to 3.1% (95% CI 0.2, 6.0) (negative values represent lower MACE associated with intensive glycemic control). Intensive glycemic control was associated with lower MACE in pooled study data in subgroups with low (- 4.2% [95% CI - 8.1, - 1.0]), intermediate (- 5.1% [95% CI - 8.7, - 1.5]), and high (- 4.3% [95% CI - 7.7, - 1.0]) MACE rates with consistent directions of effect in ACCORD and VADT alone.

CONCLUSIONS

This data-driven analysis provides evidence supporting the diabetes treatment guideline recommendation of intensive glucose lowering in diabetes patients with low cardiovascular risk and additionally suggests potential benefits of intensive glycemic control in some individuals at higher cardiovascular risk.