Machine learning analysis plans for randomised controlled trials: detecting treatment effect heterogeneity with strict control of type I error

Comparing algorithms for characterizing treatment effect heterogeneity in randomized trials

The identification and estimation of heterogeneous treatment effects in biomedical clinical trials are challenging, because trials are typically planned to assess the treatment effect in the overall trial population. Nevertheless, the identification of how the treatment effect may vary across subgroups is of major importance for drug development. In this work, we review some existing simulation work and perform a simulation study to evaluate recent methods for identifying and estimating the heterogeneous treatments effects using various metrics and scenarios relevant for drug development. Our focus is not only on a comparison of the methods in general, but on how well these methods perform in simulation scenarios that reflect real clinical trials. We provide the R package benchtm that can be used to simulate synthetic biomarker distributions based on real clinical trial data and to create interpretable scenarios to benchmark methods for identification and estimation of treatment effect heterogeneity.

A Machine-Learning Approach for Estimating Subgroup- and Individual-Level Treatment Effects: An Illustration Using the 65 Trial

Personalizing treatment recommendations or guidelines requires evidence about the heterogeneity of treatment effects (HTE). Machine-learning (ML) approaches can explore HTE by considering many covariates, including complex interactions between them. Causal ML approaches can avoid overfitting, which arises when the same dataset is used to select covariate by treatment interaction terms as to make inferences and reduce reliance on the correct specification of fixed parametric models. We investigate causal forests (CF), a ML method based on modified decision trees that can estimate subgroup- and individual-level treatment effects, without requiring correct prespecification of the effect model. We consider CF alongside parametric approaches for estimating HTE, within the 65 Trial, which evaluates the effect of a permissive hypotension strategy versus usual care on 90-d mortality for critically ill patients aged 65 y or older with vasodilatory hypotension. Here, the CF approach provides similar estimates of treatment effectiveness for prespecified and post hoc subgroups to the parametric approach, and the results of a test for overall HTE show weak evidence of heterogeneity. The CF estimates of individual-level treatment effects, the expected effects of treatment for individuals in subpopulations defined by their covariates, suggest that the permissive hypotension strategy is expected to reduce 90-d mortality for 98.7% of patients but with 95% confidence intervals that include zero for 71.6% of patients. A ML approach is then used to assess the patient characteristics associated with these individual-level effects, and to help target future research that can identify those patient subgroups for whom the intervention is most effective.

LANDMark: an ensemble approach to the supervised selection of biomarkers in high-throughput sequencing data

Background

Identification of biomarkers, which are measurable characteristics of biological datasets, can be challenging. Although amplicon sequence variants (ASVs) can be considered potential biomarkers, identifying important ASVs in high-throughput sequencing datasets is challenging. Noise, algorithmic failures to account for specific distributional properties, and feature interactions can complicate the discovery of ASV biomarkers. In addition, these issues can impact the replicability of various models and elevate false-discovery rates. Contemporary machine learning approaches can be leveraged to address these issues. Ensembles of decision trees are particularly effective at classifying the types of data commonly generated in high-throughput sequencing (HTS) studies due to their robustness when the number of features in the training data is orders of magnitude larger than the number of samples. In addition, when combined with appropriate model introspection algorithms, machine learning algorithms can also be used to discover and select potential biomarkers. However, the construction of these models could introduce various biases which potentially obfuscate feature discovery.

Results

We developed a decision tree ensemble, LANDMark, which uses oblique and non-linear cuts at each node. In synthetic and toy tests LANDMark consistently ranked as the best classifier and often outperformed the Random Forest classifier. When trained on the full metabarcoding dataset obtained from Canada’s Wood Buffalo National Park, LANDMark was able to create highly predictive models and achieved an overall balanced accuracy score of 0.96 ± 0.06. The use of recursive feature elimination did not impact LANDMark’s generalization performance and, when trained on data from the BE amplicon, it was able to outperform the Linear Support Vector Machine, Logistic Regression models, and Stochastic Gradient Descent models (p ≤ 0.05). Finally, LANDMark distinguishes itself due to its ability to learn smoother non-linear decision boundaries.

Conclusions

Our work introduces LANDMark, a meta-classifier which blends the characteristics of several machine learning models into a decision tree and ensemble learning framework. To our knowledge, this is the first study to apply this type of ensemble approach to amplicon sequencing data and we have shown that analyzing these datasets using LANDMark can produce highly predictive and consistent models.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-022-04631-z.

Predictive Modeling Identifies Total Bleeds at 12-Weeks Postswitch to N8-GP Prophylaxis as a Predictor of Treatment Response

Background  Predicting annualized bleeding rate (ABR) during factor VIII (FVIII) prophylaxis for severe hemophilia A (SHA) is important for long-term outcomes. This study used supervised machine learning-based predictive modeling to identify predictors of long-term ABR during prophylaxis with an extended half-life FVIII.

Methods  Data were from 166 SHA patients who received N8-GP prophylaxis (50 IU/kg every 4 days) in the pathfinder 2 study. Predictive models were developed to identify variables associated with an ABR of ≤1 versus >1 during the trial's main phase (median follow-up of 469 days). Model performance was assessed using area under the receiver operator characteristic curve (AUROC). Pre-N8-GP prophylaxis models learned from data collected at baseline; post-N8-GP prophylaxis models learned from data collected up to 12-weeks postswitch to N8-GP, and predicted ABR at the end of the outcome period (final year of treatment in the main phase).

Results  The predictive model using baseline variables had moderate performance (AUROC = 0.64) for predicting observed ABR. The most performant model used data collected at 12-weeks postswitch (AUROC = 0.79) with cumulative bleed count up to 12 weeks as the most informative variable, followed by baseline von Willebrand factor and mean FVIII at 30 minutes postdose. Univariate cumulative bleed count at 12 weeks performed equally well to the 12-weeks postswitch model (AUROC = 0.75). Pharmacokinetic measures were indicative, but not essential, to predict ABR.

Conclusion  Cumulative bleed count up to 12-weeks postswitch was as informative as the 12-week post-switch predictive model for predicting long-term ABR, supporting alterations in prophylaxis based on treatment response.

Assessing effect heterogeneity of a randomized treatment using conditional inference trees

Treatment effect heterogeneity occurs when individual characteristics influence the effect of a treatment. We propose a novel approach that combines prognostic score matching and conditional inference trees to characterize effect heterogeneity of a randomized binary treatment. One key feature that distinguishes our method from alternative approaches is that it controls the Type I error rate, that is, the probability of identifying effect heterogeneity if none exists and retains the underlying subgroups. This feature makes our technique particularly appealing in the context of clinical trials, where there may be significant costs associated with erroneously declaring that effects differ across population subgroups. Treatment effect heterogeneity trees are able to identify heterogeneous subgroups, characterize the relevant subgroups and estimate the associated treatment effects. We demonstrate the efficacy of the proposed method using a comprehensive simulation study and illustrate our method using a nutrition trial dataset to evaluate effect heterogeneity within a patient population.

The role of machine learning in clinical research: transforming the future of evidence generation

Background

Interest in the application of machine learning (ML) to the design, conduct, and analysis of clinical trials has grown, but the evidence base for such applications has not been surveyed. This manuscript reviews the proceedings of a multi-stakeholder conference to discuss the current and future state of ML for clinical research. Key areas of clinical trial methodology in which ML holds particular promise and priority areas for further investigation are presented alongside a narrative review of evidence supporting the use of ML across the clinical trial spectrum.

Results

Conference attendees included stakeholders, such as biomedical and ML researchers, representatives from the US Food and Drug Administration (FDA), artificial intelligence technology and data analytics companies, non-profit organizations, patient advocacy groups, and pharmaceutical companies. ML contributions to clinical research were highlighted in the pre-trial phase, cohort selection and participant management, and data collection and analysis. A particular focus was paid to the operational and philosophical barriers to ML in clinical research. Peer-reviewed evidence was noted to be lacking in several areas.

Conclusions

ML holds great promise for improving the efficiency and quality of clinical research, but substantial barriers remain, the surmounting of which will require addressing significant gaps in evidence.

Machine Learning for Precision Health Economics and Outcomes Research (P-HEOR): Conceptual Review of Applications and Next Steps

Precision health economics and outcomes research (P-HEOR) integrates economic and clinical value assessment by explicitly discovering distinct clinical and health care utilization phenotypes among patients. Through a conceptualized example, the objective of this review is to highlight the capabilities and limitations of machine learning (ML) applications to P-HEOR and to contextualize the potential opportunities and challenges for the wide adoption of ML for health economics. We outline a P-HEOR conceptual framework extending the ML methodology to comparatively assess the economic value of treatment regimens. Latest methodology developments on bias and confounding control in ML applications to precision medicine are also summarized.

Optimal individualized decision rules from a multi-arm trial: A comparison of methods and an application to tailoring inter-donation intervals among blood donors in the UK

There is a growing interest in precision medicine where individual heterogeneity is incorporated into decision-making and treatments are tailored to individuals to provide better healthcare. One important aspect of precision medicine is the estimation of the optimal individualized treatment rule (ITR) that optimizes the expected outcome. Most methods developed for this purpose are restricted to the setting with two treatments, while clinical studies with more than two treatments are common in practice. In this work, we summarize methods to estimate the optimal ITR in the multi-arm setting and compare their performance in large-scale clinical trials via simulation studies. We then illustrate their utilities with a case study using the data from the INTERVAL trial, which randomly assigned over 20,000 male blood donors from England to one of the three inter-donation intervals (12-week, 10-week, and eight-week) over two years. We estimate the optimal individualized donation strategies under three different objectives. Our findings are fairly consistent across five different approaches that are applied: when we target the maximization of the total units of blood collected, almost all donors are assigned to the eight-week inter-donation interval, whereas if we aim at minimizing the low hemoglobin deferral rates, almost all donors are assigned to donate every 12 weeks. However, when the goal is to maximize the utility score that "discounts" the total units of blood collected by the incidences of low hemoglobin deferrals, we observe some heterogeneity in the optimal inter-donation interval across donors and the optimal donor assignment strategy is highly dependent on the trade-off parameter in the utility function.