Influence of trial duration on the bias of the estimated treatment effect in clinical trials when individual heterogeneity is ignored

Clinical trial design and treatment effects: a meta-analysis of randomised controlled and single-arm trials supporting 437 FDA approvals of cancer drugs and indications

OBJECTIVES

This study aims to analyse the association between clinical trial design and treatment effects for cancer drugs with US Food and Drug Administration (FDA) approval.

DESIGN

Cross-sectional study and meta-analysis.

SETTING

Data from Drugs@FDA, FDA labels, ClincialTrials.gov and the Global Burden of Disease study.

PARTICIPANTS

Pivotal trials for 170 drugs with FDA approval across 437 cancer indications between 2000 and 2022.

MAIN OUTCOME MEASURES

Treatment effects were measured in HRs for overall survival (OS) and progression-free survival (PFS), and in relative risk for tumour response. Random-effects meta-analyses and meta-regressions explored the association between treatment effect estimates and clinical trial design for randomised controlled trials (RCTs) and single-arm trials.

RESULTS

Across RCTs, greater effect estimates were observed in smaller trials for OS (ß=0.06, p<0.001), PFS (ß=0.15, p<0.001) and tumour response (ß=-3.61, p<0.001). Effect estimates were larger in shorter trials for OS (ß=0.08, p<0.001) and PFS (ß=0.09, p=0.002). OS (ß=0.04, p=0.006), PFS (ß=0.10, p<0.001) and tumour response (ß=-2.91, p=0.004) outcomes were greater in trials with fewer centres. HRs for PFS (0.54 vs 0.62, p=0.011) were lower in trials testing the new drug to an inactive (placebo/no treatment) rather than an active comparator. The analysed efficacy population (intention-to-treat, per-protocol, or as-treated) was not consistently associated with treatment effects. Results were consistent for single-arm trials and in multivariable analyses.

CONCLUSIONS

Pivotal trial design is significantly associated with measured treatment effects. Particularly small, short, single-centre trials testing a new drug compared with an inactive rather than an active comparator could overstate treatment outcomes. Future studies should verify results in unsuccessful trials, adjust for further confounders and examine other therapeutic areas. The FDA, manufacturers and trialists must strive to conduct robust clinical trials with a low risk of bias.

How hazardous are hazard ratios? An empirical investigation of individual patient data from 27 large randomized clinical trials

The use of hazard ratios as the standard treatment effect estimators for randomized trials with time-to-event outcomes has been the subject of repeated criticisms in recent years, e.g., for its non-collapsibility or with respect to (causal) interpretation. Another important issue is the built-in selection bias, which arises when the treatment is effective and when there are unobserved or not included prognostic factors that influence time-to-event. In these cases, the hazard ratio has even been termed "hazardous" because it is estimated from groups that increasingly differ in their (unobserved or omitted) baseline characteristics, yielding biased treatment estimates. We therefore adapt the Landmarking approach to assess the effect of ignoring a gradually increasing proportion of early events on the estimated hazard ratio. We propose an extension called "Dynamic Landmarking". This approach is based on successive deletion of observations, refitting Cox models and balance checking of omitted but observed prognostic factors, to obtain a visualization that can indicate built-in selection bias. In a small proof-of-concept simulation, we show that our approach is valid under the given assumptions. We further use "Dynamic Landmarking" to assess the suspected selection bias in the individual patient data sets of 27 large randomized clinical trials (RCTs). Surprisingly, we find no empirical evidence of selection bias in these RCTs and thus conclude that the supposed bias of the hazard ratio is of little practical relevance in most cases. This is mainly due to treatment effects in RCTs being small and the patient populations being homogeneous, e.g., due to inclusion and exclusion criteria.

Marginal hazard ratio estimates in joint frailty models for heart failure trials

This work is motivated by clinical trials in chronic heart failure disease, where treatment has effects both on morbidity (assessed as recurrent non-fatal hospitalisations) and on mortality (assessed as cardiovascular death, CV death). Recently, a joint frailty proportional hazards model has been proposed for these kind of efficacy outcomes to account for a potential association between the risk rates for hospital admissions and CV death. However, more often clinical trial results are presented by treatment effect estimates that have been derived from marginal proportional hazards models, that is, a Cox model for mortality and an Andersen-Gill model for recurrent hospitalisations. We show how these marginal hazard ratios and their estimates depend on the association between the risk processes, when these are actually linked by shared or dependent frailty terms. First we derive the marginal hazard ratios as a function of time. Then, applying least false parameter theory, we show that the marginal hazard ratio estimate for the hospitalisation rate depends on study duration and on parameters of the underlying joint frailty model. In particular, we identify parameters, for example the treatment effect on mortality, that determine if the marginal hazard ratio estimate for hospitalisations is smaller, equal or larger than the conditional one. How this affects rejection probabilities is further investigated in simulation studies. Our findings can be used to interpret marginal hazard ratio estimates in heart failure trials and are illustrated by the results of the CHARM-Preserved trial (where CHARM is the 'Candesartan in Heart failure Assessment of Reduction in Mortality and morbidity' programme).

A DAG-based comparison of interventional effect underestimation between composite endpoint and multi-state analysis in cardiovascular trials

Background

Composite endpoints comprising hospital admissions and death are the primary outcome in many cardiovascular clinical trials. For statistical analysis, a Cox proportional hazards model for the time to first event is commonly applied. There is an ongoing debate on whether multiple episodes per individual should be incorporated into the primary analysis. While the advantages in terms of power are readily apparent, potential biases have been mostly overlooked so far.

Methods

Motivated by a randomized controlled clinical trial in heart failure patients, we use directed acyclic graphs (DAG) to investigate potential sources of bias in treatment effect estimates, depending on whether only the first or multiple episodes are considered. The biases first are explained in simplified examples and then more thoroughly investigated in simulation studies that mimic realistic patterns.

Results

Particularly the Cox model is prone to potentially severe selection bias and direct effect bias, resulting in underestimation when restricting the analysis to first events. We find that both kinds of bias can simultaneously be reduced by adequately incorporating recurrent events into the analysis model. Correspondingly, we point out appropriate proportional hazards-based multi-state models for decreasing bias and increasing power when analyzing multiple-episode composite endpoints in randomized clinical trials.

Conclusions

Incorporating multiple episodes per individual into the primary analysis can reduce the bias of a treatment’s total effect estimate. Our findings will help to move beyond the paradigm of considering first events only for approaches that use more information from the trial and augment interpretability, as has been called for in cardiovascular research.