Underestimation of treatment effects in sequentially monitored clinical trials that did not stop early for benefit

Point estimation for adaptive trial designs II: Practical considerations and guidance

In adaptive clinical trials, the conventional end-of-trial point estimate of a treatment effect is prone to bias, that is, a systematic tendency to deviate from its true value. As stated in recent FDA guidance on adaptive designs, it is desirable to report estimates of treatment effects that reduce or remove this bias. However, it may be unclear which of the available estimators are preferable, and their use remains rare in practice. This article is the second in a two-part series that studies the issue of bias in point estimation for adaptive trials. Part I provided a methodological review of approaches to remove or reduce the potential bias in point estimation for adaptive designs. In part II, we discuss how bias can affect standard estimators and assess the negative impact this can have. We review current practice for reporting point estimates and illustrate the computation of different estimators using a real adaptive trial example (including code), which we use as a basis for a simulation study. We show that while on average the values of these estimators can be similar, for a particular trial realization they can give noticeably different values for the estimated treatment effect. Finally, we propose guidelines for researchers around the choice of estimators and the reporting of estimates following an adaptive design. The issue of bias should be considered throughout the whole lifecycle of an adaptive design, with the estimation strategy prespecified in the statistical analysis plan. When available, unbiased or bias-reduced estimates are to be preferred.

Point estimation for adaptive trial designs I: A methodological review

Recent FDA guidance on adaptive clinical trial designs defines bias as "a systematic tendency for the estimate of treatment effect to deviate from its true value," and states that it is desirable to obtain and report estimates of treatment effects that reduce or remove this bias. The conventional end-of-trial point estimates of the treatment effects are prone to bias in many adaptive designs, because they do not take into account the potential and realized trial adaptations. While much of the methodological developments on adaptive designs have tended to focus on control of type I error rates and power considerations, in contrast the question of biased estimation has received relatively less attention. This article is the first in a two-part series that studies the issue of potential bias in point estimation for adaptive trials. Part I provides a comprehensive review of the methods to remove or reduce the potential bias in point estimation of treatment effects for adaptive designs, while part II illustrates how to implement these in practice and proposes a set of guidelines for trial statisticians. The methods reviewed in this article can be broadly classified into unbiased and bias-reduced estimation, and we also provide a classification of estimators by the type of adaptive design. We compare the proposed methods, highlight available software and code, and discuss potential methodological gaps in the literature.

Estimation of the treatment effect following a clinical trial that stopped early for benefit

When a clinical trial stops early for benefit, the maximum likelihood estimate (MLE) of the treatment effect may be subject to overestimation bias. Several authors have proposed adjusting for this bias using the conditional MLE, which is obtained by conditioning on early stopping. However, this approach has a fundamental problem in that the adjusted estimate may not be in the direction of benefit, even though the study has stopped early due to benefit. In this paper, we address this problem by embedding both the MLE and the conditional MLE within a broader class of penalised likelihood estimates, and choosing a member of the class that is a favourable compromise between the two. This penalised MLE, and its associated confidence interval, always lie in the direction of benefit when the study stops early for benefit. We study its properties using both simulations and analyses of the ENZAMET trial in metastatic prostate cancer. Conditional on stopping early for benefit, the method is found to have good unbiasedness and coverage properties, along with very favourable efficiency at earlier interim analyses. We recommend the penalised MLE as a supplementary analysis to a conventional primary analysis when a clinical trial stops early for benefit.

Analysis of adaptive platform trials using a network approach

Background

Adaptive platform trials allow randomized controlled comparisons of multiple treatments using a common infrastructure and the flexibility to adapt key design features during the study. Nonetheless, they have been criticized due to the potential for time trends in the underlying risk level of the population. Such time trends lead to confounding between design features and risk level, which may introduce bias favoring one or more treatments. This is particularly true when experimental treatments are not all randomized during the same time period as the control, leading to the potential for bias from non-concurrent controls.

Methods

Two analysis methods addressing this bias are stratification and adjustment. Stratification uses only comparisons between treatment cohorts randomized during identical time periods and does not use non-concurrent randomizations. Adjustment uses a modeled analysis including time period adjustment, allowing all data to be used, even from periods without concurrent randomization. We show that these competing approaches may be embedded in a common framework using network meta-analysis principles. We interpret the stages between adaptations in a platform trial as separate fixed design trials. This allows platform trials to be viewed as networks of direct randomized comparisons and indirect non-randomized comparisons. Network meta-analysis methodology can be re-purposed to aggregate the total information from a platform trial and to transparently decompose this total information into direct randomized evidence and indirect non-randomized evidence. This allows sensitivity to indirect information to be assessed and the two analysis methods to be clearly compared.

Results

Simulations of platform trials were analyzed using a network approach implemented in the netmeta package in R. The results demonstrated bias of unadjusted methods in the presence of time trends in risk level. Adjustment and stratification were both unbiased when direct evidence and indirect evidence were consistent. Network tests of inconsistency may be used to diagnose inconsistency when it exists. In an illustrative network analysis of one of the treatment comparisons from the STAMPEDE platform trial in metastatic prostate cancer, indirect comparisons using non-concurrent controls were inconsistent with the information from direct randomized comparisons. This supports the primary analysis approach of STAMPEDE, which used only direct randomized comparisons.

Conclusion

Network meta-analysis provides a natural methodology for analyzing the network of direct and indirect treatment comparisons from a platform trial. Such analyses provide transparent separation of direct and indirect evidence, allowing assessment of the impact of non-concurrent controls. We recommend time-stratified analysis of concurrently controlled comparisons for primary analyses, with time-adjusted analyses incorporating non-concurrent controls reserved for secondary analyses. However, regardless of which methodology is used, a network analysis provides a useful supplement to the primary analysis.

The randomized Tracheal Occlusion To Accelerate Lung growth (TOTAL)-trials on fetal surgery for congenital diaphragmatic hernia: reanalysis using pooled data

BACKGROUND

Two randomized controlled trials compared the neonatal and infant outcomes after fetoscopic endoluminal tracheal occlusion with expectant prenatal management in fetuses with severe and moderate isolated congenital diaphragmatic hernia, respectively. Fetoscopic endoluminal tracheal occlusion was carried out at 27⁺⁰ to 29⁺⁶ weeks' gestation (referred to as "early") for severe and at 30⁺⁰ to 31⁺⁶ weeks ("late") for moderate hypoplasia. The reported absolute increase in the survival to discharge was 13% (95% confidence interval, -1 to 28; P=.059) and 25% (95% confidence interval, 6-46; P=.0091) for moderate and severe hypoplasia.

OBJECTIVE

Data from the 2 trials were pooled to study the heterogeneity of the treatment effect by observed over expected lung-to-head ratio and explore the effect of gestational age at balloon insertion.

STUDY DESIGN

Individual participant data from the 2 trials were reanalyzed. Women were assessed between 2008 and 2020 at 14 experienced fetoscopic endoluminal tracheal occlusion centers and were randomized in a 1:1 ratio to either expectant management or fetoscopic endoluminal tracheal occlusion. All received standardized postnatal management. The combined data involved 287 patients (196 with moderate hypoplasia and 91 with severe hypoplasia). The primary endpoint was survival to discharge from the neonatal intensive care unit. The secondary endpoints were survival to 6 months of age, survival to 6 months without oxygen supplementation, and gestational age at live birth. Penalized regression was used with the following covariates: intervention (fetoscopic endoluminal tracheal occlusion vs expectant), early balloon insertion (yes vs no), observed over expected lung-to-head ratio, liver herniation (yes vs no), and trial (severe vs moderate). The interaction between intervention and the observed over expected lung-to-head ratio was evaluated to study treatment effect heterogeneity.

RESULTS

For survival to discharge, the adjusted odds ratio of fetoscopic endoluminal tracheal occlusion was 1.78 (95% confidence interval, 1.05-3.01; P=.031). The additional effect of early balloon insertion was highly uncertain (adjusted odds ratio, 1.53; 95% confidence interval, 0.60-3.91; P=.370). When combining these 2 effects, the adjusted odds ratio of fetoscopic endoluminal tracheal occlusion with early balloon insertion was 2.73 (95% confidence interval, 1.15-6.49). The results for survival to 6 months and survival to 6 months without oxygen dependence were comparable. The gestational age at delivery was on average 1.7 weeks earlier (95% confidence interval, 1.1-2.3) following fetoscopic endoluminal tracheal occlusion with late insertion and 3.2 weeks earlier (95% confidence interval, 2.3-4.1) following fetoscopic endoluminal tracheal occlusion with early insertion compared with expectant management. There was no evidence that the effect of fetoscopic endoluminal tracheal occlusion depended on the observed over expected lung-to-head ratio for any of the endpoints.

CONCLUSION

This analysis suggests that fetoscopic endoluminal tracheal occlusion increases survival for both moderate and severe lung hypoplasia. The difference between the results for the Tracheal Occlusion To Accelerate Lung growth trials, when considered apart, may be because of the difference in the time point of balloon insertion. However, the effect of the time point of balloon insertion could not be robustly assessed because of a small sample size and the confounding effect of disease severity. Fetoscopic endoluminal tracheal occlusion with early balloon insertion in particular strongly increases the risk for preterm delivery.

Conditional information and inference in response-adaptive allocation designs

Response-adaptive allocation designs refer to a class of designs where the probability an observation is assigned to a treatment is changed throughout an experiment based on the accrued responses. Such procedures result in random treatment sample sizes. Most of the current literature considers unconditional inference procedures in the analysis of response-adaptive allocation designs. The focus of this article is inference conditional on the observed treatment sample sizes. The inverse of information is a description of the large sample variance of the parameter estimates. A simple form for the conditional information relative to unconditional information is derived. It is found that conditional information can be greater than unconditional information. A conditional bootstrap procedure is developed that, in the majority of cases examined, resulted in narrower confidence intervals than relevant unconditional procedures.

A pragmatic randomized controlled trial of multi-dose oral ondansetron for pediatric gastroenteritis (the DOSE-AGE study): statistical analysis plan

Background

Acute gastroenteritis is a leading cause of emergency department visits and hospitalizations among children in North America. Oral-rehydration therapy is recommended for children with mild-to-moderate dehydration, but children who present with vomiting are frequently offered intravenous rehydration in the emergency department (ED). Recent studies have demonstrated that the anti-emetic ondansetron can reduce vomiting, intravenous rehydration, and hospitalization when administered in the ED to children with dehydration. However, there is little evidence of additional benefit from prescribing ondansetron beyond the initial ED dose. Moreover, repeat dosing may increase the frequency of diarrhea. Despite the lack of evidence and potential adverse side effects, many physicians across North America provide multiple doses of ondansetron to be taken following ED disposition. Thus, the Multi-Dose Oral Ondansetron for Pediatric Gastroenteritis (DOSE-AGE) trial will evaluate the effectiveness of prescribing multiple doses of ondansetron to treat acute gastroenteritis-associated vomiting. This article specifies the statistical analysis plan (SAP) for the DOSE-AGE trial and was submitted before the outcomes of the study were available for analysis.

Methods/design

The DOSE-AGE study is a phase III, 6-center, placebo-controlled, double-blind, parallel design randomized controlled trial designed to determine whether participants who are prescribed multiple doses of oral ondansetron to administer, as needed, following their ED visit have a lower incidence of experiencing moderate-to-severe gastroenteritis, as measured by the Modified Vesikari Scale score, compared with a placebo. To assess safety, the DOSE-AGE trial will investigate the frequency and maximum number of diarrheal episodes following ED disposition, and the occurrence of palpitations, pre-syncope/syncope, chest pain, arrhythmias, and serious adverse events. For the secondary outcomes, the DOSE-AGE trial will investigate the individual elements of the Modified Vesikari Scale score and caregiver satisfaction with the therapy.

Discussion

The DOSE-AGE trial will provide evidence on the effectiveness of multiple doses of oral ondansetron, taken as needed, following an initial ED dose in children with acute gastroenteritis-associated vomiting. The data from the DOSE-AGE trial will be analyzed using this SAP. This will reduce the risk of producing data-driven results and bias in our reported outcomes. The DOSE-AGE study was registered on ClinicalTrials.gov on February 22, 2019.

Trial registration

ClinicalTrials.gov NCT03851835. Registered on 22 February 2019.

Sensitivity analyses assessing the impact of early stopping on systematic reviews: Recommendations for interpreting guidelines

The CONSORT Statement says that data-driven early stopping of a clinical trial is likely to weaken the inferences that can be drawn from the trial. The GRADE guidelines go further, saying that early stopping is a study limitation that carries the risk of bias, and recommending sensitivity analyses in which trials stopped early are omitted from evidence synthesis. Despite extensive debate in the literature over these issues, the existence of clear recommendations in high profile guidelines makes it inevitable that systematic reviewers will consider sensitivity analyses investigating the impact of early stopping. The purpose of this article is to assess methodologies for conducting such sensitivity analyses, and to make recommendations about how the guidelines should be interpreted. We begin with a clarifying overview of the impacts of early stopping on treatment effect estimation in single studies and meta-analyses. We then warn against naive approaches for conducting sensitivity analyses, including simply omitting trials stopped early from meta-analyses. This approach underestimates treatment effects, which may have serious implications if cost-effectiveness analyses determine whether treatments are made widely available. Instead, we discuss two unbiased approaches to sensitivity analysis, one of which is straightforward but statistically inefficient, and the other of which achieves greater statistical efficiency by making use of recent methodological developments in the analysis of clinical trials. We end with recommendations for interpreting: (a) the CONSORT Statement on reporting of reasons for early stopping, and (b) the GRADE guidelines on sensitivity analyses assessing the impact of early stopping.