Estimation of the odds ratio from multi-stage randomized trials

A multi-stage design for a randomized trial is to allow early termination of the study when the experimental arm is found to have low or high efficacy compared to the control during the study. In such a trial, an early stopping rule results in bias in the maximum likelihood estimator of the treatment effect. We consider multi-stage randomized trials on a dichotomous outcome, such as treatment response, and investigate the estimation of the odds ratio. Typically, randomized phase II cancer clinical trials have two-stage designs with small sample sizes, which makes the estimation of odds ratio more challenging. In this paper, we evaluate several existing estimation methods of odds ratio and propose bias-corrected estimators for randomized multi-stage trials, including randomized phase II cancer clinical trials. Through numerical studies, the proposed estimators are shown to have a smaller bias and a smaller mean squared error overall.

Point estimation following a two-stage group sequential trial

Repeated testing in a group sequential trial can result in bias in the maximum likelihood estimate of the unknown parameter of interest. Many authors have therefore proposed adjusted point estimation procedures, which attempt to reduce such bias. Here, we describe nine possible point estimators within a common general framework for a two-stage group sequential trial. We then contrast their performance in five example trial settings, examining their conditional and marginal biases and residual mean square error. By focusing on the case of a trial with a single interim analysis, additional new results aiding the determination of the estimators are given. Our findings demonstrate that the uniform minimum variance unbiased estimator, whilst being marginally unbiased, often has large conditional bias and residual mean square error. If one is concerned solely about inference on progression to the second trial stage, the conditional uniform minimum variance unbiased estimator may be preferred. Two estimators, termed mean adjusted estimators, which attempt to reduce the marginal bias, arguably perform best in terms of the marginal residual mean square error. In all, one should choose an estimator accounting for its conditional and marginal biases and residual mean square error; the most suitable estimator will depend on relative desires to minimise each of these factors. If one cares solely about the conditional and marginal biases, the conditional maximum likelihood estimate may be preferred provided lower and upper stopping boundaries are included. If the conditional and marginal residual mean square error are also of concern, two mean adjusted estimators perform well.

Point estimation for adaptive trial designs I: A methodological review

Recent FDA guidance on adaptive clinical trial designs defines bias as "a systematic tendency for the estimate of treatment effect to deviate from its true value," and states that it is desirable to obtain and report estimates of treatment effects that reduce or remove this bias. The conventional end-of-trial point estimates of the treatment effects are prone to bias in many adaptive designs, because they do not take into account the potential and realized trial adaptations. While much of the methodological developments on adaptive designs have tended to focus on control of type I error rates and power considerations, in contrast the question of biased estimation has received relatively less attention. This article is the first in a two-part series that studies the issue of potential bias in point estimation for adaptive trials. Part I provides a comprehensive review of the methods to remove or reduce the potential bias in point estimation of treatment effects for adaptive designs, while part II illustrates how to implement these in practice and proposes a set of guidelines for trial statisticians. The methods reviewed in this article can be broadly classified into unbiased and bias-reduced estimation, and we also provide a classification of estimators by the type of adaptive design. We compare the proposed methods, highlight available software and code, and discuss potential methodological gaps in the literature.

Estimation of the treatment effect following a clinical trial that stopped early for benefit

When a clinical trial stops early for benefit, the maximum likelihood estimate (MLE) of the treatment effect may be subject to overestimation bias. Several authors have proposed adjusting for this bias using the conditional MLE, which is obtained by conditioning on early stopping. However, this approach has a fundamental problem in that the adjusted estimate may not be in the direction of benefit, even though the study has stopped early due to benefit. In this paper, we address this problem by embedding both the MLE and the conditional MLE within a broader class of penalised likelihood estimates, and choosing a member of the class that is a favourable compromise between the two. This penalised MLE, and its associated confidence interval, always lie in the direction of benefit when the study stops early for benefit. We study its properties using both simulations and analyses of the ENZAMET trial in metastatic prostate cancer. Conditional on stopping early for benefit, the method is found to have good unbiasedness and coverage properties, along with very favourable efficiency at earlier interim analyses. We recommend the penalised MLE as a supplementary analysis to a conventional primary analysis when a clinical trial stops early for benefit.

Optimised point estimators for multi-stage single-arm phase II oncology trials

The uniform minimum variance unbiased estimator (UMVUE) is, by definition, a solution to removing bias in estimation following a multi-stage single-arm trial with a primary dichotomous outcome. However, the UMVUE is known to have large residual mean squared error (RMSE). Therefore, we develop an optimisation approach to finding estimators with reduced RMSE for many response rates, which attain low bias. We demonstrate that careful choice of the optimisation parameters can lead to an estimator with often substantially reduced RMSE, without the introduction of appreciable bias.

Sensitivity analyses assessing the impact of early stopping on systematic reviews: Recommendations for interpreting guidelines

The CONSORT Statement says that data-driven early stopping of a clinical trial is likely to weaken the inferences that can be drawn from the trial. The GRADE guidelines go further, saying that early stopping is a study limitation that carries the risk of bias, and recommending sensitivity analyses in which trials stopped early are omitted from evidence synthesis. Despite extensive debate in the literature over these issues, the existence of clear recommendations in high profile guidelines makes it inevitable that systematic reviewers will consider sensitivity analyses investigating the impact of early stopping. The purpose of this article is to assess methodologies for conducting such sensitivity analyses, and to make recommendations about how the guidelines should be interpreted. We begin with a clarifying overview of the impacts of early stopping on treatment effect estimation in single studies and meta-analyses. We then warn against naive approaches for conducting sensitivity analyses, including simply omitting trials stopped early from meta-analyses. This approach underestimates treatment effects, which may have serious implications if cost-effectiveness analyses determine whether treatments are made widely available. Instead, we discuss two unbiased approaches to sensitivity analysis, one of which is straightforward but statistically inefficient, and the other of which achieves greater statistical efficiency by making use of recent methodological developments in the analysis of clinical trials. We end with recommendations for interpreting: (a) the CONSORT Statement on reporting of reasons for early stopping, and (b) the GRADE guidelines on sensitivity analyses assessing the impact of early stopping.

Reducing overestimation of the treatment effect by interim analysis when designing clinical trials

WHAT IS KNOWN AND OBJECTIVE

Several researchers in the statistical and medical communities have noted the overestimation of the treatment effect when a trial is stopped early in the interim analysis for efficacy; however, methods to reduce this overestimation are rarely used because the overestimation mechanisms are not well understood by many in clinical trial practice. A trial design that leads to less overestimation is needed.

METHODS

A computer simulation of hypothetical clinical trials is used to visually explain why the overestimation occurs. A quantitative evaluation of the magnitude of the overestimation is made according to the characteristics of the trial design, such as the total number of events, number of events in the interim analysis, proportion of the number of events to total events and the type of α-spending function.

RESULTS AND DISCUSSION

When the total number of events was more than or equal to 300 and the proportion of the interim events was larger than 50%, the overestimation was acceptable. Moreover, even if the total number of events was 150, the overestimation was sufficiently small when the proportion of the interim events was >70% and a Pocock type α-spending function was used. The overestimation decreased when the total number of events and the proportion of the number of events in the interim analysis increased. In addition, the overestimation of the Pocock type α-spending function was smaller in comparison with that of the O'Brien-Fleming type, which is widely accepted for confirmatory trials.

WHAT IS NEW AND CONCLUSION

We recommend setting the proportion of events in the interim analysis at 50% when the O'Brien-Fleming type α-spending function is used in confirmatory trials to reduce the risk of overestimation. In contrast, the Pocock type boundary could be used in explanatory trials.

Conditional estimation using prior information in 2-stage group sequential designs assuming asymptotic normality when the trial terminated early

Two-stage designs are widely used to determine whether a clinical trial should be terminated early. In such trials, a maximum likelihood estimate is often adopted to describe the difference in efficacy between the experimental and reference treatments; however, this method is known to display conditional bias. To reduce such bias, a conditional mean-adjusted estimator (CMAE) has been proposed, although the remaining bias may be nonnegligible when a trial is stopped for efficacy at the interim analysis. We propose a new estimator for adjusting the conditional bias of the treatment effect by extending the idea of the CMAE. This estimator is calculated by weighting the maximum likelihood estimate obtained at the interim analysis and the effect size prespecified when calculating the sample size. We evaluate the performance of the proposed estimator through analytical and simulation studies in various settings in which a trial is stopped for efficacy or futility at the interim analysis. We find that the conditional bias of the proposed estimator is smaller than that of the CMAE when the information time at the interim analysis is small. In addition, the mean-squared error of the proposed estimator is also smaller than that of the CMAE. In conclusion, we recommend the use of the proposed estimator for trials that are terminated early for efficacy or futility.

Underestimation of treatment effects in sequentially monitored clinical trials that did not stop early for benefit

In recent years, there has been a prominent discussion in the literature about the potential for overestimation of the treatment effect when a clinical trial stops at an interim analysis due to the experimental treatment showing a benefit over the control. However, there has been much less attention paid to the converse issue, namely, that sequentially monitored clinical trials which did not stop early for benefit tend to underestimate the treatment effect. In meta-analyses of many studies, these two sources of bias will tend to balance each other to produce an unbiased estimate of the treatment effect. However, for the interpretation of a single study in isolation, underestimation due to interim analysis may be an important consideration. In this paper, we discuss the nature of this underestimation, including theoretical and simulation results demonstrating that it can be substantial in some contexts. Furthermore, we show how a conditional approach to estimation, in which we condition on the study reaching its final analysis, may be used to validly inflate the observed treatment difference from a sequentially monitored clinical trial. Expressions for the conditional bias and information are derived, and these are used in supplied R code that computes the bias-adjusted estimate and an associated confidence interval. As well as simulation results demonstrating the validity of the methods, we present a data analysis example from a pivotal clinical trial in cardiovascular disease. The methods will be most useful when an unbiased treatment effect estimate is critical, such as in cost-effectiveness analysis or risk prediction.

Comparison of conditional bias-adjusted estimators for interim analysis in clinical trials with survival data

Group sequential designs are widely used in clinical trials to determine whether a trial should be terminated early. In such trials, maximum likelihood estimates are often used to describe the difference in efficacy between the experimental and reference treatments; however, these are well known for displaying conditional and unconditional biases. Established bias-adjusted estimators include the conditional mean-adjusted estimator (CMAE), conditional median unbiased estimator, conditional uniformly minimum variance unbiased estimator (CUMVUE), and weighted estimator. However, their performances have been inadequately investigated. In this study, we review the characteristics of these bias-adjusted estimators and compare their conditional bias, overall bias, and conditional mean-squared errors in clinical trials with survival endpoints through simulation studies. The coverage probabilities of the confidence intervals for the four estimators are also evaluated. We find that the CMAE reduced conditional bias and showed relatively small conditional mean-squared errors when the trials terminated at the interim analysis. The conditional coverage probability of the conditional median unbiased estimator was well below the nominal value. In trials that did not terminate early, the CUMVUE performed with less bias and an acceptable conditional coverage probability than was observed for the other estimators. In conclusion, when planning an interim analysis, we recommend using the CUMVUE for trials that do not terminate early and the CMAE for those that terminate early. Copyright © 2017 John Wiley & Sons, Ltd.

Quantifying over-estimation in early stopped clinical trials and the "freezing effect" on subsequent research

BACKGROUND

Despite the wide use of the design with statistical stopping guidelines to stop a randomized clinical trial early for efficacy, there are unsettled debates of potential harmful consequences of such designs. These concerns include the possible over-estimation of treatment effects in early stopped trials and a newer argument of a "freezing effect" that will halt future randomized clinical trials on the same comparison since an early stopped trial represents an effective declaration that randomization to the unfavored arm is unethical. The purpose of this study is to determine the degree of bias in designs that allow for early stopping and to assess the impact on estimation if indeed future experimentation is "frozen" by an early stopped trial.

METHODS

We perform simulations to study the effect of early stopping. We simulate a collection of trials and contrast the treatment-effect estimates (risk differences and ratios) with the simulation truth. Simulations consider various scenarios of between-study variation, including an empirically derived distribution of effects from the clinical literature.

RESULTS

Across the trials whose true effects are sampled from a uniform distribution, estimates from trials that stop early for efficacy deviate minimally from the simulation truth (median bias of the estimate of risk difference is 0.005). Over-estimation becomes appreciable only when the true effect is close to the null value 0 (median bias of the risk difference estimate is 0.04) or when stopping happens with 40% information or less; however, stopping under these situations is rare. We also find slight reverse bias of the estimated treatment effect (median bias of the risk difference estimate is -0.002) among trials that do not cross the early stopping boundaries but continue to the final analysis. Similar results occur with relative risk estimates. In contrast, Bayesian estimation of the treatment effect shrinks the estimate from trials stopping early and pulls back under-estimation from completed trials, largely rectifying any over-estimation among trials that terminate early. Regarding the so-called freezing effect, the pooled effects from meta-analyses that include truncated randomized clinical trials show an unimportant deviation from the true value, even when no subsequent trials are conducted after a truncated randomized clinical trial.

CONCLUSION

Group sequential designs with stopping rules seek to minimize exposure of patients to a disfavored therapy and speed dissemination of results, and such designs do not lead to materially biased estimates. The likelihood and magnitude of a "freezing effect" is minimal. Superiority demonstrated in a randomized clinical trial stopping early and designed with appropriate statistical stopping rules is likely a valid inference, even if the estimate may be slightly inflated.

Conditional estimation after a two-stage diagnostic biomarker study that allows early termination for futility

Many biomarkers identified in marker discovery are shown to have inadequate performance in validation studies. This motivates the use of group sequential designs that allow early termination for futility. However, an option for early termination will lead to biased estimates for studies that reach full enrollment. We propose conditional estimators and confidence intervals that correct for this bias assuming that an unadjusted estimator exists that has an independent increments covariance structure. The proposed estimators and confidence intervals are applied to conditional estimation of the receiver operating characteristic curve and the positive predictive value curve after a two-stage study that allows early termination for futility, and their performance is evaluated by simulation.

Group sequential and adaptive designs - a review of basic concepts and points of discussion

In recent times, group sequential and adaptive designs for clinical trials have attracted great attention from industry, academia and regulatory authorities. These designs allow analyses on accumulating data - as opposed to classical, "fixed-sample" statistics. The rapid development of a great variety of statistical procedures is accompanied by a lively debate on their potential merits and shortcomings. The purpose of this review article is to ease orientation in both respects. First, we provide a concise overview of the essential technical concepts, with special emphasis on their interrelationships. Second, we give a structured review of the current controversial discussion on practical issues, opportunities and challenges of these new designs.

Estimating odds ratios in genome scans: an approximate conditional likelihood approach

In modern whole-genome scans, the use of stringent thresholds to control the genome-wide testing error distorts the estimation process, producing estimated effect sizes that may be on average far greater in magnitude than the true effect sizes. We introduce a method, based on the estimate of genetic effect and its standard error as reported by standard statistical software, to correct for this bias in case-control association studies. Our approach is widely applicable, is far easier to implement than competing approaches, and may often be applied to published studies without access to the original data. We evaluate the performance of our approach via extensive simulations for a range of genetic models, minor allele frequencies, and genetic effect sizes. Compared to the naive estimation procedure, our approach reduces the bias and the mean squared error, especially for modest effect sizes. We also develop a principled method to construct confidence intervals for the genetic effect that acknowledges the conditioning on statistical significance. Our approach is described in the specific context of odds ratios and logistic modeling but is more widely applicable. Application to recently published data sets demonstrates the relevance of our approach to modern genome scans.