Comments on ‘Adaptive increase in sample size when interim results are promising: A practical guide with examples’

Optimal adaptive promising zone designs

We develop optimal decision rules for sample size re-estimation in two-stage adaptive group sequential clinical trials. It is usual for the initial sample size specification of such trials to be adequate to detect a realistic treatment effect δ a with good power, but not sufficient to detect the smallest clinically meaningful treatment effect δ min . Moreover it is difficult for the sponsors of such trials to make the up-front commitment needed to adequately power a study to detect δ min . It is easier to justify increasing the sample size if the interim data enter a so-called "promising zone" that ensures with high probability that the trial will succeed. We have considered promising zone designs that optimize unconditional power and promising zone designs that optimize conditional power and have discussed the tension that exists between these two objectives. Where there is reluctance to base the sample size re-estimation rule on the parameter δ min we propose a Bayesian option whereby a prior distribution is assigned to the unknown treatment effect δ , which is then integrated out of the objective function with respect to its posterior distribution at the interim analysis.

Sample size re-estimation for pivotal clinical trials

It is well known that if the hypothesis test is left unchanged, the Type I error rate may be inflated for sample size re-estimation (SSR) designs. To address this issue, three main approaches have been proposed in the literature: combination test, conditional error and conventional test with sample size increase in the allowable region (AR) only. These three seemingly different approaches are in fact connected. For each combination test, there is a corresponding conditional error function and AR. Designing adaptation rules in this AR with conventional test guarantees the Type I error rate control but at the same time always leads to smaller power comparing to the corresponding combination test (or conditional error) approach. In cases where conventional test is still preferable, step-wise type adaptation rules that do not fully reside in the AR can be alternatively considered. We believe controversies in the statistical community on the efficiency comparisons between group sequential (GS) and SSR design stem partially from the misalignment of performance metrics and conditional versus unconditional evaluations. We advocate summary metrics, such as median, variance or tail probabilities of the sample size in addition to expectation and personalizing efficiency definition for each trial sponsor. Conditional metrics by favorable, promising and unfavorable zones of the interim results provide additional insights and should always be incorporated into the decision-making process.

Sample size estimation and re-estimation of cluster randomized controlled trials for real-time feedback, debriefing, and retraining system of cardiopulmonary resuscitation for out-of-hospital cardiac arrests

Background

In cluster randomized controlled trials (RCTs) for emergency medical services (EMS) system, we encounter the situation that the actual cluster size and ratio of allocated patients between two groups eventually differ from those used for sample size estimation because of the nature of patient enrollment. In such trials, estimations of effect size of test intervention and intra-cluster correlation coefficient (ICC) used for sample size estimation are also difficult. To improve efficient management on clinical cluster RCTs, we need to understand the effect of such inconsistencies of the design parameters on the type I error rate and statistical power of testing.

Methods

We planned the trial which evaluated the 1-month favorable neurological survival of out-of-hospital cardiac arrest patients with or without real-time feedback, debriefing, and retraining system by EMS personnel. Under the conditions that we possibly encountered in this trial, we examined the effect of inconsistencies in the actual ICC, cluster size, and ratio of patient allocation with those expected for sample size estimation on the type I error rate and power, using simulation studies. We further investigated the contribution of incorporating sample size re-estimation, based on the results of interim analysis of the trial, on the power increase.

Results

This simulation study showed that the inconsistencies of cluster size and patient allocation ratio decreased the power by 5-10% in some cases. In addition, the power decreased by 3-4% when the actual ICC was larger than that expected for sample size estimation. Furthermore, the use of a generalized estimating equation method to evaluate the difference in the 1-month favorable neurological survival between two groups caused inflation of type I error rate. Finally, the increase in power by incorporating sample size re-estimation was limited.

Conclusions

We identified remarkable effects of sample size estimation and re-estimations in a cluster RCT for real-time feedback, debriefing, and retraining system of cardiopulmonary resuscitation for out-of-hospital cardiac arrests. The estimation of design parameters for sample size estimation is generally challenging in cluster RCTs for EMS system; therefore, it is important to conduct a trial simulation that assesses the statistical performances under sample sizes based on the various expected values of the design parameters before beginning the trial.

Optimal promising zone designs

Clinical trials with adaptive sample size reassessment based on an unblinded analysis of interim results are perhaps the most popular class of adaptive designs (see Elsäßer et al., 2007). Such trials are typically designed by prespecifying a zone for the interim test statistic, termed the promising zone, along with a decision rule for increasing the sample size within that zone. Mehta and Pocock (2011) provided some examples of promising zone designs and discussed several procedures for controlling their type‐1 error. They did not, however, address how to choose the promising zone or the corresponding sample size reassessment rule, and proposed instead that the operating characteristics of alternative promising zone designs could be compared by simulation. Jennison and Turnbull (2015) developed an approach based on maximizing expected utility whereby one could evaluate alternative promising zone designs relative to a gold‐standard optimal design. In this paper, we show how, by eliciting a few preferences from the trial sponsor, one can construct promising zone designs that are both intuitive and achieve the Jennison and Turnbull (2015) gold‐standard for optimality.

Sample size adjustment designs with time-to-event outcomes: A caution

BACKGROUND

Sample size adjustment designs, which allow increasing the study sample size based on interim analysis of outcome data from a randomized clinical trial, have been increasingly promoted in the biostatistical literature. Although it is recognized that group sequential designs can be at least as efficient as sample size adjustment designs, many authors argue that a key advantage of these designs is their flexibility; interim sample size adjustment decisions can incorporate information and business interests external to the trial. Recently, Chen et al. (Clinical Trials 2015) considered sample size adjustment applications in the time-to-event setting using a design (CDL) that limits adjustments to situations where the interim results are promising. The authors demonstrated that while CDL provides little gain in unconditional power (versus fixed-sample-size designs), there is a considerable increase in conditional power for trials in which the sample size is adjusted.

METHODS

In time-to-event settings, sample size adjustment allows an increase in the number of events required for the final analysis. This can be achieved by either (a) following the original study population until the additional events are observed thus focusing on the tail of the survival curves or (b) enrolling a potentially large number of additional patients thus focusing on the early differences in survival curves. We use the CDL approach to investigate performance of sample size adjustment designs in time-to-event trials.

RESULTS

Through simulations, we demonstrate that when the magnitude of the true treatment effect changes over time, interim information on the shape of the survival curves can be used to enrich the final analysis with events from the time period with the strongest treatment effect. In particular, interested parties have the ability to make the end-of-trial treatment effect larger (on average) based on decisions using interim outcome data. Furthermore, in "clinical null" cases where there is no benefit due to crossing survival curves, the sample size adjustment design is shown to increase the probability of recommending an ineffective therapy.

CONCLUSION

Access to interim information on the shape of the survival curves may jeopardize the perceived integrity of trials using sample size adjustment designs. Therefore, given the lack of efficiency advantage over group sequential designs, sample size adjustment designs in time-to-event settings remain unjustified.

An evaluation of increasing sample size based on conditional power

We evaluate properties of sample size re-estimation (SSR) designs similar to the promising zone design considered by Mehta and Pocock (2011). We evaluate these designs under the assumption of a true effect size of 1.1 down to 0.4 of the protocol-specified effect size by six measures: 1. The probability of a sample size increase, 2. The mean proportional increase in sample size given an increase; 3 and 4. The mean true conditional power with and without a sample size increase; 5 and 6. The expected increase in sample size and power due to the SSR procedure. These measures show the probability of a sample size increase and the cost/benefit for given true effect sizes, particularly when the SSR may either be pursuing a small effect size of little clinical importance or be unnecessary when the true effect size is close to the protocol-specified effect size. The results show the clear superiority of conducting the SSR late in the study and the inefficiency of a mid-study SSR. The results indicate that waiting until late in the study for the SSR yields a smaller, better targeted set of studies with a greater increase in overall power than a mid-study SSR.

Adaptive designs from a Data Safety Monitoring Board perspective: Some controversies and some case studies

This article describes vignettes concerning interactions with Data Safety Monitoring Boards during the design and monitoring of some clinical trials with an adaptive design. Most reflect personal experiences by the author.

Statistical challenges in a regulatory review of cardiovascular and CNS clinical trials

There are several challenging statistical problems identified in the regulatory review of large cardiovascular (CV) clinical outcome trials and central nervous system (CNS) trials. The problems can be common or distinct due to disease characteristics and the differences in trial design elements such as endpoints, trial duration, and trial size. In schizophrenia trials, heavy missing data is a big problem. In Alzheimer trials, the endpoints for assessing symptoms and the endpoints for assessing disease progression are essentially the same; it is difficult to construct a good trial design to evaluate a test drug for its ability to slow the disease progression. In CV trials, reliance on a composite endpoint with low event rate makes the trial size so large that it is infeasible to study multiple doses necessary to find the right dose for study patients. These are just a few typical problems. In the past decade, adaptive designs were increasingly used in these disease areas and some challenges occur with respect to that use. Based on our review experiences, group sequential designs (GSDs) have borne many successful stories in CV trials and are also increasingly used for developing treatments targeting CNS diseases. There is also a growing trend of using more advanced unblinded adaptive designs for producing efficacy evidence. Many statistical challenges with these kinds of adaptive designs have been identified through our experiences with the review of regulatory applications and are shared in this article.

Testing multiple primary endpoints in clinical trials with sample size adaptation

In this paper, we propose a design that uses a short-term endpoint for accelerated approval at interim analysis and a long-term endpoint for full approval at final analysis with sample size adaptation based on the long-term endpoint. Two sample size adaptation rules are compared: an adaptation rule to maintain the conditional power at a prespecified level and a step function type adaptation rule to better address the bias issue. Three testing procedures are proposed: alpha splitting between the two endpoints; alpha exhaustive between the endpoints; and alpha exhaustive with improved critical value based on correlation. Family-wise error rate is proved to be strongly controlled for the two endpoints, sample size adaptation, and two analysis time points with the proposed designs. We show that using alpha exhaustive designs greatly improve the power when both endpoints are effective, and the power difference between the two adaptation rules is minimal. The proposed design can be extended to more general settings.

Some challenges with statistical inference in adaptive designs

Adaptive designs have generated a great deal of attention to clinical trial communities. The literature contains many statistical methods to deal with added statistical uncertainties concerning the adaptations. Increasingly encountered in regulatory applications are adaptive statistical information designs that allow modification of sample size or related statistical information and adaptive selection designs that allow selection of doses or patient populations during the course of a clinical trial. For adaptive statistical information designs, a few statistical testing methods are mathematically equivalent, as a number of articles have stipulated, but arguably there are large differences in their practical ramifications. We pinpoint some undesirable features of these methods in this work. For adaptive selection designs, the selection based on biomarker data for testing the correlated clinical endpoints may increase statistical uncertainty in terms of type I error probability, and most importantly the increased statistical uncertainty may be impossible to assess.

Adaptive sample size modification in clinical trials: start small then ask for more?

We consider sample size re-estimation in a clinical trial, in particular when there is a significant delay before the measurement of patient response. Mehta and Pocock have proposed methods in which sample size is increased when interim results fall in a 'promising zone' where it is deemed worthwhile to increase conditional power by adding more subjects. Our analysis reveals potential pitfalls in applying this approach. Mehta and Pocock use results of Chen, DeMets and Lan to identify when increasing sample size, but applying a conventional level α significance test at the end of the trial does not inflate the type I error rate: we have found the greatest gains in power per additional observation are liable to lie outside the region defined by this method. Mehta and Pocock increase sample size to achieve a particular conditional power, calculated under the current estimate of treatment effect: this leads to high increases in sample size for a small range of interim outcomes, whereas we have found it more efficient to make moderate increases in sample size over a wider range of cases. If the aforementioned pitfalls are avoided, we believe the broad framework proposed by Mehta and Pocock is valuable for clinical trial design. Working in this framework, we propose sample size rules that apply explicitly the principle of adding observations when they are most beneficial. The resulting trial designs are closely related to efficient group sequential tests for a delayed response proposed by Hampson and Jennison.

Two-stage designs for cross-over bioequivalence trials

The topic of applying two-stage designs in the field of bioequivalence studies has recently gained attention in the literature and in regulatory guidelines. While there exists some methodological research on the application of group sequential designs in bioequivalence studies, implementation of adaptive approaches has focused up to now on superiority and non-inferiority trials. Especially, no comparison of the features and performance characteristics of these designs has been performed, and therefore, the question of which design to employ in this setting remains open. In this paper, we discuss and compare 'classical' group sequential designs and three types of adaptive designs that offer the option of mid-course sample size recalculation. A comprehensive simulation study demonstrates that group sequential designs can be identified, which show power characteristics that are similar to those of the adaptive designs but require a lower average sample size. The methods are illustrated with a real bioequivalence study example.

A review and re-interpretation of a group-sequential approach to sample size re-estimation in two-stage trials

In this paper, we review the adaptive design methodology of Li et al. (Biostatistics 3:277-287) for two-stage trials with mid-trial sample size adjustment. We argue that it is closer in principle to a group sequential design, in spite of its obvious adaptive element. Several extensions are proposed that aim to make it even more attractive and transparent alternative to a standard (fixed sample size) trial for funding bodies to consider. These enable a cap to be put on the maximum sample size and for the trial data to be analysed using standard methods at its conclusion. The regulatory view of trials incorporating unblinded sample size re-estimation is also discussed.

Adaptive Budgets in Clinical Trials

We consider situations where a drug developer gets access to additional financial resources when a promising result has been observed in a pre-planned interim analysis during a clinical trial which should lead to the registration of the drug. First the option that the drug developer completely puts the additional resources into increasing the second stage sample size has been investigated. If investors invest the more the larger the observed interim effect, this may not be a reasonable strategy: Then additional sample sizes are applied when the conditional power is already very large and hardly any impact on the overall power can be expected. Nevertheless, further reducing the type II error rate in promising situations may be of interest for a drug developer. In a second step, sample size was based on a utility function including the reward of registration (which was allowed to depend on the observed effect size at the end of the trial) and sampling costs. Utility as a function of the sample size may have more than one local maximum, one of them at the lowest per group sample size. For small effects an optimal strategy could be to apply the smallest sample size accepted by regulators.

Adaptive clinical trial designs with pre-specified rules for modifying the sample size: understanding efficient types of adaptation

Adaptive clinical trial design has been proposed as a promising new approach that may improve the drug discovery process. Proponents of adaptive sample size re-estimation promote its ability to avoid 'up-front' commitment of resources, better address the complicated decisions faced by data monitoring committees, and minimize accrual to studies having delayed ascertainment of outcomes. We investigate aspects of adaptation rules, such as timing of the adaptation analysis and magnitude of sample size adjustment, that lead to greater or lesser statistical efficiency. Owing in part to the recent Food and Drug Administration guidance that promotes the use of pre-specified sampling plans, we evaluate alternative approaches in the context of well-defined, pre-specified adaptation. We quantify the relative costs and benefits of fixed sample, group sequential, and pre-specified adaptive designs with respect to standard operating characteristics such as type I error, maximal sample size, power, and expected sample size under a range of alternatives. Our results build on others' prior research by demonstrating in realistic settings that simple and easily implemented pre-specified adaptive designs provide only very small efficiency gains over group sequential designs with the same number of analyses. In addition, we describe optimal rules for modifying the sample size, providing efficient adaptation boundaries on a variety of scales for the interim test statistic for adaptation analyses occurring at several different stages of the trial. We thus provide insight into what are good and bad choices of adaptive sampling plans when the added flexibility of adaptive designs is desired.