Reconsidering some aspects of the two-trials paradigm

The assessment of replicability using the sum of p-values

Statistical significance of both the original and the replication study is a commonly used criterion to assess replication attempts, also known as the two-trials rule in drug development. However, replication studies are sometimes conducted although the original study is non-significant, in which case Type-I error rate control across both studies is no longer guaranteed. We propose an alternative method to assess replicability using the sum of p -values from the two studies. The approach provides a combined p -value and can be calibrated to control the overall Type-I error rate at the same level as the two-trials rule but allows for replication success even if the original study is non-significant. The unweighted version requires a less restrictive level of significance at replication if the original study is already convincing which facilitates sample size reductions of up to 10%. Downweighting the original study accounts for possible bias and requires a more stringent significance level and larger sample sizes at replication. Data from four large-scale replication projects are used to illustrate and compare the proposed method with the two-trials rule, meta-analysis and Fisher's combination method.

New strategies for confirmatory testing of secondary hypotheses on combined data from multiple trials

BACKGROUND

Pivotal evidence of efficacy of a new drug is typically generated by (at least) two clinical trials which independently provide statistically significant and mutually corroborating evidence of efficacy based on a primary endpoint. In this situation, showing drug effects on clinically important secondary objectives can be demanding in terms of sample size requirements. Statistically efficient methods to power for such endpoints while controlling the Type I error are needed.

METHODS

We review existing strategies for establishing claims on important but sample size-intense secondary endpoints. We present new strategies based on combined data from two independent, identically designed and concurrent trials, controlling the Type I error at the submission level. We explain the methodology and provide three case studies.

RESULTS

Different strategies have been used for establishing secondary claims. One new strategy, involving a protocol planned analysis of combined data across trials, and controlling the Type I error at the submission level, is particularly efficient. It has already been successfully used in support of label claims. Regulatory views on this strategy differ.

CONCLUSIONS

Inference on combined data across trials is a useful approach for generating pivotal evidence of efficacy for important but sample size-intense secondary endpoints. It requires careful preparation and regulatory discussion.

A Generalization of the Two Trials Paradigm

The two trials paradigm plays a prominent role in drug development and has been widely and controversially discussed. Its purpose is to ensure replicability or substantiation of study results. This note investigates a simple generalization of the paradigm to more than two trials that preserves the project wise type-I error rate and power.

Replicability of studies following a dual-criterion design

Replicability of results is regarded as the corner stone of science. Recent research seems to raise doubts about whether this requirement is generally fulfilled. Often, replicability of results is defined as repeating a statistically significant result. However, since significance may not imply scientific relevance, dual-criterion study designs that take both aspects into account have been proposed and investigated during the last decade. Originally developed for proof-of-concept trials, the design could be appropriate for phase III trials as well. In fact, a dual-criterion design has been requested for COVID-19 vaccine applications by major health authorities. In this article, replicability of dual-criterion designs is investigated. It turns out that the probability to replicate a significant and relevant result can become as low as 0.5. The replication probability increases if the effect estimator exceeds the minimum relevant effect in the original study by an extra amount.

Combined-Indications Significance Level of Multiple Related Indications Developed Simultaneously

Background:

Typically, regulatory requirements include 2 confirmatory studies, each at a 1-sided .025 significance level, for a medicine to be approved for a specific indication. When the same medicine has been approved in related indications, 1 confirmatory study at a 1-sided .025 significance level could constitute adequate evidence of efficacy for a new indication.

Methods:

This article does not contain any studies with human or animal subjects performed by any of the authors. For multiple related indications developed simultaneously to constitute sufficient evidence of clinical efficacy, the combined-studies significance level can be set at the same level as if those indications are developed sequentially.

Results:

This article establishes possible strategies to develop a few related indications at the same time for marketing registration approval, maintaining a desired combined-studies significance level; for example, 1-sided .0000156 for 2 indications, with 1 option having each indication assessed with 1 confirmatory study at .00395 significance level.

Conclusion:

It is possible to develop a few indications at the same time for marketing registration approval, where the combined-studies significance level is less stringent than that of the usual paradigm with 2 confirmatory studies each at 1-sided .025 significance level for every indication.

Optimal sample size allocation and go/no-go decision rules for phase II/III programs where several phase III trials are performed

The conduct of phase II and III programs is costly, time-consuming and, due to high failure rates in late development stages, risky. There is a strong connection between phase II and III trials as the go/no-go decision and the sample size chosen for phase III are based on the results observed in phase II. An integrated planning of phase II and III is therefore reasonable. The success of phase II/III programs crucially depends on the allocation of the resources to phase II and III in terms of sample size and the rule applied to decide whether to stop or to proceed with phase III. Recently, a utility-based approach was proposed, where optimal planning of phase II/III programs is achieved by taking fixed and variable costs of the drug development program and potential gains after a successful launch into account. However, this method is restricted to programs with a single phase III trial, while regulatory authorities usually require statistical significance in two or more phase III trials. We present a generalization of this procedure to programs where two or more phase III trials are performed. Optimal phase II sample sizes and go/no-go decision rules are provided for time-to-event outcomes and cases, where at least one, two, or three phase III trials need to be successful. Different drug development program strategies (e.g. one large vs. two phase III trials) are compared within these different cases. Application to practical examples typically met in oncology trials illustrates the proposed method.

Power and sample size when multiple endpoints are considered

A common approach to analysing clinical trials with multiple outcomes is to control the probability for the trial as a whole of making at least one incorrect positive finding under any configuration of true and false null hypotheses. Popular approaches are to use Bonferroni corrections or structured approaches such as, for example, closed-test procedures. As is well known, such strategies, which control the family-wise error rate, typically reduce the type I error for some or all the tests of the various null hypotheses to below the nominal level. In consequence, there is generally a loss of power for individual tests. What is less well appreciated, perhaps, is that depending on approach and circumstances, the test-wise loss of power does not necessarily lead to a family wise loss of power. In fact, it may be possible to increase the overall power of a trial by carrying out tests on multiple outcomes without increasing the probability of making at least one type I error when all null hypotheses are true. We examine two types of problems to illustrate this. Unstructured testing problems arise typically (but not exclusively) when many outcomes are being measured. We consider the case of more than two hypotheses when a Bonferroni approach is being applied while for illustration we assume compound symmetry to hold for the correlation of all variables. Using the device of a latent variable it is easy to show that power is not reduced as the number of variables tested increases, provided that the common correlation coefficient is not too high (say less than 0.75). Afterwards, we will consider structured testing problems. Here, multiplicity problems arising from the comparison of more than two treatments, as opposed to more than one measurement, are typical. We conduct a numerical study and conclude again that power is not reduced as the number of tested variables increases.

Challenges in implementing adaptive designs: comments on the viewpoints expressed by regulatory statisticians

This is a discussion of the following three papers appearing in this special issue on adaptive designs: 'FDA's critical path initiative: A perspective on contributions of biostatistics' by Robert T. O'Neill; 'A regulatory view on adaptive/flexible clinical trial design' by H. M. James Hung, Robert T. O'Neill, Sue-Jane Wang and John Lawrence; and 'Confirmatory clinical trials with an adaptive design' by Armin Koch.