Closed Testing in Pharmaceutical Research: Historical and Recent Developments

Optimal multiple testing and design in clinical trials

A central goal in designing clinical trials is to find the test that maximizes power (or equivalently minimizes required sample size) for finding a false null hypothesis subject to the constraint of type I error. When there is more than one test, such as in clinical trials with multiple endpoints, the issues of optimal design and optimal procedures become more complex. In this paper, we address the question of how such optimal tests should be defined and how they can be found. We review different notions of power and how they relate to study goals, and also consider the requirements of type I error control and the nature of the procedures. This leads us to an explicit optimization problem with objective and constraints that describe its specific desiderata. We present a complete solution for deriving optimal procedures for two hypotheses, which have desired monotonicity properties, and are computationally simple. For some of the optimization formulations this yields optimal procedures that are identical to existing procedures, such as Hommel's procedure or the procedure of Bittman et al. (2009), while for other cases it yields completely novel and more powerful procedures than existing ones. We demonstrate the nature of our novel procedures and their improved power extensively in a simulation and on the APEX study (Cohen et al., 2016).

Proof of Concept: Drug Selection? Or Dose Selection? Thoughts on Multiplicity Issues

In new drug development process, one of the most important milestones for a drug candidate is to establish Proof of Concept (PoC) at early Phase II stage. Among many challenges in PoC clinical trial design and analysis, the application of multiplicity comparison procedures (MCP) is frequently discussed when multiple doses or drugs are included in one PoC study. In such discussion, one fundamental question of applying multiplicity adjustment is which error one should consider to control and at what level. Should it be the experiment-wise error or the compound-wise error? In this paper, the multiplicity issues in two cases of PoC studies are used as examples to discuss the concept of different types of error and the level of the error rate control. With a clear understanding of the type of error and error rate control, the debate of applications of the multiplicity adjustment procedures in the PoC studies can be reconciled.

An improved closed procedure for testing multiple hypotheses

Clinical trials routinely involve multiple hypothesis testing. The closed testing procedure (CTP) is a fundamental principle in testing multiple hypotheses. This article presents an improved CTP in which intersection hypotheses can be tested at a level greater than α such that the control of the familywise error rate at level α remains. Consequently, our method uniformly improves the power of discovering false hypotheses over the original CTP. We illustrate that an improvement by our method exists for many commonly used tests. An empirical study on the effectiveness of a glucose-lowering drug is provided.

A graphical causal model for resolving species identity effects and biodiversity-ecosystem function correlations

Identifying and clearly communicating the drivers of ecosystem function is a crucially important goal for both basic and applied ecology. This has proven difficult because the putative causes (e.g., environment, species identity, biodiversity, and functional traits) are numerous and correlated. The problem is exacerbated by a lack of a formal framework for unambiguously relating theoretical language to precise, quantitative expressions of that language. Using a formal framework for the graphical expression of complex causal hypotheses, we developed a causal diagram of the concepts required to comprehensively test whether hypothesized sets of functional traits mediate the relationship between community structure and ecosystem function. We then used causal analysis, simulations, and field data to develop and test analytical strategies for understanding how community structure influences ecosystem functions via functional traits. Formal causal analysis showed that biodiversity-ecosystem function correlations are noncausal associations. Using simulations, we showed how biodiversity correlations and species identity effects can arise from misspecification or incomplete mediation by functional trait composites. We also found that different types of model misspecification result in different patterns of residuals, which may be used to diagnose gaps in functional trait hypotheses. Treating the model misspecifications eliminated associations between species identity or biodiversity and ecosystem function. Finally, we provide an example of the analysis of field data to demonstrate how to use these insights to conduct a research program that has the goal of understanding the mechanistic trait relationships that link community structure to ecosystem function.

Generalized multiple contrast tests in dose-response studies

In the process of developing drugs, proof-of-concept studies can be helpful in determining whether there is any evidence of a dose-response relationship. A global test for this purpose that has gained popularity is a component of the multiple comparisons procedure with modeling techniques (MCP-Mod), which involves the specification of a candidate set of several plausible dose-response models. For each model, a test is performed for significance of an optimally chosen contrast among the sample means. An overall P-value is obtained from the distribution of the maximum of the contrast statistics. This is equivalent to basing the test on the minimum of the P-values arising from these contrast statistics and, hence, can be viewed as a method for combining dependent P-values. We generalize this idea to the use of different statistics for combining the dependent P-values, such as Fisher's combination method or the inverse normal combination method. Simulation studies show that the generalized multiple contrast tests (GMCTs) based on the Fisher and inverse normal methods are generally more powerful than the MCP-Mod procedure based on the minimum of the P-values except for cases where the true dose-response model is, in a sense, near the extremes of the candidate set of dose-response models. The proposed GMCTs can also be used for model selection and dosage selection by employing a closed testing procedure.

Fast closed testing for exchangeable local tests

Multiple hypothesis testing problems arise naturally in science. This note introduces a new fast closed testing method for multiple testing which controls the familywise error rate. Controlling the familywise error rate is state-of-the-art in many important application areas and is preferred over false discovery rate control for many reasons, including that it leads to stronger reproducibility. The closure principle rejects an individual hypothesis if all global nulls of subsets containing it are rejected using some test statistics. It takes exponential time in the worst case. When the tests are symmetric and monotone, the proposed method is an exact algorithm for computing the closure, is quadratic in the number of tests, and is linear in the number of discoveries. Our framework generalizes most examples of closed testing, such as Holm’s method and the Bonferroni method. As a special case of the method, we propose the Simes and higher criticism fusion test, which is powerful both for detecting a few strong signals and for detecting many moderate signals.

Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing

Closed testing procedures are classically used for familywise error rate control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion in all subsets of the hypotheses, allowing for inference robust to post hoc selection of subsets. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut and use it to investigate the power of this approach when the number of hypotheses goes to infinity. We show that if a minimal level of signal is present, the average power to detect false hypotheses at any desired false discovery proportion does not vanish. Additionally, we show that the confidence bounds for false discovery proportion are consistent estimators for the true false discovery proportion for every nonvanishing subset. We also show close connections between Simes-based closed testing and the procedure of Benjamini and Hochberg.

Closed testing using surrogate hypotheses with restricted alternatives

Introduction

The closed testing principle provides strong control of the type I error probabilities of tests of a set of hypotheses that are closed under intersection such that a given hypothesis H can only be tested and rejected at level α if all intersection hypotheses containing that hypothesis are also tested and rejected at level α. For the higher order hypotheses, multivariate tests (> 1df) are generally employed. However, such tests are directed to an omnibus alternative hypothesis of a difference in any direction for any component that may be less meaningful than a test directed against a restricted alternative hypothesis of interest.

Methods

Herein we describe applications of this principle using an α-level test of a surrogate hypothesis H˜ such that the type I error probability is preserved if H⇒H˜ such that rejection of H˜ implies rejection of H. Applications include the analysis of multiple event times in a Wei-Lachin test against a one-directional alternative, a test of the treatment group difference in the means of K repeated measures using a 1 df test of the difference in the longitudinal LSMEANS, and analyses within subgroups when a test of treatment by subgroup interaction is significant. In such cases the successive higher order surrogate tests can be aimed at detecting parameter values that fall within a more desirable restricted subspace of the global alternative hypothesis parameter space.

Conclusion

Closed testing using α-level tests of surrogate hypotheses will protect the type I error probability and detect specific alternatives of interest, as opposed to the global alternative hypothesis of any difference in any direction.

Identification of potential carcinogenic and chemopreventive effects of prescription drugs: a protocol for a Norwegian registry-based study

INTRODUCTION

Surveillance of unintended effects of pharmaceuticals (pharmacovigilance or drug safety) is crucial, as knowledge of rare or late side effects is limited at the time of the introduction of new medications into the market. Side effects of drugs may involve increased or decreased risk of cancer, but these typically appear after a long induction period. This fact, together with low incidences of many cancer types, limits the usefulness of traditional pharmacovigilance strategies, primarily based on spontaneous reporting of adverse events, to identify associations between drug use and cancer risk. Postmarketing observational pharmacoepidemiological studies are therefore crucial in the evaluation of drug-cancer associations.

METHODS AND ANALYSIS

The main data sources in this project will be the Norwegian Prescription Database and the Cancer Registry of Norway. The underlying statistical model will be based on a multiple nested case-control design including all adult (~200 000) incident cancer cases within the age-range 18-85 years from 2007 through 2015 in Norway as cases. 10 cancer-free population controls will be individually matched to these cases with respect to birth year, sex and index date (date of cancer diagnosis). Drug exposure will be modelled as chronic user/non-user by counting prescriptions, and cumulative use by summarising all dispensions' daily defined doses over time. Conditional logistic regression models adjusted for comorbidity (National Patient Register), socioeconomic parameters (Statistics Norway), concomitant drug use and, for female cancers, reproduction data (Medical Birth Registry), will be applied to identify drug-use-cancer-risk associations.

ETHICS AND DISSEMINATION

The study is approved by the regional ethical committee and the Norwegian data protection authority. Results of the initial screening step and analysis pipeline will be described in a key paper. Subsequent papers will report the evaluation of identified signals in replication studies. Results will be published in peer-reviewed journals, at scientific conferences and through press releases.