1
|
Heller R, Krieger A, Rosset S. Optimal multiple testing and design in clinical trials. Biometrics 2023; 79:1908-1919. [PMID: 35899317 DOI: 10.1111/biom.13726] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2022] [Accepted: 07/22/2022] [Indexed: 02/01/2023]
Abstract
A central goal in designing clinical trials is to find the test that maximizes power (or equivalently minimizes required sample size) for finding a false null hypothesis subject to the constraint of type I error. When there is more than one test, such as in clinical trials with multiple endpoints, the issues of optimal design and optimal procedures become more complex. In this paper, we address the question of how such optimal tests should be defined and how they can be found. We review different notions of power and how they relate to study goals, and also consider the requirements of type I error control and the nature of the procedures. This leads us to an explicit optimization problem with objective and constraints that describe its specific desiderata. We present a complete solution for deriving optimal procedures for two hypotheses, which have desired monotonicity properties, and are computationally simple. For some of the optimization formulations this yields optimal procedures that are identical to existing procedures, such as Hommel's procedure or the procedure of Bittman et al. (2009), while for other cases it yields completely novel and more powerful procedures than existing ones. We demonstrate the nature of our novel procedures and their improved power extensively in a simulation and on the APEX study (Cohen et al., 2016).
Collapse
Affiliation(s)
- Ruth Heller
- Department of Statistics and Operations Research, Tel-Aviv University, Tel Aviv, Israel
| | - Abba Krieger
- Department of Statistics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Saharon Rosset
- Department of Statistics and Operations Research, Tel-Aviv University, Tel Aviv, Israel
| |
Collapse
|
2
|
Li QH, Deng Q, Ting N. Proof of Concept: Drug Selection? Or Dose Selection? Thoughts on Multiplicity Issues. Ther Innov Regul Sci 2021; 55:1001-1005. [PMID: 34028669 PMCID: PMC8142878 DOI: 10.1007/s43441-021-00299-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2020] [Accepted: 04/20/2021] [Indexed: 11/26/2022]
Abstract
In new drug development process, one of the most important milestones for a drug candidate is to establish Proof of Concept (PoC) at early Phase II stage. Among many challenges in PoC clinical trial design and analysis, the application of multiplicity comparison procedures (MCP) is frequently discussed when multiple doses or drugs are included in one PoC study. In such discussion, one fundamental question of applying multiplicity adjustment is which error one should consider to control and at what level. Should it be the experiment-wise error or the compound-wise error? In this paper, the multiplicity issues in two cases of PoC studies are used as examples to discuss the concept of different types of error and the level of the error rate control. With a clear understanding of the type of error and error rate control, the debate of applications of the multiplicity adjustment procedures in the PoC studies can be reconciled.
Collapse
Affiliation(s)
- Qian H Li
- Global Biometrics and Data Sciences, Bristol Myers Squibb - Global Biopharmaceutical Company, NewYork, USA
| | - Qiqi Deng
- Biostatistics and Data Sciences, Boehringer Ingelheim Pharmaceuticals, Inc, Ridgefield, USA
| | - Naitee Ting
- Biostatistics and Data Sciences, Boehringer Ingelheim Pharmaceuticals, Inc, Ridgefield, USA.
| |
Collapse
|
3
|
Goeman JJ, Hemerik J, Solari A. Only closed testing procedures are admissible for controlling false discovery proportions. Ann Stat 2021. [DOI: 10.1214/20-aos1999] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Affiliation(s)
- Jelle J. Goeman
- Department of Biomedical Data Sciences, Leiden University Medical Center
| | - Jesse Hemerik
- Oslo Centre for Biostatistics and Epidemiology, University of Oslo, and Biometris, Wageningen University & Research
| | - Aldo Solari
- Department of Economics, Management and Statistics, University of Milano-Bicocca
| |
Collapse
|
4
|
Gou J. Least Conservative Critical Boundaries of Multiple Hypothesis Testing in a Range of Correlation Values. Stat Biopharm Res 2021. [DOI: 10.1080/19466315.2021.1873842] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
Affiliation(s)
- Jiangtao Gou
- Department of Mathematics and Statistics, Villanova University, Villanova, PA
| |
Collapse
|
5
|
Lu ZH. An improved closed procedure for testing multiple hypotheses. Stat Med 2020; 39:3772-3786. [PMID: 32706424 DOI: 10.1002/sim.8692] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2020] [Revised: 06/18/2020] [Accepted: 06/19/2020] [Indexed: 11/09/2022]
Abstract
Clinical trials routinely involve multiple hypothesis testing. The closed testing procedure (CTP) is a fundamental principle in testing multiple hypotheses. This article presents an improved CTP in which intersection hypotheses can be tested at a level greater than α such that the control of the familywise error rate at level α remains. Consequently, our method uniformly improves the power of discovering false hypotheses over the original CTP. We illustrate that an improvement by our method exists for many commonly used tests. An empirical study on the effectiveness of a glucose-lowering drug is provided.
Collapse
Affiliation(s)
- Zeng-Hua Lu
- University of South Australia, Adelaide, South Australia, Australia
| |
Collapse
|
6
|
Schoolmaster DR, Zirbel CR, Cronin JP. A graphical causal model for resolving species identity effects and biodiversity-ecosystem function correlations. Ecology 2020; 101:e03070. [PMID: 32301506 DOI: 10.1002/ecy.3070] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/19/2019] [Revised: 02/24/2020] [Accepted: 03/16/2020] [Indexed: 11/10/2022]
Abstract
Identifying and clearly communicating the drivers of ecosystem function is a crucially important goal for both basic and applied ecology. This has proven difficult because the putative causes (e.g., environment, species identity, biodiversity, and functional traits) are numerous and correlated. The problem is exacerbated by a lack of a formal framework for unambiguously relating theoretical language to precise, quantitative expressions of that language. Using a formal framework for the graphical expression of complex causal hypotheses, we developed a causal diagram of the concepts required to comprehensively test whether hypothesized sets of functional traits mediate the relationship between community structure and ecosystem function. We then used causal analysis, simulations, and field data to develop and test analytical strategies for understanding how community structure influences ecosystem functions via functional traits. Formal causal analysis showed that biodiversity-ecosystem function correlations are noncausal associations. Using simulations, we showed how biodiversity correlations and species identity effects can arise from misspecification or incomplete mediation by functional trait composites. We also found that different types of model misspecification result in different patterns of residuals, which may be used to diagnose gaps in functional trait hypotheses. Treating the model misspecifications eliminated associations between species identity or biodiversity and ecosystem function. Finally, we provide an example of the analysis of field data to demonstrate how to use these insights to conduct a research program that has the goal of understanding the mechanistic trait relationships that link community structure to ecosystem function.
Collapse
Affiliation(s)
- Donald R Schoolmaster
- US Geological Survey, Wetland and Aquatic Research Center, 700 Cajundome Blvd, Lafayette, Louisiana, 70506, USA.,W.K. Kellogg Biological Station, 3700 E. Gull Lake Dr., Hickory Corners, Michigan, 49060, USA
| | - Chad R Zirbel
- Department of Plant Biology and Program in Ecology, Evolutionary Biology and Behavior, Michigan State University, 612 Wilson Road, East Lansing, Michigan, 48824, USA.,Department of Ecology, Evolution & Behavior, University of Minnesota, Saint Paul, Minnesota, 55108, USA
| | - James Patrick Cronin
- US Geological Survey, Wetland and Aquatic Research Center, 700 Cajundome Blvd, Lafayette, Louisiana, 70506, USA
| |
Collapse
|
7
|
Ma S, McDermott MP. Generalized multiple contrast tests in dose-response studies. Stat Med 2020; 39:757-772. [PMID: 31793014 DOI: 10.1002/sim.8444] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2019] [Revised: 09/19/2019] [Accepted: 11/16/2019] [Indexed: 11/10/2022]
Abstract
In the process of developing drugs, proof-of-concept studies can be helpful in determining whether there is any evidence of a dose-response relationship. A global test for this purpose that has gained popularity is a component of the multiple comparisons procedure with modeling techniques (MCP-Mod), which involves the specification of a candidate set of several plausible dose-response models. For each model, a test is performed for significance of an optimally chosen contrast among the sample means. An overall P-value is obtained from the distribution of the maximum of the contrast statistics. This is equivalent to basing the test on the minimum of the P-values arising from these contrast statistics and, hence, can be viewed as a method for combining dependent P-values. We generalize this idea to the use of different statistics for combining the dependent P-values, such as Fisher's combination method or the inverse normal combination method. Simulation studies show that the generalized multiple contrast tests (GMCTs) based on the Fisher and inverse normal methods are generally more powerful than the MCP-Mod procedure based on the minimum of the P-values except for cases where the true dose-response model is, in a sense, near the extremes of the candidate set of dose-response models. The proposed GMCTs can also be used for model selection and dosage selection by employing a closed testing procedure.
Collapse
Affiliation(s)
- Shiyang Ma
- Department of Biostatistics and Computational Biology, University of Rochester, Rochester, New York
| | - Michael P McDermott
- Department of Biostatistics and Computational Biology, University of Rochester, Rochester, New York
| |
Collapse
|
8
|
Abstract
Summary
Multiple hypothesis testing problems arise naturally in science. This note introduces a new fast closed testing method for multiple testing which controls the familywise error rate. Controlling the familywise error rate is state-of-the-art in many important application areas and is preferred over false discovery rate control for many reasons, including that it leads to stronger reproducibility. The closure principle rejects an individual hypothesis if all global nulls of subsets containing it are rejected using some test statistics. It takes exponential time in the worst case. When the tests are symmetric and monotone, the proposed method is an exact algorithm for computing the closure, is quadratic in the number of tests, and is linear in the number of discoveries. Our framework generalizes most examples of closed testing, such as Holm’s method and the Bonferroni method. As a special case of the method, we propose the Simes and higher criticism fusion test, which is powerful both for detecting a few strong signals and for detecting many moderate signals.
Collapse
Affiliation(s)
- E Dobriban
- Department of Statistics, The Wharton School, University of Pennsylvania, 3730 Walnut Street, Philadelphia, Pennsylvania 19104, USA
| |
Collapse
|
9
|
Goeman JJ, Meijer RJ, Krebs TJP, Solari A. Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing. Biometrika 2019. [DOI: 10.1093/biomet/asz041] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023] Open
Abstract
Summary
Closed testing procedures are classically used for familywise error rate control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion in all subsets of the hypotheses, allowing for inference robust to post hoc selection of subsets. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut and use it to investigate the power of this approach when the number of hypotheses goes to infinity. We show that if a minimal level of signal is present, the average power to detect false hypotheses at any desired false discovery proportion does not vanish. Additionally, we show that the confidence bounds for false discovery proportion are consistent estimators for the true false discovery proportion for every nonvanishing subset. We also show close connections between Simes-based closed testing and the procedure of Benjamini and Hochberg.
Collapse
Affiliation(s)
- Jelle J Goeman
- Biomedical Data Sciences, Leiden University Medical Center, Einthovenweg 20, 2333 ZC Leiden, The Netherlands
| | - Rosa J Meijer
- Data Research Office, Antes, Parnassia Group, Rotterdam, The Netherlands
| | - Thijmen J P Krebs
- Delft University of Technology, Mekelweg 4, CD Delft, The Netherlands
| | - Aldo Solari
- Department of Economics, Management and Statistics, University of Milano-Bicocca, Piazza dell’Ateneo Nuovo 1, 20126 Milano, Italy
| |
Collapse
|
10
|
Closed testing using surrogate hypotheses with restricted alternatives. PLoS One 2019; 14:e0219520. [PMID: 31299051 PMCID: PMC6625735 DOI: 10.1371/journal.pone.0219520] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2019] [Accepted: 06/25/2019] [Indexed: 12/04/2022] Open
Abstract
Introduction The closed testing principle provides strong control of the type I error probabilities of tests of a set of hypotheses that are closed under intersection such that a given hypothesis H can only be tested and rejected at level α if all intersection hypotheses containing that hypothesis are also tested and rejected at level α. For the higher order hypotheses, multivariate tests (> 1df) are generally employed. However, such tests are directed to an omnibus alternative hypothesis of a difference in any direction for any component that may be less meaningful than a test directed against a restricted alternative hypothesis of interest. Methods Herein we describe applications of this principle using an α-level test of a surrogate hypothesis H˜ such that the type I error probability is preserved if H⇒H˜ such that rejection of H˜ implies rejection of H. Applications include the analysis of multiple event times in a Wei-Lachin test against a one-directional alternative, a test of the treatment group difference in the means of K repeated measures using a 1 df test of the difference in the longitudinal LSMEANS, and analyses within subgroups when a test of treatment by subgroup interaction is significant. In such cases the successive higher order surrogate tests can be aimed at detecting parameter values that fall within a more desirable restricted subspace of the global alternative hypothesis parameter space. Conclusion Closed testing using α-level tests of surrogate hypotheses will protect the type I error probability and detect specific alternatives of interest, as opposed to the global alternative hypothesis of any difference in any direction.
Collapse
|
11
|
Andreassen BK, Støer NC, Martinsen JI, Ursin G, Weiderpass E, Thoresen GH, Debernard KB, Karlstad Ø, Pottegard A, Friis S. Identification of potential carcinogenic and chemopreventive effects of prescription drugs: a protocol for a Norwegian registry-based study. BMJ Open 2019; 9:e028504. [PMID: 30962244 PMCID: PMC6500356 DOI: 10.1136/bmjopen-2018-028504] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/28/2022] Open
Abstract
INTRODUCTION Surveillance of unintended effects of pharmaceuticals (pharmacovigilance or drug safety) is crucial, as knowledge of rare or late side effects is limited at the time of the introduction of new medications into the market. Side effects of drugs may involve increased or decreased risk of cancer, but these typically appear after a long induction period. This fact, together with low incidences of many cancer types, limits the usefulness of traditional pharmacovigilance strategies, primarily based on spontaneous reporting of adverse events, to identify associations between drug use and cancer risk. Postmarketing observational pharmacoepidemiological studies are therefore crucial in the evaluation of drug-cancer associations. METHODS AND ANALYSIS The main data sources in this project will be the Norwegian Prescription Database and the Cancer Registry of Norway. The underlying statistical model will be based on a multiple nested case-control design including all adult (~200 000) incident cancer cases within the age-range 18-85 years from 2007 through 2015 in Norway as cases. 10 cancer-free population controls will be individually matched to these cases with respect to birth year, sex and index date (date of cancer diagnosis). Drug exposure will be modelled as chronic user/non-user by counting prescriptions, and cumulative use by summarising all dispensions' daily defined doses over time. Conditional logistic regression models adjusted for comorbidity (National Patient Register), socioeconomic parameters (Statistics Norway), concomitant drug use and, for female cancers, reproduction data (Medical Birth Registry), will be applied to identify drug-use-cancer-risk associations. ETHICS AND DISSEMINATION The study is approved by the regional ethical committee and the Norwegian data protection authority. Results of the initial screening step and analysis pipeline will be described in a key paper. Subsequent papers will report the evaluation of identified signals in replication studies. Results will be published in peer-reviewed journals, at scientific conferences and through press releases.
Collapse
Affiliation(s)
| | - Nathalie C Støer
- Norwegian National Advisory Unit on Women's Health, Women's Clinic, Oslo University Hospital, Oslo, Norway
| | | | | | - Elisabete Weiderpass
- Department of Research, Cancer Registry of Norway, Oslo, Norway
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
- Genetic Epidemiology Group, Folkhälsan Research Centerand Faculty of Medicine, Helsinki University, Helsinki, Finland
- Department of Community Medicine, University of Tromsø, The Arctic University of Norway, Tromsø, Norway
| | - G Hege Thoresen
- Department of Pharmacology, Institute of Clinical Medicine, University of Oslo, Oslo, Norway
- Section for Pharmacology and PharmaceuticalBiosciences, Department of Pharmacy, University of Oslo, Oslo, Norway
| | - Karen Boldingh Debernard
- Regional Medicines Information and Pharmacovigilance Centre (RELIS), Department of Pharmacology, Oslo University Hospital, Oslo, Norway
| | - Øystein Karlstad
- Department of Chronic Diseases and Aging, Nasjonalt folkehelseinstitutt, Oslo, Norway
| | - Anton Pottegard
- Clinical Pharmacology and Pharmacy, Department ofPublic Health, University of Southern Denmark, Odense, Denmark
| | - Søren Friis
- Danish Cancer Society Research Center, Danish Cancer Society, Copenhagen, Denmark
| |
Collapse
|
12
|
Ristl R, Xi D, Glimm E, Posch M. Optimal exact tests for multiple binary endpoints. Comput Stat Data Anal 2018. [DOI: 10.1016/j.csda.2018.01.001] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|