A general consonance principle for closure tests based on p-values

The closure principle is a powerful approach to constructing efficient testing procedures controlling the familywise error rate in the strong sense. For small numbers of hypotheses and the setting of independent elementary p -values we consider closed tests where each intersection hypothesis is tested with a p -value combination test. Examples of such combination tests are the Fisher combination test, the Stouffer test, the Omnibus test, the truncated test, or the Wilson test. Some of these tests, such as the Fisher combination, the Stouffer, or the Omnibus test, are not consonant and rejection of the global null hypothesis does not always lead to rejection of at least one elementary null hypothesis. We develop a general principle to uniformly improve closed tests based on p -value combination tests by modifying the rejection regions such that the new procedure becomes consonant. For the Fisher combination test and the Stouffer test, we show by simulations that this improvement can lead to a substantial increase in power.

Combining Partial True Discovery Guarantee Procedures

Closed testing has recently been shown to be optimal for simultaneous true discovery proportion control. It is, however, challenging to construct true discovery guarantee procedures in such a way that it focuses power on some feature sets chosen by users based on their specific interest or expertise. We propose a procedure that allows users to target power on prespecified feature sets, that is, "focus sets." Still, the method also allows inference for feature sets chosen post hoc, that is, "nonfocus sets," for which we deduce a true discovery lower confidence bound by interpolation. Our procedure is built from partial true discovery guarantee procedures combined with Holm's procedure and is a conservative shortcut to the closed testing procedure. A simulation study confirms that the statistical power of our method is relatively high for focus sets, at the cost of power for nonfocus sets, as desired. In addition, we investigate its power property for sets with specific structures, for example, trees and directed acyclic graphs. We also compare our method with AdaFilter in the context of replicability analysis. The application of our method is illustrated with a gene ontology analysis in gene expression data.

Optimal weighted Bonferroni tests and their graphical extensions

Regulatory guidelines mandate the strong control of the familywise error rate in confirmatory clinical trials with primary and secondary objectives. Bonferroni tests are one of the popular choices for multiple comparison procedures and are building blocks of more advanced procedures. It is usually of interest to find the optimal weighted Bonferroni split for multiple hypotheses. We consider two popular quantities as the optimization objectives, which are the disjunctive power and the conjunctive power. The former is the probability to reject at least one false hypothesis and the latter is the probability to reject all false hypotheses. We investigate the behavior of each of them as a function of different Bonferroni splits, given assumptions about the alternative hypotheses and correlations between test statistics. Under independent tests, unique optimal Bonferroni weights exist; under dependence, optimal Bonferroni weights may not be unique based on a fine grid search. In general, we propose an optimization algorithm based on constrained nonlinear optimization and multiple starting points. The proposed algorithm efficiently identifies optimal Bonferroni weights to maximize the disjunctive or conjunctive power. In addition, we apply the proposed algorithm to graphical approaches, which include many Bonferroni-based multiple comparison procedures. Utilizing the closed testing principle, we adopt a two-step approach to find optimal graphs using the disjunctive power. We also identify a class of closed test procedures that optimize the conjunctive power. We apply the proposed algorithm to a case study to illustrate the utility of optimal graphical approaches that reflect study objectives.

Reverse graphical approaches for multiple test procedures

The graphical approach has been proposed as a general framework for clinical trial designs involving multiple hypotheses, where decisions are made only based on the observed marginal p -values. The graphical approach starts from a graph that includes all hypotheses as vertices and gradually removes some vertices when their corresponding hypotheses are rejected. In this paper, we propose a reverse graphical approach, which starts from a set of singleton graphs and gradually adds vertices into graphs until rejection of a set of hypotheses is made. Proofs of familywise error rate control are provided. A simulation study is conducted for statistical power analysis, and a case study is included to illustrate how the proposed approach can be applied to clinical studies.

Sample size optimization for clinical trials using graphical approaches for multiplicity adjustment

Graphical approach provides a useful framework for multiplicity adjustment in clinical trials with multiple endpoints. When designing a graphical approach, initial weight and transition probability for the endpoints are often assigned based on clinical importance. For example, practitioners may prefer putting more weights on some primary endpoints. The clinical preference can be formulated as a constrain in the sample size optimization problem. However, there has been a lack of theoretical guidance on how to specify initial weight and transition probability in a graphical approach to meet the clinical preference but at the same time to minimize the sample size needed for a power requirement. To fill this gap, we propose statistical methods to optimize sample size over initial weight and transition probability in a graphical approach under a common setting, which is to use marginal power for each endpoint in a trial design. Importantly, we prove that some of the commonly used graphical approaches such as putting all initial weights on one endpoint are suboptimal. Our methods are flexible, which can be used for both single-arm trials and randomized controlled trials with either continuous or binary or mixed types of endpoints. Additionally, we prove the existence of optimal solution where all marginal powers are placed exactly at the prespecified values, assuming continuity. Two hypothetical clinical trial designs are presented to illustrate the application of our methods under different scenarios. Results are first presented for a design with two endpoints and are further generalized to three or more endpoints. Our findings are helpful to guide the design of a graphical approach and the sample size calculation in clinical trials.

On dependence assumption in p-value based multiple test procedures

There are various multiple comparison procedures used in confirmatory clinical studies and exploratory research for multiplicity adjustment. Among them are the Hochberg and Benjamini-Hochberg procedures. A common misconception is that these procedures control the type I error rate properly if the test statistics are independent or positively correlated. In fact, a much stronger positive dependence assumption needs to be satisfied to guarantee the type I error rate control. We give a comprehensive review of the dependence conditions used in multiple testing procedures. We show that a weaker positive dependence assumption may result an inflation of type I error rate by a factor of 2 and discuss the type I error rate control under certain negative dependence conditions.

A fortune cookie problem: A test for nominal data whether two samples are from the same population of equally likely elements

This article considers a way to test the hypothesis that two collections of objects are from the same uniform distribution of such objects. The exact p-value is calculated based on the distribution for the observed overlaps. In addition, an interval estimate of the number of distinct objects, when all objects are equally likely, is indicated.

Closed testing with Globaltest, with application in metabolomics

The Globaltest is a powerful test for the global null hypothesis that there is no association between a group of features and a response of interest, which is popular in pathway testing in metabolomics. Evaluating multiple feature sets, however, requires multiple testing correction. In this paper, we propose a multiple testing method, based on closed testing, specifically designed for the Globaltest. The proposed method controls the family-wise error rate simultaneously over all possible feature sets, and therefore allows post hoc inference, i.e. the researcher may choose feature sets of interest after seeing the data without jeopardizing error control. To circumvent the exponential computation time of closed testing, we derive a novel shortcut that allows exact closed testing to be performed on the scale of metabolomics data. An R package ctgt is available on CRAN for the implementation of the shortcut procedure, with applications on several real metabolomics data examples. This article is protected by copyright. All rights reserved.

Improved group sequential Holm procedures for testing multiple correlated hypotheses over time

Clinical trials can typically feature two different types of multiple inference: testing of more than one null hypothesis and testing at multiple time points. These modes of multiplicity are closely related mathematically but distinct statistically and philosophically. Regulatory agencies require strong control of the family-wise error rate (FWER), the risk of falsely rejecting any null hypothesis at any analysis. The correlations between test statistics at interim analyses and the final analysis are therefore routinely used in group sequential designs to achieve less conservative critical values. However, the same type of correlations between different comparisons, endpoints or sub-populations are less commonly used. As a result, FWER is in practice often controlled conservatively for commonly applied procedures.Repeated testing of the same null hypothesis may give changing results, when the hypothesis is rejected at an interim but accepted at the final analysis. The mathematically correct overall rejection is at odds with an inference theoretic approach and with common sense. We discuss these two issues, of incorporating correlations and how to interpret time-changing conclusions, and provide case studies where power can be increased while adhering to sound statistical principles.

Incorporating the sample correlation into the testing of two endpoints in clinical trials

We introduce an improved Bonferroni method for testing two primary endpoints in clinical trial settings using a new data-adaptive critical value that explicitly incorporates the sample correlation coefficient. Our methodology is developed for the usual Student's t-test statistics for testing the means under normal distributional setting with unknown population correlation and variances. Specifically, we construct a confidence interval for the unknown population correlation and show that the estimated type-1 error rate of the Bonferroni method with the population correlation being estimated by its lower confidence limit can be bounded from above less conservatively than using the traditional Bonferroni upper bound. We also compare the new procedure with other procedures commonly used for the multiple testing problem addressed in this paper.

Sample size optimization and initial allocation of the significance levels in group sequential trials with multiple endpoints

We consider multistage tests of multiple hypotheses under a flexible setting of calendar time and information fraction, focusing on the case where there are two hypotheses under testing. Explicit expressions of statistical powers are derived. With a proof of existence and uniqueness of solution, we develop a numerical method to search the optimal sample size. The proposed method allows us to find the suitable allocation of initial significance level along with the minimum sample size for group sequential designs, with and without hierarchical structures among different endpoints.

Revealing posturographic profile of patients with Parkinsonian syndromes through a novel hypothesis testing framework based on machine learning

Falling in Parkinsonian syndromes (PS) is associated with postural instability and consists a common cause of disability among PS patients. Current posturographic practices record the body's center-of-pressure displacement (statokinesigram) while the patient stands on a force platform. Statokinesigrams, after appropriate processing, can offer numerous posturographic features. This fact, although beneficial, challenges the efforts for valid statistics via standard univariate approaches. In this work, 123 PS patients were classified into fallers (PSF) or non-faller (PSNF) based on the clinical assessment, and underwent simple Romberg Test (eyes open/eyes closed). We developed a non-parametric multivariate two-sample test (ts-AUC) based on machine learning, in order to examine statokinesigrams' differences between PSF and PSNF. We analyzed posturographic features using both multiple testing with p-value adjustment and ts-AUC. While ts-AUC showed significant difference between groups (p-value = 0.01), multiple testing did not agree with this result (eyes open). PSF showed significantly increased antero-posterior movements as well as increased posturographic area compared to PSNF. Our study highlights the superiority of ts-AUC compared to standard statistical tools in distinguishing PSF and PSNF in multidimensional space. Machine learning-based statistical tests can be seen as a natural extension of classical statistics and should be considered, especially when dealing with multifactorial assessments.

RiboDiPA: a novel tool for differential pattern analysis in Ribo-seq data

Ribosome profiling, also known as Ribo-seq, has become a popular approach to investigate regulatory mechanisms of translation in a wide variety of biological contexts. Ribo-seq not only provides a measurement of translation efficiency based on the relative abundance of ribosomes bound to transcripts, but also has the capacity to reveal dynamic and local regulation at different stages of translation based on positional information of footprints across individual transcripts. While many computational tools exist for the analysis of Ribo-seq data, no method is currently available for rigorous testing of the pattern differences in ribosome footprints. In this work, we develop a novel approach together with an R package, RiboDiPA, for Differential Pattern Analysis of Ribo-seq data. RiboDiPA allows for quick identification of genes with statistically significant differences in ribosome occupancy patterns for model organisms ranging from yeast to mammals. We show that differential pattern analysis reveals information that is distinct and complimentary to existing methods that focus on translational efficiency analysis. Using both simulated Ribo-seq footprint data and three benchmark data sets, we illustrate that RiboDiPA can uncover meaningful pattern differences across multiple biological conditions on a global scale, and pinpoint characteristic ribosome occupancy patterns at single codon resolution.

Fast closed testing for exchangeable local tests

Multiple hypothesis testing problems arise naturally in science. This note introduces a new fast closed testing method for multiple testing which controls the familywise error rate. Controlling the familywise error rate is state-of-the-art in many important application areas and is preferred over false discovery rate control for many reasons, including that it leads to stronger reproducibility. The closure principle rejects an individual hypothesis if all global nulls of subsets containing it are rejected using some test statistics. It takes exponential time in the worst case. When the tests are symmetric and monotone, the proposed method is an exact algorithm for computing the closure, is quadratic in the number of tests, and is linear in the number of discoveries. Our framework generalizes most examples of closed testing, such as Holm’s method and the Bonferroni method. As a special case of the method, we propose the Simes and higher criticism fusion test, which is powerful both for detecting a few strong signals and for detecting many moderate signals.

Symmetric graphs for equally weighted tests, with application to the Hochberg procedure

The graphical approach to multiple testing provides a convenient tool for designing, visualizing, and performing multiplicity adjustments in confirmatory clinical trials while controlling the familywise error rate. It assigns a set of weights to each intersection null hypothesis within the closed test framework. These weights form the basis for intersection tests using weighted individual p-values, such as the weighted Bonferroni test. In this paper, we extend the graphical approach to intersection tests that assume equal weights for the elementary null hypotheses associated with any intersection hypothesis, including the Hochberg procedure as well as omnibus tests such as Fisher's combination, O'Brien's, and F tests. More specifically, we introduce symmetric graphs that generate sets of equal weights so that the aforementioned tests can be applied with the graphical approach. In addition, we visualize the Hochberg and the truncated Hochberg procedures in serial and parallel gatekeeping settings using symmetric component graphs. We illustrate the method with two clinical trial examples.

Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing

Closed testing procedures are classically used for familywise error rate control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion in all subsets of the hypotheses, allowing for inference robust to post hoc selection of subsets. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut and use it to investigate the power of this approach when the number of hypotheses goes to infinity. We show that if a minimal level of signal is present, the average power to detect false hypotheses at any desired false discovery proportion does not vanish. Additionally, we show that the confidence bounds for false discovery proportion are consistent estimators for the true false discovery proportion for every nonvanishing subset. We also show close connections between Simes-based closed testing and the procedure of Benjamini and Hochberg.

Context-dependent roles of MDMX (MDM4) and MDM2 in breast cancer proliferation and circulating tumor cells

INTRODUCTION

Many human breast cancers overexpress the E3 ubiquitin ligase MDM2 and its homolog MDMX. Expression of MDM2 and MDMX occurs in estrogen receptor α-positive (ERα+) breast cancer and triple-negative breast cancer (TNBC). There are p53-independent influences of MDM2 and MDMX, and 80% of TNBC express mutant p53 (mtp53). MDM2 drives TNBC circulating tumor cells (CTCs) in mice, but the context-dependent influences of MDM2 and MDMX on different subtypes of breast cancers expressing mtp53 have not been determined.

METHODS

To assess the context-dependent roles, we carried out MDM2 and MDMX knockdown in orthotopic tumors of TNBC MDA-MB-231 cells expressing mtp53 R280K and MDM2 knockdown in ERα+ T47D cells expressing mtp53 L194F. The corresponding cell proliferation was scored in vitro by growth curves and in vivo by orthotopic tumor volumes. Cell migration was assessed in vitro by wound-healing assays and cell intravasation in vivo by sorting GFP-positive CTCs by flow cytometry. The metastasis gene targets were determined by an RT-PCR array card screen and verified by qRT-PCR and Western blot analysis.

RESULTS

Knocking down MDMX or MDM2 in MDA-MB-231 cells reduced cell migration and CTC detection, but only MDMX knockdown reduced tumor volumes at early time points. This is the first report of MDMX overexpression in TNBC enhancing the CTC phenotype with correlated upregulation of CXCR4. Experiments were carried out to compare MDM2-knockdown outcomes in nonmetastatic ERα+ T47D cells. The knockdown of MDM2 in ERα+ T47D orthotopic tumors decreased primary tumor volumes, supporting our previous finding that estrogen-activated MDM2 increases cell proliferation.

CONCLUSIONS

This is the first report showing that the expression of MDM2 in ERα+ breast cancer and TNBC can result in different tumor-promoting outcomes. Both MDMX and MDM2 overexpression in TNBC MDA-MB-231 cells enhanced the CTC phenotype. These data indicate that both MDM2 and MDMX can promote TNBC metastasis and that it is important to consider the context-dependent roles of MDM2 family members in different subtypes of breast cancer.

Advances in p-Value Based Multiple Test Procedures

In this article we review recent advances in [Formula: see text]-value-based multiple test procedures (MTPs). We begin with a brief review of the basic tests of Bonferroni and Simes. Standard stepwise MTPs derived from them using the closure method of Marcus et al. (1976) are discussed next. They include the well-known MTPs of Holm (1979), Hochberg (1988) and Hommel (1988), and their extensions and improvements. This is followed by stepwise MTPs for a priori ordered hypotheses. Next we present gatekeeping MTPs (Dmitrienko and Tamhane, 2007) for hierarchically ordered families of hypotheses with logical relations among them. Finally, we give a brief review of the graphical approach (Bretz et al., 2009) to constructing and visualizing gatekeeping and other MTPs. Simple numerical examples are given to illustrate the various procedures.

A P-value model for theoretical power analysis and its applications in multiple testing procedures

Background

Power analysis is a critical aspect of the design of experiments to detect an effect of a given size. When multiple hypotheses are tested simultaneously, multiplicity adjustments to p-values should be taken into account in power analysis. There are a limited number of studies on power analysis in multiple testing procedures. For some methods, the theoretical analysis is difficult and extensive numerical simulations are often needed, while other methods oversimplify the information under the alternative hypothesis. To this end, this paper aims to develop a new statistical model for power analysis in multiple testing procedures.

Methods

We propose a step-function-based p-value model under the alternative hypothesis, which is simple enough to perform power analysis without simulations, but not too simple to lose the information from the alternative hypothesis. The first step is to transform distributions of different test statistics (e.g., t, chi-square or F) to distributions of corresponding p-values. We then use a step function to approximate each of the p-value’s distributions by matching the mean and variance. Lastly, the step-function-based p-value model can be used for theoretical power analysis.

Results

The proposed model is applied to problems in multiple testing procedures. We first show how the most powerful critical constants can be chosen using the step-function-based p-value model. Our model is then applied to the field of multiple testing procedures to explain the assumption of monotonicity of the critical constants. Lastly, we apply our model to a behavioral weight loss and maintenance study to select the optimal critical constants.

Conclusions

The proposed model is easy to implement and preserves the information from the alternative hypothesis.