Generalized Augmentation to Control the False Discovery Exceedance in Multiple Testing

Permutation-based simultaneous confidence bounds for the false discovery proportion

When multiple hypotheses are tested, interest is often in ensuring that the proportion of false discoveries is small with high confidence. In this paper, confidence upper bounds for the false discovery proportion are constructed, which are simultaneous over all rejection cut-offs. In particular, this allows the user to select a set of hypotheses post hoc such that the false discovery proportion lies below some constant with high confidence. Our method uses permutations to account for the dependence structure in the data. So far only Meinshausen (2006) has developed an exact, permutation-based and computationally feasible method for obtaining simultaneous false discovery proportion bounds. We propose an exact method which uniformly improves that procedure. Further, we provide a generalization of the method that lets the user select the shape of the simultaneous confidence bounds; this gives the user more freedom in determining the power properties of the method. Interestingly, several existing permutation methods, such as significance analysis of microarrays and the maxT method of Westfall & Young (1993), are obtained as special cases.

Sample Size Calculation for Controlling False Discovery Proportion

The false discovery proportion (FDP), the proportion of incorrect rejections among all rejections, is a direct measure of abundance of false positive findings in multiple testing. Many methods have been proposed to control FDP, but they are too conservative to be useful for power analysis. Study designs for controlling the mean of FDP, which is false discovery rate, have been commonly used. However, there has been little attempt to design study with direct FDP control to achieve certain level of efficiency. We provide a sample size calculation method using the variance formula of the FDP under weak-dependence assumptions to achieve the desired overall power. The relationship between design parameters and sample size is explored. The adequacy of the procedure is assessed by simulation. We illustrate the method using estimated correlations from a prostate cancer dataset.