On the Weak Consistency of Permutation Tests

A robust Spearman correlation coefficient permutation test

In this work, we show that Spearman's correlation coefficient test about H 0 : ρ s = 0 found in most statistical software is theoretically incorrect and performs poorly when bivariate normality assumptions are not met or the sample size is small. There is common misconception that the tests about ρ s = 0 are robust to deviations from bivariate normality. However, we found under certain scenarios violation of the bivariate normality assumption has severe effects on type I error control for the common tests. To address this issue, we developed a robust permutation test for testing the hypothesis H 0 : ρ s = 0 based on an appropriately studentized statistic. We will show that the test is asymptotically valid in general settings. This was demonstrated by a comprehensive set of simulation studies, where the proposed test exhibits robust type I error control, even when the sample size is small. We also demonstrated the application of this test in two real world examples.

A unified approach to permutation testing for equivalence

The notion of testing for equivalence of two treatments is widely used in clinical trials, pharmaceutical experiments, bioequivalence and quality control. It is traditionally operated within the intersection–union principle (IU). According to this principle the null hypothesis is stated as the set of effects the differences $$\delta$$ δ of which lie outside a suitable equivalence interval and the alternative as the set of $$\delta$$ δ that lie inside it. In the literature related solutions are essentially based on likelihood techniques, which in turn are rather difficult to deal with. A recently published paper goes beyond most of likelihood limitations by using the IU approach within the permutation theory. One more paper, based on Roy’s union–intersection principle (UI) within the permutation theory, goes beyond some limitations of traditional two-sided tests. Such UI approach, effectively a mirror image of IU, assumes a null hypothesis where $$\delta$$ δ lies inside the equivalence interval and an alternative where it lies outside. Since testing for equivalence can rationally be analyzed by both principles but, as the two differ in terms of the mirror-like roles assigned to the hypotheses under study, they are not strictly comparable. The present paper’s main goal is to look into these problems and provide a sort of comparative analysis of both by highlighting the related requirements, properties, limitations, difficulties, and pitfalls so as to get practitioners properly acquainted with their correct use in practical contexts.

Small sample sizes: A big data problem in high-dimensional data analysis

In many experiments and especially in translational and preclinical research, sample sizes are (very) small. In addition, data designs are often high dimensional, i.e. more dependent than independent replications of the trial are observed. The present paper discusses the applicability of max t-test-type statistics (multiple contrast tests) in high-dimensional designs (repeated measures or multivariate) with small sample sizes. A randomization-based approach is developed to approximate the distribution of the maximum statistic. Extensive simulation studies confirm that the new method is particularly suitable for analyzing data sets with small sample sizes. A real data set illustrates the application of the methods.

Multivariate permutation tests for two sample testing in presence of nondetects with application to microarray data

Very often, data collected in medical research are characterized by censored observations and/or data with mass on the value zero. This happens for example when some measurements fall below the detection limits of the specific instrument used. This type of left censored observations is called "nondetects". Such a situation of an excessive number of zeros in a data set is also referred to as zero-inflated data. In the present work, we aim at comparing different multivariate permutation procedures in two-sample testing for data with nondetects. The effect of censoring is investigated with regard to the different values that may be attributed to nondetected values, both under the null hypothesis and under alternative. We motivate the problem using data from allergy research.

Exact testing with random permutations

When permutation methods are used in practice, often a limited number of random permutations are used to decrease the computational burden. However, most theoretical literature assumes that the whole permutation group is used, and methods based on random permutations tend to be seen as approximate. There exists a very limited amount of literature on exact testing with random permutations, and only recently a thorough proof of exactness was given. In this paper, we provide an alternative proof, viewing the test as a “conditional Monte Carlo test” as it has been called in the literature. We also provide extensions of the result. Importantly, our results can be used to prove properties of various multiple testing procedures based on random permutations.