Finite-sample consistency of combination-based permutation tests with application to repeated measures designs

Recognition and perception of emotions in juvenile myoclonic epilepsy

OBJECTIVE

Perception and recognition of emotions are fundamental prerequisites of human life. Patients with juvenile myoclonic epilepsy (JME) may have emotional and behavioral impairments that might influence socially desirable interactions. We aimed to investigate perception and recognition of emotions in patients with JME by means of neuropsychological tests and functional magnetic resonance imaging (fMRI).

METHODS

Sixty-five patients with JME (median age = 27 years, interquartile range [IQR] = 23-34) were prospectively recruited at the Department of Neurology, Christian Doppler University Hospital, Paracelsus Medical University, Salzburg, Austria. Patients were compared to 68 healthy controls (median age = 24 years, IQR = 21-31), matched for sex, age, and education. All study participants underwent the Networks of Emotion Processing test battery (NEmo), an fMRI paradigm of "dynamic fearful faces," a structured interview for psychiatric and personality disorders, and comprehensive neuropsychological testing.

RESULTS

JME patients versus healthy controls demonstrated significant deficits in emotion recognition in facial and verbal tasks of all emotions, especially fear. fMRI revealed decreased amygdala activation in JME patients as compared to healthy controls. Patients were at a higher risk of experiencing psychiatric disorders as compared to healthy controls. Cognitive evaluation revealed impaired attentional and executive functioning, namely psychomotor speed, tonic alertness, divided attention, mental flexibility, and inhibition of automated reactions. Duration of epilepsy correlated negatively with parallel prosodic and facial emotion recognition in NEmo. Deficits in emotion recognition were not associated with psychiatric comorbidities, impaired attention and executive functions, types of seizures, and treatment.

SIGNIFICANCE

This prospective study demonstrated that as compared to healthy subjects, patients with JME had significant deficits in recognition and perception of emotions as shown by neuropsychological tests and fMRI. The results of this study may have importance for psychological/psychotherapeutic interventions in the management of patients with JME.

A robust Spearman correlation coefficient permutation test

In this work, we show that Spearman's correlation coefficient test about H 0 : ρ s = 0 found in most statistical software is theoretically incorrect and performs poorly when bivariate normality assumptions are not met or the sample size is small. There is common misconception that the tests about ρ s = 0 are robust to deviations from bivariate normality. However, we found under certain scenarios violation of the bivariate normality assumption has severe effects on type I error control for the common tests. To address this issue, we developed a robust permutation test for testing the hypothesis H 0 : ρ s = 0 based on an appropriately studentized statistic. We will show that the test is asymptotically valid in general settings. This was demonstrated by a comprehensive set of simulation studies, where the proposed test exhibits robust type I error control, even when the sample size is small. We also demonstrated the application of this test in two real world examples.

A Multi-Aspect Permutation Test for Goodness-of-Fit Problems

Parametric techniques commonly rely on specific distributional assumptions. It is therefore fundamental to preliminarily identify the eventual violations of such assumptions. Therefore, appropriate testing procedures are required for this purpose to deal with a the goodness-of-fit (GoF) problem. This task can be quite challenging, especially with small sample sizes and multivariate data. Previous studiesshowed how a GoF problem can be easily represented through a traditional two-sample system of hypotheses. Following this idea, in this paper, we propose a multi-aspect permutation-based test to deal with the multivariate goodness-of-fit, taking advantage of the nonparametric combination (NPC) methodology. A simulation study is then conducted to evaluate the performance of our proposal and to identify the eventual critical scenarios. Finally, a real data application is considered.

STATegra: Multi-Omics Data Integration - A Conceptual Scheme With a Bioinformatics Pipeline

Technologies for profiling samples using different omics platforms have been at the forefront since the human genome project. Large-scale multi-omics data hold the promise of deciphering different regulatory layers. Yet, while there is a myriad of bioinformatics tools, each multi-omics analysis appears to start from scratch with an arbitrary decision over which tools to use and how to combine them. Therefore, it is an unmet need to conceptualize how to integrate such data and implement and validate pipelines in different cases. We have designed a conceptual framework (STATegra), aiming it to be as generic as possible for multi-omics analysis, combining available multi-omic anlaysis tools (machine learning component analysis, non-parametric data combination, and a multi-omics exploratory analysis) in a step-wise manner. While in several studies, we have previously combined those integrative tools, here, we provide a systematic description of the STATegra framework and its validation using two The Cancer Genome Atlas (TCGA) case studies. For both, the Glioblastoma and the Skin Cutaneous Melanoma (SKCM) cases, we demonstrate an enhanced capacity of the framework (and beyond the individual tools) to identify features and pathways compared to single-omics analysis. Such an integrative multi-omics analysis framework for identifying features and components facilitates the discovery of new biology. Finally, we provide several options for applying the STATegra framework when parametric assumptions are fulfilled and for the case when not all the samples are profiled for all omics. The STATegra framework is built using several tools, which are being integrated step-by-step as OpenSource in the STATegRa Bioconductor package.

Small sample sizes: A big data problem in high-dimensional data analysis

In many experiments and especially in translational and preclinical research, sample sizes are (very) small. In addition, data designs are often high dimensional, i.e. more dependent than independent replications of the trial are observed. The present paper discusses the applicability of max t-test-type statistics (multiple contrast tests) in high-dimensional designs (repeated measures or multivariate) with small sample sizes. A randomization-based approach is developed to approximate the distribution of the maximum statistic. Extensive simulation studies confirm that the new method is particularly suitable for analyzing data sets with small sample sizes. A real data set illustrates the application of the methods.

Joint Analysis of Cortical Area and Thickness as a Replacement for the Analysis of the Volume of the Cerebral Cortex

Cortical surface area is an increasingly used brain morphology metric that is ontogenetically and phylogenetically distinct from cortical thickness and offers a separate index of neurodevelopment and disease. However, the various existing methods for assessment of cortical surface area from magnetic resonance images have never been systematically compared. We show that the surface area method implemented in FreeSurfer corresponds closely to the exact, but computationally more demanding, mass-conservative (pycnophylactic) method, provided that images are smoothed. Thus, the data produced by this method can be interpreted as estimates of cortical surface area, as opposed to areal expansion. In addition, focusing on the joint analysis of thickness and area, we compare an improved, analytic method for measuring cortical volume to a permutation-based nonparametric combination (NPC) method. We use the methods to analyze area, thickness and volume in young adults born preterm with very low birth weight, and show that NPC analysis is a more sensitive option for studying joint effects on area and thickness, giving equal weight to variation in both of these 2 morphological features.

Multivariate permutation tests for two sample testing in presence of nondetects with application to microarray data

Very often, data collected in medical research are characterized by censored observations and/or data with mass on the value zero. This happens for example when some measurements fall below the detection limits of the specific instrument used. This type of left censored observations is called "nondetects". Such a situation of an excessive number of zeros in a data set is also referred to as zero-inflated data. In the present work, we aim at comparing different multivariate permutation procedures in two-sample testing for data with nondetects. The effect of censoring is investigated with regard to the different values that may be attributed to nondetected values, both under the null hypothesis and under alternative. We motivate the problem using data from allergy research.

A hybrid method of the sequential Monte Carlo and the Edgeworth expansion for computation of very small p-values in permutation tests

Permutation tests are very useful when parametric assumptions are violated or distributions of test statistics are mathematically intractable. The major advantage of permutation tests is that the procedure is so general that it is applicable to most test statistics. The computational expense is, however, impractical in high-dimensional settings such as genomewide association studies. This study provides a comprehensive review of existing methods that can compute very small p-values efficiently. A common issue with existing methods is that they can only be applied to a specific test statistic. To fill in the knowledge gap, we propose a hybrid method of the sequential Monte Carlo and the Edgeworth expansion approximation for a studentized statistic, which is applicable to a variety of test statistics. The simulation results show that the proposed method performs better than competing methods. Furthermore, applications of the proposed method are demonstrated by statistical analysis on the genomewide association studies data from the Study of Addiction: Genetics and Environment (SAGE).

Bonferroni-Holm and permutation tests to compare health data: methodological and applicative issues

BACKGROUND

Statistical methodology is a powerful tool in the health research; however, there is wide accord that statistical methodologies are not usually used properly. In particular when multiple comparisons are needed, it is necessary to check the rate of false positive results and the potential inflation of type I errors. In this case, permutation testing methods are useful to check the simultaneous significance level and identify the most significant factors.

METHODS

In this paper an application of permutation tests, in the medical context of Inflammatory Bowel Diseases, is performed. The main goal is to assess the existence of significant differences between Crohn's Disease (CD) and Ulcerative Colitis (UC). The Sequentially Rejective Multiple Test (Bonferroni-Holm procedure) is used to find which of the partial tests are effectively significant and solve the problem of the multiplicity control.

RESULTS

Applying Non-Parametric Combination (NPC) Test for partial and combined tests we conclude that Crohn's Disease patients and Ulcerative Colitis patients differ between them for most examined variables. UC patients compared with the CD patients, have a higher diagnosis age, not show smoking status, proportion of patients treated with immunosuppressants or with biological drugs is lower than the CD patients, even if the duration of such therapies is longer. CD patients have a higher rate of re-hospitalization. Diabetes is more present in the sub-population of UC patients. Analyzing the Charlson score we can highlight that UC patients have a more severe clinical situation than CD patients. Finally, CD patients are more frequently subject to surgery compared to UC. Appling of the Bonferroni Holm procedure, which provided adjusted p-values, we note that only nine of the examined variables are statistically significant: Smoking habit, Immunosuppressive therapy, Surgery, Biological Drug, Diabetes, Adverse Events, Re-hospitalization, Gender and Duration of Immunosoppressive Therapy. Therefore, we can conclude that these are the specific variables that can discriminate effectively the Crohn's Disease and Ulcerative Colitis groups.

CONCLUSIONS

We identified significant variables that discriminate the two groups, satisfying the multiplicity problem, in fact we can affirm that Smoking habit, Immunosuppressive therapy, Surgery, Biological Drug, Diabetes, Adverse Events, Hospitalization, Gender and Duration of Immunosoppressive Therapy are the effectively significant variables.

Sexually Diergic Trophic Effects of Estradiol Exposure on Developing Bovine Cerebellar Granule Cells

In the mammalian brain, the differentiation of neural cells and the developmental organization of the underlying circuitry are influenced by steroid hormones. The estrogen 17-β estradiol (E2) is one of the most potent regulators of neural growth during prenatal life, synthetized locally from steroid precursors including prenatal testicular testosterone. Estradiol promotes brain differentiation counting sexually dimorphic neural circuits by binding to the estrogen receptors, ER-α and ER-β. The cerebellum has been described as a site of estrogen action and a potentially sexually dimorphic area. The goal of this study was to analyze the capacity of E2 to affect the growth of male and female fetal bovine cerebellar granule. We performed primary cultures of fetal cerebellar granules, and verified the mRNA expression of the ER-α and ER-β in both sexes. Moreover, the distribution of ERs in the male and female cerebellar granules of the second fetal stage was characterized by immunohistochemistry. We measured morphological parameters in presence (or absence) of estradiol administration, focusing on the variations of the dendritic branching pattern of granule neurons. By using the nonparametric combination and permutation testing approach, we proposed a sophisticated multivariate statistical analysis to demonstrate that E2 induces multifarious and dimorphic changes in the granule cells. E2 exerts trophic effects in both female and male granules and this effect is stronger in female. Male granules treated with E2 became similar to female control granule. Bos taurus species has a long gestation and a large brain that offers an interesting alternative in comparative neuroscience.

The potential for increased power from combining P-values testing the same hypothesis

The conventional approach to hypothesis testing for formal inference is to prespecify a single test statistic thought to be optimal. However, we usually have more than one test statistic in mind for testing the null hypothesis of no treatment effect but we do not know which one is the most powerful. Rather than relying on a single p-value, combining p-values from prespecified multiple test statistics can be used for inference. Combining functions include Fisher's combination test and the minimum p-value. Using randomization-based tests, the increase in power can be remarkable when compared with a single test and Simes's method. The versatility of the method is that it also applies when the number of covariates exceeds the number of observations. The increase in power is large enough to prefer combined p-values over a single p-value. The limitation is that the method does not provide an unbiased estimator of the treatment effect and does not apply to situations when the model includes treatment by covariate interaction.

Non-parametric combination and related permutation tests for neuroimaging

In this work, we show how permutation methods can be applied to combination analyses such as those that include multiple imaging modalities, multiple data acquisitions of the same modality, or simply multiple hypotheses on the same data. Using the well-known definition of union-intersection tests and closed testing procedures, we use synchronized permutations to correct for such multiplicity of tests, allowing flexibility to integrate imaging data with different spatial resolutions, surface and/or volume-based representations of the brain, including non-imaging data. For the problem of joint inference, we propose and evaluate a modification of the recently introduced non-parametric combination (NPC) methodology, such that instead of a two-phase algorithm and large data storage requirements, the inference can be performed in a single phase, with reasonable computational demands. The method compares favorably to classical multivariate tests (such as MANCOVA), even when the latter is assessed using permutations. We also evaluate, in the context of permutation tests, various combining methods that have been proposed in the past decades, and identify those that provide the best control over error rate and power across a range of situations. We show that one of these, the method of Tippett, provides a link between correction for the multiplicity of tests and their combination. Finally, we discuss how the correction can solve certain problems of multiple comparisons in one-way ANOVA designs, and how the combination is distinguished from conjunctions, even though both can be assessed using permutation tests. We also provide a common algorithm that accommodates combination and correction.

[Statistical tests in medical research: traditional methods vs. multivariate NPC permutation tests]

Statistical tests in medical research: traditional methods vs. multivariate npc permutation tests.Within medical research, a useful statistical tool is based on hypotheses testing in terms of the so-called null, that is the treatment has no effect, and alternative hypotheses, that is the treatment has some effects. By controlling the risks of wrong decisions, empirical data are used in order to possibly reject the null hypotheses in favour of the alternative, so that demonstrating the efficacy of a treatment of interest. The multivariate permutation tests, based on the nonparametric combination - NPC method, provide an innovative, robust and effective hypotheses testing solution to many real problems that are commonly encountered in medical research when multiple end-points are observed. This paper discusses the various approaches to hypothesis testing and the main advantages of NPC tests, which consist in the fact that they require much less stringent assumptions than traditional statistical tests. Moreover, the related results may be extended to the reference population even in case of selection-bias, that is non-random sampling. In this work, we review and discuss some basic testing procedures along with the theoretical and practical relevance of NPC tests showing their effectiveness in medical research. Within the non-parametric methods, NPC tests represent the current "frontier" of statistical research, but already widely available in the practice of analysis of clinical data.

Iterated combination-based paired permutation tests to determine shape effects of chemotherapy in patients with esophageal cancer

The nonparametric combination of dependent permutation tests method is a useful general tool when a testing problem can be broken down into a set of different k > 1 partial tests. These partial tests, after adjustment of p-values to control for multiplicity, can be marginally analyzed, but jointly considered they can provide information on an overall hypothesis, which might represent the true goal of the testing problem. On the one hand, independence among the partial tests is usually an unrealistic assumption; on the other, even when the underlying dependence relations are known quite often they are difficult to cope with properly. Therefore this combination must be achieved nonparametrically, by implicitly taking into account the dependence structure of tests without explicitly describing it. An important property of the tests based on nonparametric combination methodology, when the number of response variables is high compared to the sample sizes, consists in the finite sample consistency. A practical problem involves choosing the most suitable combining function for each specific testing problem given that the final result can be affected by this crucial choice. The purpose of this article is to present an nonparametric combination solution based on the iterated combination of partial tests, evaluate its power behavior using a Monte Carlo simulation study and apply it to a real medical problem, namely the evaluation of the effects of chemotherapy on the shape of esophageal tumors. R code has been implemented to carry out the analyses.

Evaluating treatment effect within a multivariate stochastic ordering framework: Nonparametric combination methodology applied to a study on multiple sclerosis

Multiple sclerosis is an autoimmune complex disease that affects the central nervous system. It has a multitude of symptoms that are observed in different people in many different ways. At this time, there is no definite cure for multiple sclerosis. However, therapies that slow the progression of disability, controlling symptoms and helping patients to maintain a normal quality of life, are available. We will focus on relapsing-remitting multiple sclerosis patients treated with interferons or glatiramer acetate. These treatments have been shown to be effective, but their relative effectiveness has not been well established yet. To assess the superiority of a treatment, instead of classical parametric methods, we propose a statistical approach within the permutation setting and the nonparametric combination of dependent permutation tests. In this framework, we may easily handle with hypothesis testing problems for multivariate monotonic stochastic ordering. This approach has been motivated by the analysis of a large observational Italian multicentre study on multiple sclerosis, with several continuous and categorical outcomes measured at multiple time points.

Robust non-parametric tests for complex-repeated measures problems in ophthalmology

The NonParametric Combination methodology (NPC) of dependent permutation tests allows the experimenter to face many complex multivariate testing problems and represents a convincing and powerful alternative to standard parametric methods. The main advantage of this approach lies in its flexibility in handling any type of variable (categorical and quantitative, with or without missing values) while at the same time taking dependencies among those variables into account without the need of modelling them. NPC methodology enables to deal with repeated measures, paired data, restricted alternative hypotheses, missing data (completely at random or not), high-dimensional and small sample size data. Hence, NPC methodology can offer a significant contribution to successful research in biomedical studies with several endpoints, since it provides reasonably efficient solutions and clear interpretations of inferential results. Pesarin F. Multivariate permutation tests: with application in biostatistics. Chichester-New York: John Wiley &Sons, 2001; Pesarin F, Salmaso L. Permutation tests for complex data: theory, applications and software. Chichester, UK: John Wiley &Sons, 2010. We focus on non-parametric permutation solutions to two real-case studies in ophthalmology, concerning complex-repeated measures problems. For each data set, different analyses are presented, thus highlighting characteristic aspects of the data structure itself. Our goal is to present different solutions to multivariate complex case studies, guiding researchers/readers to choose, from various possible interpretations of a problem, the one that has the highest flexibility and statistical power under a set of less stringent assumptions. MATLAB code has been implemented to carry out the analyses.