Properties of balanced permutations

Measure of Strength of Evidence for Visually Observed Differences between Subpopulations

For measuring the strength of visually-observed subpopulation differences, the Population Difference Criterion is proposed to assess the statistical significance of visually observed subpopulation differences. It addresses the following challenges: in high-dimensional contexts, distributional models can be dubious; in high-signal contexts, conventional permutation tests give poor pairwise comparisons. We also make two other contributions: Based on a careful analysis we find that a balanced permutation approach is more powerful in high-signal contexts than conventional permutations. Another contribution is the quantification of uncertainty due to permutation variation via a bootstrap confidence interval. The practical usefulness of these ideas is illustrated in the comparison of subpopulations of modern cancer data.

Statistical quantification of confounding bias in machine learning models

BACKGROUND

The lack of nonparametric statistical tests for confounding bias significantly hampers the development of robust, valid, and generalizable predictive models in many fields of research. Here I propose the partial confounder test, which, for a given confounder variable, probes the null hypotheses of the model being unconfounded.

RESULTS

The test provides a strict control for type I errors and high statistical power, even for nonnormally and nonlinearly dependent predictions, often seen in machine learning. Applying the proposed test on models trained on large-scale functional brain connectivity data (N= 1,865) (i) reveals previously unreported confounders and (ii) shows that state-of-the-art confound mitigation approaches may fail preventing confounder bias in several cases.

CONCLUSIONS

The proposed test (implemented in the package mlconfound; https://mlconfound.readthedocs.io) can aid the assessment and improvement of the generalizability and validity of predictive models and, thereby, fosters the development of clinically useful machine learning biomarkers.

A modified F-test for hypothesis testing in large-scale data

Nowadays, performing simultaneous hypothesis testing under small sample sizes in large-scale features has received unprecedented attention in practice. Traditional approaches are not appropriate for this purpose. Instead, a novel approach which is based on the null statistic is used in the literature. The null statistic approach, using the permutation samples of all features, permits the more accurate estimation of the [Formula: see text]-value for a target feature. However, the known methods, working based on the null statistic, cannot be applied to features with three or more levels. In the present study, an attempt is made to introduce a new permutation test statistic which can be modified to the null statistic approach when features have three or more levels. The robustness of the proposed test statistic and its performance in estimating the [Formula: see text]-value are demonstrated through conducting simulation studies and the analysis of two real datasets.

Exact testing with random permutations

When permutation methods are used in practice, often a limited number of random permutations are used to decrease the computational burden. However, most theoretical literature assumes that the whole permutation group is used, and methods based on random permutations tend to be seen as approximate. There exists a very limited amount of literature on exact testing with random permutations, and only recently a thorough proof of exactness was given. In this paper, we provide an alternative proof, viewing the test as a “conditional Monte Carlo test” as it has been called in the literature. We also provide extensions of the result. Importantly, our results can be used to prove properties of various multiple testing procedures based on random permutations.

Aging mice show a decreasing correlation of gene expression within genetic modules

In this work we present a method for the differential analysis of gene co-expression networks and apply this method to look for large-scale transcriptional changes in aging. We derived synonymous gene co-expression networks from AGEMAP expression data for 16-month-old and 24-month-old mice. We identified a number of functional gene groups that change co-expression with age. Among these changing groups we found a trend towards declining correlation with age. In particular, we identified a modular (as opposed to uniform) decline in general correlation with age. We identified potential transcriptional mechanisms that may aid in modular correlation decline. We found that computationally identified targets of the NF-ΚB transcription factor decrease expression correlation with age. Finally, we found that genes that are prone to declining co-expression tend to be co-located on the chromosome. Our results conclude that there is a modular decline in co-expression with age in mice. They also indicate that factors relating to both chromosome domains and specific transcription factors may contribute to the decline.

There is mounting evidence that mammalian aging is marked by increased gene transcriptional variation. This trend was shown not only by studying gene expression in single cells (Bahar et al. 2006), but at the coarse tissue resolution as well (Somel et al. 2006; Li et al. 2009). These led us to believe that looking at absolute changes in expression level alone may not tell the whole story of transcriptional changes in age. Instead the story may be in the more subtle changes in the coordination of expression among multiple genes. For this reason, we decided to look at changes in co-expression relationships with age. To this end, we developed a methodology for differential co-expression network analysis for the comparison gene co-expression on a global scale. We applied this methodology to compare co-expression between young (16-month) and old (24-month) mice. This allowed us to find both gene groups whose coordination appear to be affected by age and to propose potential mechanisms for the change. We believe our work is of broad importance because it represents a different paradigm for looking not only at aging but also at any complex condition or disease—away from changes in individual genes towards changes in gene relationships.