Multiple testing in the genomics era: findings from Genetic Analysis Workshop 15, Group 15

Optimal alpha reduces error rates in gene expression studies: a meta-analysis approach

Background

Transcriptomic approaches (microarray and RNA-seq) have been a tremendous advance for molecular science in all disciplines, but they have made interpretation of hypothesis testing more difficult because of the large number of comparisons that are done within an experiment. The result has been a proliferation of techniques aimed at solving the multiple comparisons problem, techniques that have focused primarily on minimizing Type I error with little or no concern about concomitant increases in Type II errors. We have previously proposed a novel approach for setting statistical thresholds with applications for high throughput omics-data, optimal α, which minimizes the probability of making either error (i.e. Type I or II) and eliminates the need for post-hoc adjustments.

Results

A meta-analysis of 242 microarray studies extracted from the peer-reviewed literature found that current practices for setting statistical thresholds led to very high Type II error rates. Further, we demonstrate that applying the optimal α approach results in error rates as low or lower than error rates obtained when using (i) no post-hoc adjustment, (ii) a Bonferroni adjustment and (iii) a false discovery rate (FDR) adjustment which is widely used in transcriptome studies.

Conclusions

We conclude that optimal α can reduce error rates associated with transcripts in both microarray and RNA-seq experiments, but point out that improved statistical techniques alone cannot solve the problems associated with high throughput datasets – these approaches need to be coupled with improved experimental design that considers larger sample sizes and/or greater study replication.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-017-1728-3) contains supplementary material, which is available to authorized users.

Empirical estimation of genome-wide significance thresholds based on the 1000 Genomes Project data set

To assess the statistical significance of associations between variants and traits, genome-wide association studies (GWAS) should employ an appropriate threshold that accounts for the massive burden of multiple testing in the study. Although most studies in the current literature commonly set a genome-wide significance threshold at the level of P=5.0 × 10^-8, the adequacy of this value for respective populations has not been fully investigated. To empirically estimate thresholds for different ancestral populations, we conducted GWAS simulations using the 1000 Genomes Phase 3 data set for Africans (AFR), Europeans (EUR), Admixed Americans (AMR), East Asians (EAS) and South Asians (SAS). The estimated empirical genome-wide significance thresholds were P_sig=3.24 × 10^-8 (AFR), 9.26 × 10^-8 (EUR), 1.83 × 10^-7 (AMR), 1.61 × 10^-7 (EAS) and 9.46 × 10^-8 (SAS). We additionally conducted trans-ethnic meta-analyses across all populations (ALL) and all populations except for AFR (ΔAFR), which yielded P_sig=3.25 × 10^-8 (ALL) and 4.20 × 10^-8 (ΔAFR). Our results indicate that the current threshold (P=5.0 × 10^-8) is overly stringent for all ancestral populations except for Africans; however, we should employ a more stringent threshold when conducting a meta-analysis, regardless of the presence of African samples.

Stem Cell Strategies to Evaluate Idiosyncratic Drug-induced Liver Injury

The host-dependent nature of idiosyncratic drug-induced liver injury (iDILI) suggests that rare genetic polymorphisms may contribute to the disease. Indeed, a few mutations in key genes have already been identified using conventional human genetics approaches. Over 50 commonly used drugs can precipitate iDILI, making this a substantial medical problem. Only recently have human induced pluripotent stem cells been used as a research tool to discover novel iDILI genes and to study the mechanisms of iDILI in vitro. Here we review the current state of stem cell use in the investigation of iDILI, with a special focus on genetics. In addition, the concerns and difficulties associated with genetics and animal model research are discussed. We then present the features of patient-specific pluripotent stem cells (which may be derived from iDILI patients themselves), and explain why these cells may be of great utility. A variety of recent approaches to produce hepatocyte-like cells from pluripotent cells and the associated advantages and limitations of such cells are discussed. Future directions for the use of stem cell science to investigate iDILI include novel ways to identify new iDILI genes, a consideration of epigenetic impacts on iDILI, and the development of new and improved strategies for the production of hepatocytes from human pluripotent cells.

Cancer Genomics: Technology, Discovery, and Translation

In recent years, the increasing awareness that somatic mutations and other genetic aberrations drive human malignancies has led us within reach of personalized cancer medicine (PCM). The implementation of PCM is based on the following premises: genetic aberrations exist in human malignancies; a subset of these aberrations drive oncogenesis and tumor biology; these aberrations are actionable (defined as having the potential to affect management recommendations based on diagnostic, prognostic, and/or predictive implications); and there are highly specific anticancer agents available that effectively modulate these targets. This article highlights the technology underlying cancer genomics and examines the early results of genome sequencing and the challenges met in the discovery of new genetic aberrations. Finally, drawing from experiences gained in a feasibility study of somatic mutation genotyping and targeted exome sequencing led by Princess Margaret Hospital–University Health Network and the Ontario Institute for Cancer Research, the processes, challenges, and issues involved in the translation of cancer genomics to the clinic are discussed.

What should the genome-wide significance threshold be? Empirical replication of borderline genetic associations

BACKGROUND

Robust replication is a sine qua non for the rigorous documentation of proposed associations in the genome-wide association (GWA) setting. Currently, associations of common variants reaching P ≤ 5 × 10(-8) are considered replicated. However, there is some ambiguity about the most suitable threshold for claiming genome-wide significance.

METHODS

We defined as 'borderline' associations those with P > 5 × 10(-8) and P ≤ 1 × 10(-7). The eligible associations were retrieved using the 'Catalog of Published Genome-Wide Association Studies'. For each association we assessed whether it reached P ≤ 5 × 10(-8) with inclusion of additional data from subsequent GWA studies.

RESULTS

Thirty-four eligible genotype-phenotype associations were evaluated with data and clarifications contributed from diverse investigators. Replication data from subsequent GWA studies could be obtained for 26 of them. Of those, 19 associations (73%) reached P ≤ 5 × 10(-8) for the same or a related trait implicating either the exact same allele or one in very high linkage disequilibrium and 17 reached P < 10(-8). If the seven associations that did not reach P ≤ 5 × 10(-8) when additional data were considered are assumed to have been false-positives, the false-discovery rate for borderline associations is estimated to be 27% [95% confidence interval (CI) 12-48%]. For five associations, the current P-value is > 10(-6) [corresponding false-discovery rate 19% (95% CI 7-39%)].

CONCLUSION

A substantial proportion, but not all, of the associations with borderline genome-wide significance represent replicable, possibly genuine associations. Our empirical evaluation suggests a possible relaxation in the current GWS threshold.

Association of FTO gene variants with adiposity in African-American adolescents

The prevalence of obesity continues to increase significantly, with the largest rise in the African-American adolescents. Genetic contributions to obesity are being identified with the advent of genome-wide association studies (GWAS). Specifically, variants of the fat mass and obesity associated (FTO) gene have been associated with obesity in populations of European descent. The studies in African Americans have been inconclusive. To further evaluate the association of the FTO gene and adiposity in African Americans, we genotyped 47 single-nucleotide polymorphisms (SNPs), including seven SNPs previously reported to be significant in the literature in a cohort consisting of 561 non-Hispanic white and 497 African-American individuals. Analysis of our data showed 17 SNPs to be associated with BMI Z-score (BMI-Z) in our study population. The strongest association was found in the African Americans. The most significant SNP was rs8057044, which was associated with BMI-Z in the African Americans (P = 0.00054). SNP rs9939609 was found to be significant in the non-Hispanic white population (P = 0.028). Our data confirm the association between FTO and adiposity suggesting that FTO is a childhood obesity susceptibility gene. Our data also identify a novel SNP of the FTO gene (rs8057044) that is associated with measures of adiposity in the African-American population.

A new multitest correction (SGoF) that increases its statistical power when increasing the number of tests

BACKGROUND

The detection of true significant cases under multiple testing is becoming a fundamental issue when analyzing high-dimensional biological data. Unfortunately, known multitest adjustments reduce their statistical power as the number of tests increase. We propose a new multitest adjustment, based on a sequential goodness of fit metatest (SGoF), which increases its statistical power with the number of tests. The method is compared with Bonferroni and FDR-based alternatives by simulating a multitest context via two different kinds of tests: 1) one-sample t-test, and 2) homogeneity G-test.

RESULTS

It is shown that SGoF behaves especially well with small sample sizes when 1) the alternative hypothesis is weakly to moderately deviated from the null model, 2) there are widespread effects through the family of tests, and 3) the number of tests is large.

CONCLUSION

Therefore, SGoF should become an important tool for multitest adjustment when working with high-dimensional biological data.