Combining adaptive designs with control of the false discovery rate--a generalized definition for a global p-value

Genome-Wide Association Studies and Haplotype-Sharing Analysis Targeting the Egg Production Traits in Shaoxing Duck

Age at first egg (AFE) and egg number (EN) are economically important traits related to egg production, as they directly influence the benefits of the poultry industry, but the molecular genetic research that affects those traits in laying ducks is still sparse. Our objective was to identify the genomic regions and candidate genes associated with AFE, egg production at 43 weeks (EP43w), and egg production at 66 weeks (EP66w) in a Shaoxing duck population using genome-wide association studies (GWASs) and haplotype-sharing analysis. Single-nucleotide polymorphism (SNP)-based genetic parameter estimates showed that the heritability was 0.15, 0.20, and 0.22 for AFE, EP43w, and EP66w, respectively. Subsequently, three univariate GWASs for AFE, EP43w, and EP66w were carried out independently. Twenty-four SNPs located on chromosome 25 within a 0.01-Mb region that spans from 4.511 to 4.521 Mb were associated with AFE. There are two CIs that affect EP43w, i.e., twenty-five SNPs were in strong linkage disequilibrium region spanning from 3.186 to 3.247 Mb on chromosome 25, a region spanning from 4.442 to 4.446 Mb on chromosome 25, and two interesting genes, ACAD8 and THYN1, that may affect EP43w in laying ducks. There are also two CIs that affect EP66w, i.e., a 2.412-Mb region that spans from 127.497 to 129.910 Mb on chromosome 2 and a 0.355-Mb region that spans from 4.481 to 4.837 Mb on chromosome 29, and CA2 and GAMT may be the putative candidate genes. Our study also found some haplotypes significantly associated with these three traits based on haplotype-sharing analysis. Overall, this study was the first publication of GWAS on egg production in laying ducks, and our findings will be helpful to provide some candidate genes and haplotypes to improve egg production performance based on breeding in laying duck. Additionally, we learned from a method called bootstrap test to verify the reliability of a GWAS with small experimental samples that users can access at https://github.com/xuwenwu24/Bootstrap-test.

Robin Hood: A cost-efficient two-stage approach to large-scale simultaneous inference with non-homogeneous sparse effects

A classical approach to experimental design in many scientific fields is to first gather all of the data and then analyze it in a single analysis. It has been recognized that in many areas such practice leaves substantial room for improvement in terms of the researcher's ability to identify relevant effects, in terms of cost efficiency, or both. Considerable attention has been paid in recent years to multi-stage designs, in which the user alternates between data collection and analysis and thereby sequentially reduces the size of the problem. However, the focus has generally been towards designs that require a hypothesis be tested in every single stage before it can be declared as rejected by the procedure. Such procedures are well-suited for homogeneous effects, i.e. effects of (almost) equal sizes, however, with effects of varying size a procedure that permits rejection at interim stages is much more suitable. Here we present precisely such multi-stage testing procedure called Robin Hood. We show that with heterogeneous effects our method substantially improves on the existing multi-stage procedures with an essentially zero efficiency trade-off in the homogeneous effect realm, which makes it especially useful in areas such as genetics, where heterogeneous effects are common. Our method improves on existing approaches in a number of ways including a novel way of performing two-sided testing in a multi-stage procedure with increased power for detecting small effects.

Twenty-five years of confirmatory adaptive designs: opportunities and pitfalls

'Multistage testing with adaptive designs' was the title of an article by Peter Bauer that appeared 1989 in the German journal Biometrie und Informatik in Medizin und Biologie. The journal does not exist anymore but the methodology found widespread interest in the scientific community over the past 25 years. The use of such multistage adaptive designs raised many controversial discussions from the beginning on, especially after the publication by Bauer and Köhne 1994 in Biometrics: Broad enthusiasm about potential applications of such designs faced critical positions regarding their statistical efficiency. Despite, or possibly because of, this controversy, the methodology and its areas of applications grew steadily over the years, with significant contributions from statisticians working in academia, industry and agencies around the world. In the meantime, such type of adaptive designs have become the subject of two major regulatory guidance documents in the US and Europe and the field is still evolving. Developments are particularly noteworthy in the most important applications of adaptive designs, including sample size reassessment, treatment selection procedures, and population enrichment designs. In this article, we summarize the developments over the past 25 years from different perspectives. We provide a historical overview of the early days, review the key methodological concepts and summarize regulatory and industry perspectives on such designs. Then, we illustrate the application of adaptive designs with three case studies, including unblinded sample size reassessment, adaptive treatment selection, and adaptive endpoint selection. We also discuss the availability of software for evaluating and performing such designs. We conclude with a critical review of how expectations from the beginning were fulfilled, and - if not - discuss potential reasons why this did not happen.

False discovery rate control in two-stage designs

Background

For gene expression or gene association studies with a large number of hypotheses the number of measurements per marker in a conventional single-stage design is often low due to limited resources. Two-stage designs have been proposed where in a first stage promising hypotheses are identified and further investigated in the second stage with larger sample sizes. For two types of two-stage designs proposed in the literature we derive multiple testing procedures controlling the False Discovery Rate (FDR) demonstrating FDR control by simulations: designs where a fixed number of top-ranked hypotheses are selected and designs where the selection in the interim analysis is based on an FDR threshold. In contrast to earlier approaches which use only the second-stage data in the hypothesis tests (pilot approach), the proposed testing procedures are based on the pooled data from both stages (integrated approach).

Results

For both selection rules the multiple testing procedures control the FDR in the considered simulation scenarios. This holds for the case of independent observations across hypotheses as well as for certain correlation structures. Additionally, we show that in scenarios with small effect sizes the testing procedures based on the pooled data from both stages can give a considerable improvement in power compared to tests based on the second-stage data only.

Conclusion

The proposed hypothesis tests provide a tool for FDR control for the considered two-stage designs. Comparing the integrated approaches for both selection rules with the corresponding pilot approaches showed an advantage of the integrated approach in many simulation scenarios.

Sequential interim analyses of survival data in DNA microarray experiments

BACKGROUND

Discovery of biomarkers that are correlated with therapy response and thus with survival is an important goal of medical research on severe diseases, e.g. cancer. Frequently, microarray studies are performed to identify genes of which the expression levels in pretherapeutic tissue samples are correlated to survival times of patients. Typically, such a study can take several years until the full planned sample size is available.Therefore, interim analyses are desirable, offering the possibility of stopping the study earlier, or of performing additional laboratory experiments to validate the role of the detected genes. While many methods correcting the multiple testing bias introduced by interim analyses have been proposed for studies of one single feature, there are still open questions about interim analyses of multiple features, particularly of high-dimensional microarray data, where the number of features clearly exceeds the number of samples. Therefore, we examine false discovery rates and power rates in microarray experiments performed during interim analyses of survival studies. In addition, the early stopping based on interim results of such studies is evaluated. As stop criterion we employ the achieved average power rate, i.e. the proportion of detected true positives, for which a new estimator is derived and compared to existing estimators.

RESULTS

In a simulation study, pre-specified levels of the false discovery rate are maintained in each interim analysis, where reduced levels as used in classical group sequential designs of one single feature are not necessary. Average power rates increase with each interim analysis, and many studies can be stopped prior to their planned end when a certain pre-specified power rate is achieved. The new estimator for the power rate slightly deviates from the true power rate but is comparable to other estimators.

CONCLUSIONS

Interim analyses of microarray experiments can provide evidence for early stopping of long-term survival studies. The developed simulation framework, which we also offer as a new R package 'SurvGenesInterim' available at http://survgenesinter.R-Forge.R-Project.org, can be used for sample size planning of the evaluated study design.

Post hoc power estimation in large-scale multiple testing problems

BACKGROUND

The statistical power or multiple Type II error rate in large-scale multiple testing problems as, for example, in gene expression microarray experiments, depends on typically unknown parameters and is therefore difficult to assess a priori. However, it has been suggested to estimate the multiple Type II error rate post hoc, based on the observed data.

METHODS

We consider a class of post hoc estimators that are functions of the estimated proportion of true null hypotheses among all hypotheses. Numerous estimators for this proportion have been proposed and we investigate the statistical properties of the derived multiple Type II error rate estimators in an extensive simulation study.

RESULTS

The performance of the estimators in terms of the mean squared error depends sensitively on the distributional scenario. Estimators based on empirical distributions of the null hypotheses are superior in the presence of strongly correlated test statistics.

AVAILABILITY

R-code to compute all considered estimators based on P-values and supplementary material is available on the authors web page http://statistics.msi.meduniwien.ac.at/index.php?page=pageszfnr.

Optimized multi-stage designs controlling the false discovery or the family-wise error rate

When a large number of hypotheses are investigated, we propose multi-stage designs where in each interim analysis promising hypotheses are screened, which are investigated in further stages. Given a fixed overall number of observations, this allows one to spend more observations for promising hypotheses than with single-stage designs, where the observations are equally distributed among all considered hypotheses. We propose multi-stage procedures controlling either the family-wise error rate (FWER) or the false discovery rate (FDR) and derive asymptotically optimal stopping boundaries and sample size allocations (across stages) to maximize the power of the procedure. Optimized two-stage designs lead to a considerable increase in power compared with the classical single-stage design. Going from two to three stages additionally leads to a distinctive increase in power. Adding a fourth stage leads to a further improvement, which is, however, less pronounced. Surprisingly, we found only small differences in power between optimized integrated designs, where the data of all stages are used in the final test statistics, and optimized pilot designs where only the data from the final stage are used for testing. However, the integrated design controlling the FDR appeared to be more robust against misspecifications in the planning phase. Additionally, we found that with increasing number of stages the drop in power when controlling the FWER instead of the FDR becomes negligible. Our investigations show that the crucial point is not the choice of the error rate or the type of design, but the sequential nature of the trial where non-promising hypotheses are dropped in the early phases of the experiment.