An exhaustive ADDIS principle for online FWER control

In this paper, we consider online multiple testing with familywise error rate (FWER) control, where the probability of committing at least one type I error will remain under control while testing a possibly infinite sequence of hypotheses over time. Currently, adaptive-discard (ADDIS) procedures seem to be the most promising online procedures with FWER control in terms of power. Now, our main contribution is a uniform improvement of the ADDIS principle and thus of all ADDIS procedures. This means, the methods we propose reject as least as much hypotheses as ADDIS procedures and in some cases even more, while maintaining FWER control. In addition, we show that there is no other FWER controlling procedure that enlarges the event of rejecting any hypothesis. Finally, we apply the new principle to derive uniform improvements of the ADDIS-Spending and ADDIS-Graph.

Using the 'Leapfrog' Design as a Simple Form of Adaptive Platform Trial to Develop, Test, and Implement Treatment Personalization Methods in Routine Practice

The route for the development, evaluation and dissemination of personalized psychological therapies is complex and challenging. In particular, the large sample sizes needed to provide adequately powered trials of newly-developed personalization approaches means that the traditional treatment development route is extremely inefficient. This paper outlines the promise of adaptive platform trials (APT) embedded within routine practice as a method to streamline development and testing of personalized psychological therapies, and close the gap to implementation in real-world settings. It focuses in particular on a recently-developed simplified APT design, the 'leapfrog' trial, illustrating via simulation how such a trial may proceed and the advantages it can bring, for example in terms of reduced sample sizes. Finally it discusses models of how such trials could be implemented in routine practice, including potential challenges and caveats, alongside a longer-term perspective on the development of personalized psychological treatments.

Online false discovery rate control for LORD++ and SAFFRON under positive, local dependence

Online testing procedures assume that hypotheses are observed in sequence, and allow the significance thresholds for upcoming tests to depend on the test statistics observed so far. Some of the most popular online methods include alpha investing, LORD++, and SAFFRON. These three methods have been shown to provide online control of the "modified" false discovery rate (mFDR) under a condition known as CS. However, to our knowledge, LORD++ and SAFFRON have only been shown to control the traditional false discovery rate (FDR) under an independence condition on the test statistics. Our work bolsters these results by showing that SAFFRON and LORD++ additionally ensure online control of the FDR under a "local" form of nonnegative dependence. Further, FDR control is maintained under certain types of adaptive stopping rules, such as stopping after a certain number of rejections have been observed. Because alpha investing can be recovered as a special case of the SAFFRON framework, our results immediately apply to alpha investing as well. In the process of deriving these results, we also formally characterize how the conditional super-uniformity assumption implicitly limits the allowed p-value dependencies. This implicit limitation is important not only to our proposed FDR result, but also to many existing mFDR results.

In-depth quantification of bimanual coordination using the Kinarm exoskeleton robot in children with unilateral cerebral palsy

BACKGROUND

Robots have been proposed as tools to measure bimanual coordination in children with unilateral cerebral palsy (uCP). However, previous research only examined one task and clinical interpretation remains challenging due to the large amount of generated data. This cross-sectional study aims to examine bimanual coordination by using multiple bimanual robotics tasks in children with uCP, and their relation to task execution and unimanual performance.

METHODS

The Kinarm exoskeleton robot was used in 50 children with uCP (mean age: 11 years 11 months ± 2 years 10 months, Manual Ability Classification system (MACS-levels: l = 27, ll = 16, lll = 7)) and 50 individually matched typically developing children (TDC). All participants performed three tasks: object-hit (hit falling balls), ball-on-bar (balance a ball on a bar while moving to a target) and circuit task (move a cursor along a circuit by making horizontal and vertical motions with their right and left hand, respectively). Bimanual parameters provided information about bimanual coupling and interlimb differences. Differences between groups and MACS-levels were investigated using ANCOVA with age as covariate (α < 0.05, [Formula: see text]). Correlation analysis (r) linked bimanual coordination to task execution and unimanual parameters.

RESULTS

Children with uCP exhibited worse bimanual coordination compared to TDC in all tasks (p ≤ 0.05, [Formula: see text] = 0.05-0.34). The ball-on-bar task displayed high effect size differences between groups in both bimanual coupling and interlimb differences (p < 0.001, [Formula: see text] = 0.18-0.36), while the object-hit task exhibited variations in interlimb differences (p < 0.001, [Formula: see text] = 0.22-0.34) and the circuit task in bimanual coupling (p < 0.001, [Formula: see text] = 0.31). Mainly the performance of the ball-on-bar task (p < 0.05, [Formula: see text] = 0.18-0.51) was modulated by MACS-levels, showing that children with MACS-level lll had worse bimanual coordination compared to children with MACS-level l and/or II. Ball-on-bar outcomes were highly related to task execution (r = - 0.75-0.70), whereas more interlimb differences of the object-hit task were moderately associated with a worse performance of the non-dominant hand (r = - 0.69-(- 0.53)).

CONCLUSION

This study gained first insight in important robotic tasks and outcome measures to quantify bimanual coordination deficits in children with uCP. The ball-on-bar task showed the most discriminative ability for both bimanual coupling and interlimb differences, while the object-hit and circuit tasks are unique to interlimb differences and bimanual coupling, respectively.

Online multiple hypothesis testing

Modern data analysis frequently involves large-scale hypothesis testing, which naturally gives rise to the problem of maintaining control of a suitable type I error rate, such as the false discovery rate (FDR). In many biomedical and technological applications, an additional complexity is that hypotheses are tested in an online manner, one-by-one over time. However, traditional procedures that control the FDR, such as the Benjamini-Hochberg procedure, assume that all p-values are available to be tested at a single time point. To address these challenges, a new field of methodology has developed over the past 15 years showing how to control error rates for online multiple hypothesis testing. In this framework, hypotheses arrive in a stream, and at each time point the analyst decides whether to reject the current hypothesis based both on the evidence against it, and on the previous rejection decisions. In this paper, we present a comprehensive exposition of the literature on online error rate control, with a review of key theory as well as a focus on applied examples. We also provide simulation results comparing different online testing algorithms and an up-to-date overview of the many methodological extensions that have been proposed.

Online error rate control for platform trials

Platform trials evaluate multiple experimental treatments under a single master protocol, where new treatment arms are added to the trial over time. Given the multiple treatment comparisons, there is the potential for inflation of the overall type I error rate, which is complicated by the fact that the hypotheses are tested at different times and are not necessarily pre-specified. Online error rate control methodology provides a possible solution to the problem of multiplicity for platform trials where a relatively large number of hypotheses are expected to be tested over time. In the online multiple hypothesis testing framework, hypotheses are tested one-by-one over time, where at each time-step an analyst decides whether to reject the current null hypothesis without knowledge of future tests but based solely on past decisions. Methodology has recently been developed for online control of the false discovery rate as well as the familywise error rate (FWER). In this article, we describe how to apply online error rate control to the platform trial setting, present extensive simulation results, and give some recommendations for the use of this new methodology in practice. We show that the algorithms for online error rate control can have a substantially lower FWER than uncorrected testing, while still achieving noticeable gains in power when compared with the use of a Bonferroni correction. We also illustrate how online error rate control would have impacted a currently ongoing platform trial.

Online control of the False Discovery Rate in group-sequential platform trials

When testing multiple hypotheses, a suitable error rate should be controlled even in exploratory trials. Conventional methods to control the False Discovery Rate assume that all p-values are available at the time point of test decision. In platform trials, however, treatment arms enter and leave the trial at different times during its conduct. Therefore, the actual number of treatments and hypothesis tests is not fixed in advance and hypotheses are not tested at once, but sequentially. Recently, for such a setting the concept of online control of the False Discovery Rate was introduced. We propose several heuristic variations of the LOND procedure (significance Levels based On Number of Discoveries) that incorporate interim analyses for platform trials, and study their online False Discovery Rate via simulations. To adjust for the interim looks spending functions are applied with O'Brien-Fleming or Pocock type group-sequential boundaries. The power depends on the prior distribution of effect sizes, for example, whether true alternatives are uniformly distributed over time or not. We consider the choice of design parameters for the LOND procedure to maximize the overall power and investigate the impact on the False Discovery Rate by including both concurrent and non-concurrent control data.

Global FDR control across multiple RNAseq experiments

MOTIVATION

While classical approaches for controlling the false discovery rate (FDR) of RNA sequencing (RNAseq) experiments have been well described, modern research workflows and growing databases enable a new paradigm of controlling the FDR globally across RNAseq experiments in the past, present and future. The simplest analysis strategy that analyses each RNAseq experiment separately and applies an FDR correction method can lead to inflation of the overall FDR. We propose applying recently developed methodology for online multiple hypothesis testing to control the global FDR in a principled way across multiple RNAseq experiments.

RESULTS

We show that repeated application of classical repeated offline approaches has variable control of global FDR of RNAseq experiments over time. We demonstrate that the online FDR algorithms are a principled way to control FDR. Furthermore, in certain simulation scenarios, we observe empirically that online approaches have comparable power to repeated offline approaches.

AVAILABILITY AND IMPLEMENTATION

The onlineFDR package is freely available at http://www.bioconductor.org/packages/onlineFDR. Additional code used for the simulation studies can be found at https://github.com/latlio/onlinefdr_rnaseq_simulation.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Soil Microbial Co-Occurrence Patterns under Controlled-Release Urea and Fulvic Acid Applications

The increasing amount of agricultural applications of controlled-release urea (CRU) and fulvic acids (FA) demands a better understanding of FA’s effects on microbially mediated nitrogen (N) nutrient cycling. Herein, a 0–60 day laboratory experiment and a consecutive pot experiment (2016–2018) were carried out to reveal the effects of using CRU on soil microbial N-cycling processes and soil fertility, with and without the application of FA. Compared to the CRU treatment, the CRU+FA treatment boosted wheat yield by 22.1%. To reveal the mechanism of CRU+FA affecting the soil fertility, soil nutrient supply and microbial community were assessed and contrasted in this research. From 0–60 days, compared with the CRU treatment, leaching NO₃⁻-N content of CRU+FA was dramatically decreased by 12.7–84.2% in the 20 cm depth of soil column. Different fertilizers and the day of fertilization both have an impact on the soil microbiota. The most dominant bacterial phyla Actinobacteria and Proteobacteria were increased with CRU+FA treatment during 0–60 days. Network analysis revealed that microbial co-occurrence grew more intensive during the CRU+FA treatment, and the environmental change enhanced the microbial community. The CRU+FA treatment, in particular, significantly decreased the relative abundance of Sphingomonas, Lysobacter and Nitrospira associated with nitrification reactions, Nocardioides and Gaiella related to denitrification reactions. Meanwhile, the CRU+FA treatment grew the relative abundance of Ensifer, Blastococcus, and Pseudolabrys that function in N fixation, and then could reduce NH₄⁺-N and NO₃⁻-N leaching and improve the soil nutrient supply. In conclusion, the synergistic effects of slow nutrition release of CRU and growth promoting of FA could improve the soil microbial community of N cycle, reduce the loss of nutrients, and increase the wheat yield.

A Bottom-up Approach to Testing Hypotheses That Have a Branching Tree Dependence Structure, with Error Rate Control

Modern statistical analyses often involve testing large numbers of hypotheses. In many situations, these hypotheses may have an underlying tree structure that both helps determine the order that tests should be conducted but also imposes a dependency between tests that must be accounted for. Our motivating example comes from testing the association between a trait of interest and groups of microbes that have been organized into operational taxonomic units (OTUs) or amplicon sequence variants (ASVs). Given p-values from association tests for each individual OTU or ASV, we would like to know if we can declare a certain species, genus, or higher taxonomic group to be associated with the trait. For this problem, a bottom-up testing algorithm that starts at the lowest level of the tree (OTUs or ASVs) and proceeds upward through successively higher taxonomic groupings (species, genus, family etc.) is required. We develop such a bottom-up testing algorithm that controls a novel error rate that we call the false selection rate. By simulation, we also show that our approach is better at finding driver taxa, the highest level taxa below which there are dense association signals. We illustrate our approach using data from a study of the microbiome among patients with ulcerative colitis and healthy controls.

MRPC: An R Package for Inference of Causal Graphs

Understanding the causal relationships between variables is a central goal of many scientific inquiries. Causal relationships may be represented by directed edges in a graph (or equivalently, a network). In biology, for example, gene regulatory networks may be viewed as a type of causal networks, where X→Y represents gene X regulating (i.e., being causal to) gene Y. However, existing general-purpose graph inference methods often result in a high number of false edges, whereas current causal inference methods developed for observational data in genomics can handle only limited types of causal relationships. We present MRPC (a PC algorithm with the principle of Mendelian Randomization), an R package that learns causal graphs with improved accuracy over existing methods. Our algorithm builds on the powerful PC algorithm (named after its developers Peter Spirtes and Clark Glymour), a canonical algorithm in computer science for learning directed acyclic graphs. The improvements in MRPC result in increased accuracy in identifying v-structures (i.e., X→Y←Z), and robustness to how the nodes are arranged in the input data. In the special case of genomic data that contain genotypes and phenotypes (e.g., gene expression) at the individual level, MRPC incorporates the principle of Mendelian randomization as constraints on edge direction to help orient the edges. MRPC allows for inference of causal graphs not only for general purposes, but also for biomedical data where multiple types of data may be input to provide evidence for causality. The R package is available on CRAN and is a free open-source software package under a GPL (≥2) license.

Statistical consideration when adding new arms to ongoing clinical trials: the potentials and the caveats

BACKGROUND

Platform trials improve the efficiency of the drug development process through flexible features such as adding and dropping arms as evidence emerges. The benefits and practical challenges of implementing novel trial designs have been discussed widely in the literature, yet less consideration has been given to the statistical implications of adding arms. MAIN: We explain different statistical considerations that arise from allowing new research interventions to be added in for ongoing studies. We present recent methodology development on addressing these issues and illustrate design and analysis approaches that might be enhanced to provide robust inference from platform trials. We also discuss the implication of changing the control arm, how patient eligibility for different arms may complicate the trial design and analysis, and how operational bias may arise when revealing some results of the trials. Lastly, we comment on the appropriateness and the application of platform trials in phase II and phase III settings, as well as publicly versus industry-funded trials.

CONCLUSION

Platform trials provide great opportunities for improving the efficiency of evaluating interventions. Although several statistical issues are present, there are a range of methods available that allow robust and efficient design and analysis of these trials.

Online control of the familywise error rate

Biological research often involves testing a growing number of null hypotheses as new data are accumulated over time. We study the problem of online control of the familywise error rate, that is testing an a priori unbounded sequence of hypotheses (p-values) one by one over time without knowing the future, such that with high probability there are no false discoveries in the entire sequence. This paper unifies algorithmic concepts developed for offline (single batch) familywise error rate control and online false discovery rate control to develop novel online familywise error rate control methods. Though many offline familywise error rate methods (e.g., Bonferroni, fallback procedures and Sidak's method) can trivially be extended to the online setting, our main contribution is the design of new, powerful, adaptive online algorithms that control the familywise error rate when the p-values are independent or locally dependent in time. Our numerical experiments demonstrate substantial gains in power, that are also formally proved in an idealized Gaussian sequence model. A promising application to the International Mouse Phenotyping Consortium is described.

Graphical approaches for the control of generalized error rates

When simultaneously testing multiple hypotheses, the usual approach in the context of confirmatory clinical trials is to control the familywise error rate (FWER), which bounds the probability of making at least one false rejection. In many trial settings, these hypotheses will additionally have a hierarchical structure that reflects the relative importance and links between different clinical objectives. The graphical approach of Bretz et al (2009) is a flexible and easily communicable way of controlling the FWER while respecting complex trial objectives and multiple structured hypotheses. However, the FWER can be a very stringent criterion that leads to procedures with low power, and may not be appropriate in exploratory trial settings. This motivates controlling generalized error rates, particularly when the number of hypotheses tested is no longer small. We consider the generalized familywise error rate (k-FWER), which is the probability of making k or more false rejections, as well as the tail probability of the false discovery proportion (FDP), which is the probability that the proportion of false rejections is greater than some threshold. We also consider asymptotic control of the false discovery rate, which is the expectation of the FDP. In this article, we show how to control these generalized error rates when using the graphical approach and its extensions. We demonstrate the utility of the resulting graphical procedures on three clinical trial case studies.

onlineFDR: an R package to control the false discovery rate for growing data repositories

Summary

In many areas of biological research, hypotheses are tested in a sequential manner, without having access to future P-values or even the number of hypotheses to be tested. A key setting where this online hypothesis testing occurs is in the context of publicly available data repositories, where the family of hypotheses to be tested is continually growing as new data is accumulated over time. Recently, Javanmard and Montanari proposed the first procedures that control the FDR for online hypothesis testing. We present an R package, onlineFDR, which implements these procedures and provides wrapper functions to apply them to a historic dataset or a growing data repository.

Availability and implementation

The R package is freely available through Bioconductor (http://www.bioconductor.org/packages/onlineFDR).

Supplementary information

Supplementary data are available at Bioinformatics online.

Proactive visual and statistical analysis of genomic data in Epiviz

MOTIVATION

Integrative analysis of genomic data that includes statistical methods in combination with visual exploration has gained widespread adoption. Many existing methods involve a combination of tools and resources: user interfaces that provide visualization of large genomic datasets, and computational environments that focus on data analyses over various subsets of a given dataset. Over the last few years, we have developed Epiviz as an integrative and interactive genomic data analysis tool that incorporates visualization tightly with state-of-the-art statistical analysis framework.

RESULTS

In this article, we present Epiviz Feed, a proactive and automatic visual analytics system integrated with Epiviz that alleviates the burden of manually executing data analysis required to test biologically meaningful hypotheses. Results of interest that are proactively identified by server-side computations are listed as notifications in a feed. The feed turns genomic data analysis into a collaborative work between the analyst and the computational environment, which shortens the analysis time and allows the analyst to explore results efficiently.

We discuss three ways where the proposed system advances the field of genomic data analysis: (i) takes the first step of proactive data analysis by utilizing available CPU power from the server to automate the analysis process; (ii) summarizes hypothesis test results in a way that analysts can easily understand and investigate; (iii) enables filtering and grouping of analysis results for quick search. This effort provides initial work on systems that substantially expand how computational and visualization frameworks can be tightly integrated to facilitate interactive genomic data analysis.

AVAILABILITY AND IMPLEMENTATION

The source code for Epiviz Feed application is available at http://github.com/epiviz/epiviz_feed_polymer. The Epiviz Computational Server is available at http://github.com/epiviz/epiviz-feed-computation. Please refer to Epiviz documentation site for details: http://epiviz.github.io/.

Sequential Tests of Multiple Hypotheses Controlling False Discovery and Nondiscovery Rates

We propose a general and flexible procedure for testing multiple hypotheses about sequential (or streaming) data that simultaneously controls both the false discovery rate (FDR) and false nondiscovery rate (FNR) under minimal assumptions about the data streams which may differ in distribution, dimension, and be dependent. All that is needed is a test statistic for each data stream that controls its conventional type I and II error probabilities, and no information or assumptions are required about the joint distribution of the statistics or data streams. The procedure can be used with sequential, group sequential, truncated, or other sampling schemes. The procedure is a natural extension of Benjamini and Hochberg's (1995) widely-used fixed sample size procedure to the domain of sequential data, with the added benefit of simultaneous FDR and FNR control that sequential sampling affords. We prove the procedure's error control and give some tips for implementation in commonly encountered testing situations.

An online updating approach for testing the proportional hazards assumption with streams of survival data

The Cox model-which remains the first choice for analyzing time-to-event data, even for large data sets-relies on the proportional hazards (PH) assumption. When survival data arrive sequentially in chunks, a fast and minimally storage intensive approach to test the PH assumption is desirable. We propose an online updating approach that updates the standard test statistic as each new block of data becomes available and greatly lightens the computational burden. Under the null hypothesis of PH, the proposed statistic is shown to have the same asymptotic distribution as the standard version computed on an entire data stream with the data blocks pooled into one data set. In simulation studies, the test and its variant based on most recent data blocks maintain their sizes when the PH assumption holds and have substantial power to detect different violations of the PH assumption. We also show in simulation that our approach can be used successfully with "big data" that exceed a single computer's computational resources. The approach is illustrated with the survival analysis of patients with lymphoma cancer from the Surveillance, Epidemiology, and End Results Program. The proposed test promptly identified deviation from the PH assumption, which was not captured by the test based on the entire data.