Randomization inference for treatment effects on a binary outcome

Two methods are developed for constructing randomization-based confidence sets for the average effect of a treatment on a binary outcome. The methods are nonparametric and require no assumptions about random sampling from a larger population. Both of the resulting 1 - α confidence sets are exact in the sense that the probability of containing the true treatment effect is at least 1 - α. Both types of confidence sets are also guaranteed to have width no greater than one. In contrast, a previously proposed asymptotic confidence interval is not exact and may have width greater than 1. The first approach combines Bonferroni-adjusted prediction sets for the attributable effects in the treated and untreated. The second method entails inverting a permutation test. Simulations are presented comparing the two randomization-based confidence sets with the asymptotic interval as well as the standard Wald confidence interval and a commonly used exact interval for the difference in binomial proportions. Results show for small to moderate sample sizes that the permutation confidence set attains the narrowest width on average among the methods that maintain nominal coverage. Extensions that allow for stratifying on categorical baseline covariates are also discussed.

Analysis of repeated low-dose challenge studies

Preclinical evaluation of candidate human immunodeficiency virus (HIV) vaccines entails challenge studies whereby non-human primates such as macaques are vaccinated with either an active or control vaccine and then challenged (exposed) with a simian-version of HIV. Repeated low-dose challenge (RLC) studies in which each macaque is challenged multiple times (either until infection or some maximum number of challenges is reached) are becoming more common in an effort to mimic natural exposure to HIV in humans. Statistical methods typically employed for the testing for a vaccine effect in RLC studies include a modified version of Fisher's exact test as well as large sample approaches such as the usual log-rank test. Unfortunately, these methods are not guaranteed to provide a valid test for the effect of vaccination. On the other hand, valid tests for vaccine effect such as the exact log-rank test may not be easy to implement using software available to many researchers. This paper details which statistical approaches are appropriate for the analysis of RLC studies, and how to implement these methods easily in SAS or R.

Confounder selection strategies targeting stable treatment effect estimators

Inferring the causal effect of a treatment on an outcome in an observational study requires adjusting for observed baseline confounders to avoid bias. However, adjusting for all observed baseline covariates, when only a subset are confounders of the effect of interest, is known to yield potentially inefficient and unstable estimators of the treatment effect. Furthermore, it raises the risk of finite-sample bias and bias due to model misspecification. For these stated reasons, confounder (or covariate) selection is commonly used to determine a subset of the available covariates that is sufficient for confounding adjustment. In this article, we propose a confounder selection strategy that focuses on stable estimation of the treatment effect. In particular, when the propensity score (PS) model already includes covariates that are sufficient to adjust for confounding, then the addition of covariates that are associated with either treatment or outcome alone, but not both, should not systematically change the effect estimator. The proposal, therefore, entails first prioritizing covariates for inclusion in the PS model, then using a change-in-estimate approach to select the smallest adjustment set that yields a stable effect estimate. The ability of the proposal to correctly select confounders, and to ensure valid inference of the treatment effect following data-driven covariate selection, is assessed empirically and compared with existing methods using simulation studies. We demonstrate the procedure using three different publicly available datasets commonly used for causal inference.

Randomization inference with general interference and censoring

Interference occurs between individuals when the treatment (or exposure) of one individual affects the outcome of another individual. Previous work on causal inference methods in the presence of interference has focused on the setting where it is a priori assumed that there is "partial interference," in the sense that individuals can be partitioned into groups wherein there is no interference between individuals in different groups. Bowers et al. (2012, Political Anal, 21, 97-124) and Bowers et al. (2016, Political Anal, 24, 395-403) consider randomization-based inferential methods that allow for more general interference structures in the context of randomized experiments. In this paper, extensions of Bowers et al. that allow for failure time outcomes subject to right censoring are proposed. Permitting right-censored outcomes is challenging because standard randomization-based tests of the null hypothesis of no treatment effect assume that whether an individual is censored does not depend on treatment. The proposed extension of Bowers et al. to allow for censoring entails adapting the method of Wang et al. (2010, Biostatistics, 11, 676-692) for two-sample survival comparisons in the presence of unequal censoring. The methods are examined via simulation studies and utilized to assess the effects of cholera vaccination in an individually randomized trial of 73 000 children and women in Matlab, Bangladesh.

Response to comment on 'Randomization inference for treatment effects on a binary outcome'

We thank Professor Yasutaka Chiba [1 ] for commenting on Rigdon and Hudgens (RH) [2 ]. Chiba [1 ] described a certain exact confidence interval reported in RH as “somewhat unnatural.” Chiba also presented an alternative approach to constructing confidence intervals [3 ]. In this response, we (i) provide a simple explanation why the confidence interval in RH appeared “unnatural,” and (ii) explain the relationship between the RH [2 ] and Chiba [3 ] confidence intervals. Essentially the two approaches are equivalent, except RH entails inverting one two-sided test whereas Chiba inverts two one-sided tests. We present a more computationally efficient method (RLH) for computing the RH intervals based on Chiba’s principal stratification formulation of the problem. We also propose a third method based on Blaker [4 ] which inverts a single two-sided test but forms a confidence interval that is at least as narrow as inverting two one-sided tests. Simulation results show the RLH intervals tend to be as narrow or narrower than the Chiba and Blaker intervals on average.

The role of randomization inference in unraveling individual treatment effects in early phase vaccine trials

Randomization inference is a powerful tool in early phase vaccine trials when estimating the causal effect of a regimen against a placebo or another regimen. Randomization-based inference often focuses on testing either Fisher's sharp null hypothesis of no treatment effect for any participant or Neyman's weak null hypothesis of no sample average treatment effect. Many recent efforts have explored conducting exact randomization-based inference for other summaries of the treatment effect profile, for instance, quantiles of the treatment effect distribution function. In this article, we systematically review methods that conduct exact, randomization-based inference for quantiles of individual treatment effects (ITEs) and extend some results to a special case where naïve participants are expected not to exhibit responses to highly specific endpoints. These methods are suitable for completely randomized trials, stratified completely randomized trials, and a matched study comparing two non-randomized arms from possibly different trials. We evaluate the usefulness of these methods using synthetic data in simulation studies. Finally, we apply these methods to HIV Vaccine Trials Network Study 086 (HVTN 086) and HVTN 205 and showcase a wide range of application scenarios of the methods. R code that replicates all analyses in this article can be found in first author's GitHub page at https://github.com/Zhe-Chen-1999/ITE-Inference.

On exact randomization-based covariate-adjusted confidence intervals

Randomization-based inference using the Fisher randomization test allows for the computation of Fisher-exact P-values, making it an attractive option for the analysis of small, randomized experiments with non-normal outcomes. Two common test statistics used to perform Fisher randomization tests are the difference-in-means between the treatment and control groups and the covariate-adjusted version of the difference-in-means using analysis of covariance. Modern computing allows for fast computation of the Fisher-exact P-value, but confidence intervals have typically been obtained by inverting the Fisher randomization test over a range of possible effect sizes. The test inversion procedure is computationally expensive, limiting the usage of randomization-based inference in applied work. A recent paper by Zhu and Liu developed a closed form expression for the randomization-based confidence interval using the difference-in-means statistic. We develop an important extension of Zhu and Liu to obtain a closed form expression for the randomization-based covariate-adjusted confidence interval and give practitioners a sufficiency condition that can be checked using observed data and that guarantees that these confidence intervals have correct coverage. Simulations show that our procedure generates randomization-based covariate-adjusted confidence intervals that are robust to non-normality and that can be calculated in nearly the same time as it takes to calculate the Fisher-exact P-value, thus removing the computational barrier to performing randomization-based inference when adjusting for covariates. We also demonstrate our method on a re-analysis of phase I clinical trial data.

Testing weak nulls in matched observational studies

We develop sensitivity analyses for the sample average treatment effect in matched observational studies while allowing unit-level treatment effects to vary. The methods may be applied to studies using any optimal without-replacement matching algorithm. In contrast to randomized experiments and to paired observational studies, we show for general matched designs that over a large class of test statistics, any procedure bounding the worst-case expectation while allowing for arbitrary effect heterogeneity must be unnecessarily conservative if treatment effects are actually constant across individuals. We present a sensitivity analysis which bounds the worst-case expectation while allowing for effect heterogeneity, and illustrate why it is generally conservative if effects are constant. An alternative procedure is presented that is asymptotically sharp if treatment effects are constant, and that is valid for testing the sample average effect under additional restrictions which may be deemed benign by practitioners. Simulations demonstrate that this alternative procedure results in a valid sensitivity analysis for the weak null hypothesis under a host of reasonable data-generating processes. The procedures allow practitioners to assess robustness of estimated sample average treatment effects to hidden bias while allowing for effect heterogeneity in matched observational studies.