The uniform general signed rank test and its design sensitivity

A sensitivity analysis in an observational study tests whether the qualitative conclusions of an analysis would change if we were to allow for the possibility of limited bias due to confounding. The design sensitivity of a hypothesis test quantifies the asymptotic performance of the test in a sensitivity analysis against a particular alternative. We propose a new, nonasymptotic, distribution-free test, the uniform general signed rank test, for observational studies with paired data, and examine its performance under Rosenbaum’s sensitivity analysis model. Our test can be viewed as adaptively choosing from among a large underlying family of signed rank tests, and we show that the uniform test achieves design sensitivity equal to the maximum design sensitivity over the underlying family of signed rank tests. Our test thus achieves superior design sensitivity, indicating it will perform well in sensitivity analyses on large samples. We support this conclusion with simulations and a data example, showing that the advantages of our test extend to moderate sample sizes as well.

A conditional test with demonstrated insensitivity to unmeasured bias in matched observational studies

In an observational study matched for observed covariates, an association between treatment received and outcome exhibited may indicate not an effect caused by the treatment, but merely some bias in the allocation of treatments to individuals within matched pairs. The evidence that distinguishes moderate biases from causal effects is unevenly dispersed among possible comparisons in an observational study: some comparisons are insensitive to larger biases than others. Intuitively, larger treatment effects tend to be insensitive to larger unmeasured biases, and perhaps matched pairs can be grouped using covariates, doses or response patterns so that groups of pairs with larger treatment effects may be identified. Even if an investigator has a reasoned conjecture about where to look for insensitive comparisons, that conjecture might prove mistaken, or, when not mistaken, it might be received sceptically by other scientists who doubt the conjecture or judge it to be too convenient in light of its success with the data at hand. In this article a test is proposed that searches for insensitive findings over many comparisons, but controls the probability of falsely rejecting a true null hypothesis of no treatment effect in the presence of a bias of specified magnitude. An example is studied in which the test considers many comparisons and locates an interpretable comparison that is insensitive to larger biases than a conventional comparison based on Wilcoxon’s signed rank statistic applied to all pairs. A simulation examines the power of the proposed test. The method is implemented in the R package dstat, which contains the example and reproduces the analysis.

An exact test with high power and robustness to unmeasured confounding effects

In observational studies, it is agreed that the sensitivity of the findings to unmeasured confounders needs to be assessed. The issue is that a poor choice of test statistic can result in overstated sensitivity to hidden bias of this kind. In this article, a new adaptive test is proposed, guided by considerations of low sensitivity to hidden bias: it is tailored so that its power is greater than other leading tests, both in finite and infinite samples. One way of defining power in case of possible confounders is as the probability of reporting robustness (ie, insensitivity) of a true discovery to potential bias. In case of finite samples, we compute the power by simulations. When sample size approaches infinity, a meaningful indicator of the power is the design sensitivity, which is computed analytically and found to be better in the new test than in existing tests. Another asymptotic criterion for comparing tests when there is concern for confounders is Bahadur efficiency. The proposed test outperforms commonly used tests in terms of Bahadur efficiency in most sampling situations. The advantages of the new test mainly stem from its adaptivity: it combines two test statistics and consequently achieves the best design sensitivity and the best Bahadur efficiency of the two. As a "real-world" examination, we compare 441 daily smokers to 441 nonsmokers, to test the effect of smoking on periodontal disease. The new test is more robust to unmeasured confounders than both the Wilcoxon signed rank test and the paired t-test.

An adaptive Mantel-Haenszel test for sensitivity analysis in observational studies

In a sensitivity analysis in an observational study with a binary outcome, is it better to use all of the data or to focus on subgroups that are expected to experience the largest treatment effects? The answer depends on features of the data that may be difficult to anticipate, a trade-off between unknown effect-sizes and known sample sizes. We propose a sensitivity analysis for an adaptive test similar to the Mantel-Haenszel test. The adaptive test performs two highly correlated analyses, one focused analysis using a subgroup, one combined analysis using all of the data, correcting for multiple testing using the joint distribution of the two test statistics. Because the two component tests are highly correlated, this correction for multiple testing is small compared with, for instance, the Bonferroni inequality. The test has the maximum design sensitivity of two component tests. A simulation evaluates the power of a sensitivity analysis using the adaptive test. Two examples are presented. An R package, sensitivity2x2xk, implements the procedure.

The cross-cut statistic and its sensitivity to bias in observational studies with ordered doses of treatment

A common practice with ordered doses of treatment and ordered responses, perhaps recorded in a contingency table with ordered rows and columns, is to cut or remove a cross from the table, leaving the outer corners--that is, the high-versus-low dose, high-versus-low response corners--and from these corners to compute a risk or odds ratio. This little remarked but common practice seems to be motivated by the oldest and most familiar method of sensitivity analysis in observational studies, proposed by Cornfield et al. (1959), which says that to explain a population risk ratio purely as bias from an unobserved binary covariate, the prevalence ratio of the covariate must exceed the risk ratio. Quite often, the largest risk ratio, hence the one least sensitive to bias by this standard, is derived from the corners of the ordered table with the central cross removed. Obviously, the corners use only a portion of the data, so a focus on the corners has consequences for the standard error as well as for bias, but sampling variability was not a consideration in this early and familiar form of sensitivity analysis, where point estimates replaced population parameters. Here, this cross-cut analysis is examined with the aid of design sensitivity and the power of a sensitivity analysis.

Case Definition and Design Sensitivity

In a case-referent study, cases of disease are compared to non-cases with respect to their antecedent exposure to a treatment in an effort to determine whether exposure causes some cases of the disease. Because exposure is not randomly assigned in the population, as it would be if the population were a vast randomized trial, exposed and unexposed subjects may differ prior to exposure with respect to covariates that may or may not have been measured. After controlling for measured pre-exposure differences, for instance by matching, a sensitivity analysis asks about the magnitude of bias from unmeasured covariates that would need to be present to alter the conclusions of a study that presumed matching for observed covariates removes all bias. The definition of a case of disease affects sensitivity to unmeasured bias. We explore this issue using: (i) an asymptotic tool, the design sensitivity, (ii) a simulation for finite samples, and (iii) an example. Under favorable circumstances, a narrower case definition can yield an increase in the design sensitivity, and hence an increase in the power of a sensitivity analysis. Also, we discuss an adaptive method that seeks to discover the best case definition from the data at hand while controlling for multiple testing. An implementation in R is available as SensitivityCaseControl.