On Sensitivity Value of Pair-Matched Observational Studies

A second evidence factor for a second control group

In an observational study of the effects caused by a treatment, a second control group is used in an effort to detect bias from unmeasured covariates, and the investigator is content if no evidence of bias is found. This strategy is not entirely satisfactory: two control groups may differ significantly, yet the difference may be too small to invalidate inferences about the treatment, or the control groups may not differ yet nonetheless fail to provide a tangible strengthening of the evidence of a treatment effect. Is a firmer conclusion possible? Is there a way to analyze a second control group such that the data might report measurably strengthened evidence of cause and effect, that is, insensitivity to larger unmeasured biases? Evidence factor analyses are not commonly used with a second control group: most analyses compare the treated group to each control group, but analyses of that kind are partially redundant; so, they do not constitute evidence factors. An alternative analysis is proposed here, one that does yield two evidence factors, and with a carefully designed test statistic, is capable of extracting strong evidence from the second factor. The new technical work here concerns the development of a test statistic with high design sensitivity and high Bahadur efficiency in a sensitivity analysis for the second factor. A study of binge drinking as a cause of high blood pressure is used as an illustration.

Sensitivity analyses informed by tests for bias in observational studies

In an observational study, the treatment received and the outcome exhibited may be associated in the absence of an effect caused by the treatment, even after controlling for observed covariates. Two tactics are common: (i) a test for unmeasured bias may be obtained using a secondary outcome for which the effect is known and (ii) a sensitivity analysis may explore the magnitude of unmeasured bias that would need to be present to explain the observed association as something other than an effect caused by the treatment. Can such a test for unmeasured bias inform the sensitivity analysis? If the test for bias does not discover evidence of unmeasured bias, then ask: Are conclusions therefore insensitive to larger unmeasured biases? Conversely, if the test for bias does find evidence of bias, then ask: What does that imply about sensitivity to biases? This problem is formulated in a new way as a convex quadratically constrained quadratic program and solved on a large scale using interior point methods by a modern solver. That is, a convex quadratic function of N variables is minimized subject to constraints on linear and convex quadratic functions of these variables. The quadratic function that is minimized is a statistic for the primary outcome that is a function of the unknown treatment assignment probabilities. The quadratic function that constrains this minimization is a statistic for subsidiary outcome that is also a function of these same unknown treatment assignment probabilities. In effect, the first statistic is minimized over a confidence set for the unknown treatment assignment probabilities supplied by the unaffected outcome. This process avoids the mistake of interpreting the failure to reject a hypothesis as support for the truth of that hypothesis. The method is illustrated by a study of the effects of light daily alcohol consumption on high-density lipoprotein (HDL) cholesterol levels. In this study, the method quickly optimizes a nonlinear function of N = 800 $N=800$ variables subject to linear and quadratic constraints. In the example, strong evidence of unmeasured bias is found using the subsidiary outcome, but, perhaps surprisingly, this finding makes the primary comparison insensitive to larger biases.

Analysis of local sensitivity to nonignorability with missing outcomes and predictors

The ISNI (index of sensitivity to local nonignorability) method quantifies local sensitivity of parametric inferences to nonignorable missingness in an outcome variable. Here we extend ISNI to the situations where both outcomes and predictors can be missing and where the missingness mechanism can be either parametric or semi-parametric. We define the quantity MinNI (minimum nonignorability) to be an approximation to the norm of the smallest value of the transformed nonignorability that gives a nonnegligible displacement of the estimate of the parameter of interest. We illustrate our method in a complete data set from which we synthetically delete observations according to various patterns. We then apply the method to real-data examples involving the normal linear model and conditional logistic regression.

A Semiparametric Approach to Model-Based Sensitivity Analysis in Observational Studies

When drawing causal inference from observational data, there is almost always concern about unmeasured confounding. One way to tackle this is to conduct a sensitivity analysis. One widely-used sensitivity analysis framework hypothesizes the existence of a scalar unmeasured confounder U and asks how the causal conclusion would change were U measured and included in the primary analysis. Work along this line often makes various parametric assumptions on U, for the sake of mathematical and computational convenience. In this article, we further this line of research by developing a valid sensitivity analysis that leaves the distribution of U unrestricted. Compared to many existing methods in the literature, our method allows for a larger and more flexible family of models, mitigates observable implications (Franks et al., 2019), and works seamlessly with any primary analysis that models the outcome regression parametrically. We construct both pointwise confidence intervals and confidence bands that are uniformly valid over a given sensitivity parameter space, thus formally accounting for unknown sensitivity parameters. We apply our proposed method on an influential yet controversial study of the causal relationship between war experiences and political activeness using observational data from Uganda.

Testing Biased Randomization Assumptions and Quantifying Imperfect Matching and Residual Confounding in Matched Observational Studies

One central goal of design of observational studies is to embed non-experimental data into an approximate randomized controlled trial using statistical matching. Despite empirical researchers' best intention and effort to create high-quality matched samples, residual imbalance due to observed covariates not being well matched often persists. Although statistical tests have been developed to test the randomization assumption and its implications, few provide a means to quantify the level of residual confounding due to observed covariates not being well matched in matched samples. In this article, we develop two generic classes of exact statistical tests for a biased randomization assumption. One important by-product of our testing framework is a quantity called residual sensitivity value (RSV), which provides a means to quantify the level of residual confounding due to imperfect matching of observed covariates in a matched sample. We advocate taking into account RSV in the downstream primary analysis. The proposed methodology is illustrated by re-examining a famous observational study concerning the effect of right heart catheterization (RHC) in the initial care of critically ill patients. Code implementing the method can be found in the supplementary materials.

A statistic with demonstrated insensitivity to unmeasured bias for 2 × 2 × S tables in observational studies

Are weak associations between a treatment and a binary outcome always sensitive to small unmeasured biases in observational studies? This possibility is often discussed in epidemiology. The familiar Mantel-Haenszel test for a 2 × 2 × S $$ 2\times 2\times S $$ contingency table exaggerates sensitivity to unmeasured biases when the population odds ratios vary among the S $$ S $$ strata. A statistic built from several components, here from the S $$ S $$  strata, is said to have demonstrated insensitivity to bias if it uses only those components that provide indications of insensitivity to bias. Briefly, such a statistic is a d $$ d $$ -statistic. There are 2 S - 1 $$ {2}^S-1 $$ candidate statistics with S $$ S $$ strata, and a d $$ d $$ -statistic considers them all.  To have level α $$ \alpha $$ , a test based on a d $$ d $$ -statistic must pay a price for its double use of the data, but as the sample size increases, that price becomes small, while the gain may be large. The price is paid by conditioning on the limited information used to identify components that are insensitive to a bias of specified magnitude, basing the test result on the information that remains after conditioning. In large samples, the d $$ d $$ -statistic achieves the largest possible design sensitivity, so it does not exaggerate sensitivity to unmeasured bias. A simulation verifies that the large sample result has traction in samples of practical size. A study of sunlight as a cause of cataract is used to illustrate issues and methods. Several extensions of the method are discussed. An R package dstat2x2xk implements the method.

A conditional test with demonstrated insensitivity to unmeasured bias in matched observational studies

In an observational study matched for observed covariates, an association between treatment received and outcome exhibited may indicate not an effect caused by the treatment, but merely some bias in the allocation of treatments to individuals within matched pairs. The evidence that distinguishes moderate biases from causal effects is unevenly dispersed among possible comparisons in an observational study: some comparisons are insensitive to larger biases than others. Intuitively, larger treatment effects tend to be insensitive to larger unmeasured biases, and perhaps matched pairs can be grouped using covariates, doses or response patterns so that groups of pairs with larger treatment effects may be identified. Even if an investigator has a reasoned conjecture about where to look for insensitive comparisons, that conjecture might prove mistaken, or, when not mistaken, it might be received sceptically by other scientists who doubt the conjecture or judge it to be too convenient in light of its success with the data at hand. In this article a test is proposed that searches for insensitive findings over many comparisons, but controls the probability of falsely rejecting a true null hypothesis of no treatment effect in the presence of a bias of specified magnitude. An example is studied in which the test considers many comparisons and locates an interpretable comparison that is insensitive to larger biases than a conventional comparison based on Wilcoxon’s signed rank statistic applied to all pairs. A simulation examines the power of the proposed test. The method is implemented in the R package dstat, which contains the example and reproduces the analysis.

An exact test with high power and robustness to unmeasured confounding effects

In observational studies, it is agreed that the sensitivity of the findings to unmeasured confounders needs to be assessed. The issue is that a poor choice of test statistic can result in overstated sensitivity to hidden bias of this kind. In this article, a new adaptive test is proposed, guided by considerations of low sensitivity to hidden bias: it is tailored so that its power is greater than other leading tests, both in finite and infinite samples. One way of defining power in case of possible confounders is as the probability of reporting robustness (ie, insensitivity) of a true discovery to potential bias. In case of finite samples, we compute the power by simulations. When sample size approaches infinity, a meaningful indicator of the power is the design sensitivity, which is computed analytically and found to be better in the new test than in existing tests. Another asymptotic criterion for comparing tests when there is concern for confounders is Bahadur efficiency. The proposed test outperforms commonly used tests in terms of Bahadur efficiency in most sampling situations. The advantages of the new test mainly stem from its adaptivity: it combines two test statistics and consequently achieves the best design sensitivity and the best Bahadur efficiency of the two. As a "real-world" examination, we compare 441 daily smokers to 441 nonsmokers, to test the effect of smoking on periodontal disease. The new test is more robust to unmeasured confounders than both the Wilcoxon signed rank test and the paired t-test.

Model assisted sensitivity analyses for hidden bias with binary outcomes

In medical and health sciences, observational studies are a major data source for inferring causal relationships. Unlike randomized experiments, observational studies are vulnerable to the hidden bias introduced by unmeasured confounders. The impact of unmeasured covariates on the causal effect can be assessed by conducting a sensitivity analysis. A comprehensive framework of sensitivity analyses has been developed for matching designs. Sensitivity parameters are introduced to capture the association between the missing covariates and the exposure or the outcome. Fixing sensitivity parameter values, it is possible to compute the bounds of the p-value of a randomization test on causal effects. We propose a model assisted sensitivity analysis with binary outcomes for the general 1:k matching design, which provides results equivalent to the conventional nonparametric approach in large sample. By introducing a conditional logistic outcome model, we substantially simplify the implementation and interpretation of the sensitivity analysis. More importantly, we are able to provide a closed form representation for the set of sensitivity parameters for which the maximum p-values are non-significant. This methodology can be easily extended to matching designs with multilevel treatments. We illustrate our method using a U.S. trauma care database to examine mortality difference between trauma care levels.