Design Sensitivity and Efficiency in Observational Studies

Impact of the matching algorithm on the treatment effect estimate: A neutral comparison study

Propensity score matching is increasingly being used in the medical literature. Choice of matching algorithms, reporting quality, and estimands are oftentimes not discussed. We evaluated the impact of propensity score matching algorithms, based on a recent clinical dataset, with three commonly used outcomes. The resulting estimands for different strengths of treatment effects were compared in a neutral comparison study and based on a thoroughly designed simulation study. Different algorithms yielded different levels of balance after matching. Along with full matching and genetic matching with replacement, good balance was achieved with nearest neighbor matching with caliper but thereby more than one fifth of the treated units were discarded. Average marginal treatment effect estimates were least biased with genetic or nearest neighbor matching, both with replacement and full matching. Double adjustment yielded conditional treatment effects that were closer to the true values, throughout. The choice of the matching algorithm had an impact on covariate balance after matching as well as treatment effect estimates. In comparison, genetic matching with replacement yielded better covariate balance than all other matching algorithms. A literature review in the British Medical Journal including its subjournals revealed frequent use of propensity score matching; however, the use of different matching algorithms before treatment effect estimation was only reported in one out of 21 studies. Propensity score matching is a methodology for causal treatment effect estimation from observational data; however, the methodological difficulties and low reporting quality in applied medical research need to be addressed.

Addressing unmeasured confounding bias with a prior knowledge guided approach: coronary artery bypass grafting (CABG) versus percutaneous coronary intervention (PCI) in patients with stable ischemic heart disease

Unmeasured confounding undermines the validity of observational studies. Although randomized clinical trials (RCTs) are considered the "gold standard" of study types, we often observe divergent findings between RCTs and empirical settings. We present the "L-table", a simulation-based, prior knowledge (e.g., RCTs) guided approach that estimates the true effect adjusting for the potential influence of unmeasured confounders when using observational data. Using electronic health record data from Kaiser Permanente Southern California, we compare the effectiveness of coronary artery bypass grafting (CABG) and percutaneous coronary intervention (PCI) on endpoints at 1, 3, 5, and 10 years for patients with stable ischemic heart disease. We applied the L-table approach to the propensity score adjusted cohort to derive the omitted-confounder-adjusted estimated effects. After the L-table adjustment, CABG patients are 57.6% less likely to encounter major adverse cardiac and cerebrovascular event (MACCE) at 1 year (OR [95% CI] 0.424 [0.396, 0.517]), 56.4% less likely at 3 years (OR [95% CI] 0.436 [0.369, 0.527]), and 48.9% less likely at 5 years (OR [95% CI] 0.511 [0.451, 0.538]). CABG patients are also 49.5% less likely to die by the end of 10 years than PCI patients (OR [95% CI] 0.505 [0.446, 0.582]). We found the estimated true effects all shifted towards CABG as a more effective procedure that led to better health outcomes compared to PCI. Unlike existing sensitivity tools, the L-table approach explicitly lays out probable values and can therefore better support clinical decision-making. We recommend using L-table as a supplement to available techniques of sensitivity analysis.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10742-022-00282-y.

Participation, not penalties: Community involvement and equitable governance contribute to more effective multiuse protected areas

Accelerating ecosystem degradation has spurred proposals to vastly expand the extent of protected areas (PAs), potentially affecting the livelihoods and well-being of indigenous peoples and local communities (IPLCs) worldwide. The benefits of multiuse PAs that elevate the role of IPLCs in management have long been recognized. However, quantitative examinations of how resource governance and the distribution of management rights affect conservation outcomes are vital for long-term sustainability. Here, we use a long-term, quasi-experimental monitoring dataset from four Indonesian marine PAs that demonstrates that multiuse PAs can increase fish biomass, but incorporating multiple governance principles into management regimes and enforcing rules equitably are critical to achieve ecological benefits. Furthermore, we show that PAs predicated primarily on enforcing penalties can be less effective than those where IPLCs have the capacity to engage in management. Our results suggest that well-governed multiuse PAs can achieve conservation objectives without undermining the rights of IPLCs.

Ten Rules for Conducting Retrospective Pharmacoepidemiological Analyses: Example COVID-19 Study

Since the beginning of the COVID-19 pandemic, pharmaceutical treatment hypotheses have abounded, each requiring careful evaluation. A randomized controlled trial generally provides the most credible evaluation of a treatment, but the efficiency and effectiveness of the trial depend on the existing evidence supporting the treatment. The researcher must therefore compile a body of evidence justifying the use of time and resources to further investigate a treatment hypothesis in a trial. An observational study can provide this evidence, but the lack of randomized exposure and the researcher's inability to control treatment administration and data collection introduce significant challenges. A proper analysis of observational health care data thus requires contributions from experts in a diverse set of topics ranging from epidemiology and causal analysis to relevant medical specialties and data sources. Here we summarize these contributions as 10 rules that serve as an end-to-end introduction to retrospective pharmacoepidemiological analyses of observational health care data using a running example of a hypothetical COVID-19 study. A detailed supplement presents a practical how-to guide for following each rule. When carefully designed and properly executed, a retrospective pharmacoepidemiological analysis framed around these rules will inform the decisions of whether and how to investigate a treatment hypothesis in a randomized controlled trial. This work has important implications for any future pandemic by prescribing what we can and should do while the world waits for global vaccine distribution.

Family Business Internationalization in Paradox: Effects of Socioemotional Wealth and Entrepreneurial Spirit

This study investigates the internationalization (i. e., foreign investment) of small family businesses by classifying the effects of external socioemotional wealth (family reputation) vs. internal socioemotional wealth (family involvement). The study involved 2,704 small family businesses in China, and the results support the hypothesis that family reputation has a positive effect on internationalization, while family involvement has a negative effect on internationalization. Moreover, entrepreneurial spirit reinforces the positive effect of family reputation on internationalization and enhances the negative relationship between family involvement and internationalization. This study contributes by examining the effect of entrepreneurial spirit as a potential balancing factor for the paradoxical influence of internal vs. external socioemotional wealth.

The uniform general signed rank test and its design sensitivity

A sensitivity analysis in an observational study tests whether the qualitative conclusions of an analysis would change if we were to allow for the possibility of limited bias due to confounding. The design sensitivity of a hypothesis test quantifies the asymptotic performance of the test in a sensitivity analysis against a particular alternative. We propose a new, nonasymptotic, distribution-free test, the uniform general signed rank test, for observational studies with paired data, and examine its performance under Rosenbaum’s sensitivity analysis model. Our test can be viewed as adaptively choosing from among a large underlying family of signed rank tests, and we show that the uniform test achieves design sensitivity equal to the maximum design sensitivity over the underlying family of signed rank tests. Our test thus achieves superior design sensitivity, indicating it will perform well in sensitivity analyses on large samples. We support this conclusion with simulations and a data example, showing that the advantages of our test extend to moderate sample sizes as well.

A conditional test with demonstrated insensitivity to unmeasured bias in matched observational studies

In an observational study matched for observed covariates, an association between treatment received and outcome exhibited may indicate not an effect caused by the treatment, but merely some bias in the allocation of treatments to individuals within matched pairs. The evidence that distinguishes moderate biases from causal effects is unevenly dispersed among possible comparisons in an observational study: some comparisons are insensitive to larger biases than others. Intuitively, larger treatment effects tend to be insensitive to larger unmeasured biases, and perhaps matched pairs can be grouped using covariates, doses or response patterns so that groups of pairs with larger treatment effects may be identified. Even if an investigator has a reasoned conjecture about where to look for insensitive comparisons, that conjecture might prove mistaken, or, when not mistaken, it might be received sceptically by other scientists who doubt the conjecture or judge it to be too convenient in light of its success with the data at hand. In this article a test is proposed that searches for insensitive findings over many comparisons, but controls the probability of falsely rejecting a true null hypothesis of no treatment effect in the presence of a bias of specified magnitude. An example is studied in which the test considers many comparisons and locates an interpretable comparison that is insensitive to larger biases than a conventional comparison based on Wilcoxon’s signed rank statistic applied to all pairs. A simulation examines the power of the proposed test. The method is implemented in the R package dstat, which contains the example and reproduces the analysis.

An exact test with high power and robustness to unmeasured confounding effects

In observational studies, it is agreed that the sensitivity of the findings to unmeasured confounders needs to be assessed. The issue is that a poor choice of test statistic can result in overstated sensitivity to hidden bias of this kind. In this article, a new adaptive test is proposed, guided by considerations of low sensitivity to hidden bias: it is tailored so that its power is greater than other leading tests, both in finite and infinite samples. One way of defining power in case of possible confounders is as the probability of reporting robustness (ie, insensitivity) of a true discovery to potential bias. In case of finite samples, we compute the power by simulations. When sample size approaches infinity, a meaningful indicator of the power is the design sensitivity, which is computed analytically and found to be better in the new test than in existing tests. Another asymptotic criterion for comparing tests when there is concern for confounders is Bahadur efficiency. The proposed test outperforms commonly used tests in terms of Bahadur efficiency in most sampling situations. The advantages of the new test mainly stem from its adaptivity: it combines two test statistics and consequently achieves the best design sensitivity and the best Bahadur efficiency of the two. As a "real-world" examination, we compare 441 daily smokers to 441 nonsmokers, to test the effect of smoking on periodontal disease. The new test is more robust to unmeasured confounders than both the Wilcoxon signed rank test and the paired t-test.

The Unequal Consequences of Mass Incarceration for Children

A growing literature has documented the mostly deleterious intergenerational consequences of paternal incarceration, but less research has considered heterogeneity in these relationships. In this article, I use data from the Fragile Families and Child Wellbeing Study (N = 3,065) to estimate the heterogeneous relationship between paternal incarceration and children's problem behaviors (internalizing behaviors, externalizing behaviors, and early juvenile delinquency) and cognitive skills (reading comprehension, math comprehension, and verbal ability) in middle childhood. Taking into account children's risk of experiencing paternal incarceration, measured by the social contexts in which children are embedded (e.g., father's residential status, poverty, neighborhood disadvantage) reveals that the consequences-across all outcomes except early juvenile delinquency-are more deleterious for children with relatively low risks of exposure to paternal incarceration than for children with relatively high risks of exposure to paternal incarceration. These findings suggest that the intergenerational consequences of paternal incarceration are more complicated than documented in previous research and, more generally, suggest that research on family inequality consider both differential selection into treatments and differential responses to treatments.

Evaluation of a Serious and Violent Offender Reentry Initiative (SVORI) Program in a Midwest State

The Serious and Violent Offender Reentry Initiative (SVORI) paved the way for a new era of rehabilitation in corrections’ programming. However, published outcome evaluations of SVORI programs and their progeny are limited in number. The current article presents the multiyear outcome evaluation of one prisoner reentry initiative established in a Midwestern state, which was developed within the framework of the SVORI program model. A comparison group was identified using propensity score matching to evaluate program effectiveness on the recidivism outcomes of returns to prison and new convictions. Cox proportional hazards modeling found program participants to have significantly lower hazard to incur a new conviction than the comparison group but no difference in the hazard for reincarceration. The implications of these mixed findings in recidivism outcomes are discussed for the reentry program initiative.

Integrating impact evaluation in the design and implementation of monitoring marine protected areas

Quasi-experimental impact evaluation approaches, which enable scholars to disentangle effects of conservation interventions from broader changes in the environment, are gaining momentum in the conservation sector. However, rigorous impact evaluation using statistical matching techniques to estimate the counterfactual have yet to be applied to marine protected areas (MPAs). While there are numerous studies investigating 'impacts' of MPAs that have generated considerable insights, results are variable. This variation has been linked to the biophysical and social context in which they are established, as well as attributes of management and governance. To inform decisions about MPA placement, design and implementation, we need to expand our understanding of conditions under which MPAs are likely to lead to positive outcomes by embracing advances in impact evaluation methodologies. Here, we describe the integration of impact evaluation within an MPA network monitoring programme in the Bird's Head Seascape, Indonesia. Specifically we (i) highlight the challenges of implementation 'on the ground' and in marine ecosystems and (ii) describe the transformation of an existing monitoring programme into a design appropriate for impact evaluation. This study offers one potential model for mainstreaming impact evaluation in the conservation sector.

A flexible, interpretable framework for assessing sensitivity to unmeasured confounding

When estimating causal effects, unmeasured confounding and model misspecification are both potential sources of bias. We propose a method to simultaneously address both issues in the form of a semi-parametric sensitivity analysis. In particular, our approach incorporates Bayesian Additive Regression Trees into a two-parameter sensitivity analysis strategy that assesses sensitivity of posterior distributions of treatment effects to choices of sensitivity parameters. This results in an easily interpretable framework for testing for the impact of an unmeasured confounder that also limits the number of modeling assumptions. We evaluate our approach in a large-scale simulation setting and with high blood pressure data taken from the Third National Health and Nutrition Examination Survey. The model is implemented as open-source software, integrated into the treatSens package for the R statistical programming language. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Intraoperative awareness risk, anesthetic sensitivity, and anesthetic management for patients with natural red hair: a matched cohort study

PURPOSE

The red-hair phenotype, which is often produced by mutations in the melanocortin-1 receptor gene, has been associated with an increase in sedative, anesthetic, and analgesic requirements in both animal and human studies. Nevertheless, the clinical implications of this phenomenon in red-haired patients undergoing surgery are currently unknown.

METHODS

In a secondary analysis of a prospective trial of intraoperative awareness, red-haired patients were identified and matched with five control patients, and the relative risk for intraoperative awareness was determined. Overall anesthetic management between groups was compared using Hotelling's T(2) statistic. Inhaled anesthetic requirements were compared between cohorts by evaluating the relationship between end-tidal anesthetic concentration and the bispectral index with a linear mixed-effects model. Time to recovery was compared using Kaplan-Meier analysis, and differences in postoperative pain and nausea/vomiting were evaluated with Chi square tests.

RESULTS

A cohort of 319 red-haired patients was matched with 1,595 control patients for a sample size of 1,914. There were no significant differences in the relative risk of intraoperative awareness (relative risk = 1.67; 95% confidence interval 0.34 to 8.22), anesthetic management, recovery times, or postoperative pain between red-haired patients and control patients. The relationship between pharmacokinetically stable volatile anesthetic concentrations and bispectral index values differed significantly between red-haired patients and controls (P < 0.001), but without clinical implications.

CONCLUSION

There were no demonstrable differences between red-haired patients and controls in response to anesthetic and analgesic agents or in recovery parameters. These findings suggest that perioperative anesthetic and analgesic management should not be altered based on self-reported red-hair phenotype.

Nonparametric Bounds and Sensitivity Analysis of Treatment Effects

This paper considers conducting inference about the effect of a treatment (or exposure) on an outcome of interest. In the ideal setting where treatment is assigned randomly, under certain assumptions the treatment effect is identifiable from the observable data and inference is straightforward. However, in other settings such as observational studies or randomized trials with noncompliance, the treatment effect is no longer identifiable without relying on untestable assumptions. Nonetheless, the observable data often do provide some information about the effect of treatment, that is, the parameter of interest is partially identifiable. Two approaches are often employed in this setting: (i) bounds are derived for the treatment effect under minimal assumptions, or (ii) additional untestable assumptions are invoked that render the treatment effect identifiable and then sensitivity analysis is conducted to assess how inference about the treatment effect changes as the untestable assumptions are varied. Approaches (i) and (ii) are considered in various settings, including assessing principal strata effects, direct and indirect effects and effects of time-varying exposures. Methods for drawing formal inference about partially identified parameters are also discussed.

Parental monitoring trajectories and gambling among a longitudinal cohort of urban youth

AIM

To test the strength of the association between parental monitoring trajectories throughout early adolescence (ages 11-14) and gambling behaviours by young adulthood (age 22).

DESIGN

Longitudinal cohort design.

SETTING

Baltimore, Maryland.

PARTICIPANTS

The sample of 514 participants with gambling data between ages 16-22 and parental monitoring data between ages 11-14 were predominantly African American and received subsidized lunches at age 6.

MEASUREMENTS

The South Oaks Gambling Screen and South Oaks Gambling Screen-Revised for Adolescents collected self-reports on annual gambling and gambling problems between ages 16-22. The Parental Monitoring Subscale of the Structured Interview of Parent Management Skills and Practices-Youth Version collected self-reports on annual parental monitoring between ages 11-14.

FINDINGS

General growth mixture modelling identified two parental monitoring trajectories: (i) 'stable' class (84.9%) began with a high level of parental monitoring at age 11 that remained steady to age 14; (ii) 'declining' class (15.1%) began with a significantly lower level of parental monitoring at age 11 and experienced a significant to through age 14. The declining class had increased significantly unadjusted (OR = 1.91; 95% CI = 1.59, 2.23; P ≤ 0.001) and adjusted (aOR = 1.57; 95% CI = 1.24, 1.99; P = 0.01) odds of problem gambling compared with non-gambling.

CONCLUSION

Low and/or declining parental monitoring of children between the ages of 11 and 14 is associated significantly with problem gambling when those children reach young adulthood.

Revisiting issues, drawbacks and opportunities with observational studies in comparative effectiveness research

RATIONALE

Despite their inherently pervasive limitations, data from observational studies are increasingly relied upon by health care decision makers to fill critical information gaps created by lack of evidence from randomized controlled trials.

AIM AND OBJECTIVE

The aim and objective of this article was to revisit the major issues associated with observational studies from secondary data sources.

METHOD

The method of this article was canvass of the literature.

RESULTS

Sources of bias are highlighted and steps intended to minimize bias are summarized.

CONCLUSION

Efforts should be made to improve causal inference of treatment effects from observational studies found in secondary data sources. Extra care and caution should be exercised in the interpretation and reporting of results from these studies.

Impact of multiple matched controls on design sensitivity in observational studies

In an observational study, one treated subject may be matched for observed covariates to either one or several untreated controls. The common motivation for using several controls rather than one is to increase the power of a test of no effect under the doubtful assumption that matching for observed covariates suffices to remove bias from nonrandom treatment assignment. Does the choice between one or several matched controls affect the sensitivity of conclusions to violations of this doubtful assumption? With continuous responses, it is known that reducing the heterogeneity of matched pair differences reduces sensitivity to unmeasured biases, but increasing the sample size has a highly circumscribed effect on sensitivity to bias. Is the use of several controls rather than one analogous to a reduction in heterogeneity or to an increase in sample size? The issue is examined for Huber's m-statistics, including the t-test, the examination having three components: an example, asymptotic calculations using design sensitivity, and a simulation. Use of multiple controls with continuous responses yields a nontrivial reduction in sensitivity to unmeasured biases. An example looks at lead and cadmium in the blood of smokers from the 2008 National Health and Nutrition Examination Survey. A by-product of the discussion is a new result giving the design sensitivity for the permutation distribution of m-statistics.

A new u-statistic with superior design sensitivity in matched observational studies

In an observational or nonrandomized study of treatment effects, a sensitivity analysis indicates the magnitude of bias from unmeasured covariates that would need to be present to alter the conclusions of a naïve analysis that presumes adjustments for observed covariates suffice to remove all bias. The power of sensitivity analysis is the probability that it will reject a false hypothesis about treatment effects allowing for a departure from random assignment of a specified magnitude; in particular, if this specified magnitude is "no departure" then this is the same as the power of a randomization test in a randomized experiment. A new family of u-statistics is proposed that includes Wilcoxon's signed rank statistic but also includes other statistics with substantially higher power when a sensitivity analysis is performed in an observational study. Wilcoxon's statistic has high power to detect small effects in large randomized experiments-that is, it often has good Pitman efficiency-but small effects are invariably sensitive to small unobserved biases. Members of this family of u-statistics that emphasize medium to large effects can have substantially higher power in a sensitivity analysis. For example, in one situation with 250 pair differences that are Normal with expectation 1/2 and variance 1, the power of a sensitivity analysis that uses Wilcoxon's statistic is 0.08 while the power of another member of the family of u-statistics is 0.66. The topic is examined by performing a sensitivity analysis in three observational studies, using an asymptotic measure called the design sensitivity, and by simulating power in finite samples. The three examples are drawn from epidemiology, clinical medicine, and genetic toxicology.