Using Mixed Integer Programming for Matching in an Observational Study of Kidney Failure After Surgery

Optimal refinement of strata to balance covariates

What is the best way to split one stratum into two to maximally reduce the within-stratum imbalance in many covariates? We formulate this as an integer program and approximate the solution by randomized rounding of a linear program. A linear program may assign a fraction of a person to each refined stratum. Randomized rounding views fractional people as probabilities, assigning intact people to strata using biased coins. Randomized rounding is a well-studied theoretical technique for approximating the optimal solution of certain insoluble integer programs. When the number of people in a stratum is large relative to the number of covariates, we prove the following new results: (i) randomized rounding to split a stratum does very little randomizing, so it closely resembles the linear programming relaxation without splitting intact people; (ii) the linear relaxation and the randomly rounded solution place lower and upper bounds on the unattainable integer programming solution; and because of (i), these bounds are often close, thereby ratifying the usable randomly rounded solution. We illustrate using an observational study that balanced many covariates by forming matched pairs composed of 2016 patients selected from 5735 using a propensity score. Instead, we form 5 propensity score strata and refine them into 10 strata, obtaining excellent covariate balance while retaining all patients. An R package optrefine at CRAN implements the method. Supplementary materials are available online.

An evolutionary algorithm for the direct optimization of covariate balance between nonrandomized populations

Matching reduces confounding bias in comparing the outcomes of nonrandomized patient populations by removing systematic differences between them. Under very basic assumptions, propensity score (PS) matching can be shown to eliminate bias entirely in estimating the average treatment effect on the treated. In practice, misspecification of the PS model leads to deviations from theory and matching quality is ultimately judged by the observed post-matching balance in baseline covariates. Since covariate balance is the ultimate arbiter of successful matching, we argue for an approach to matching in which the success criterion is explicitly specified and describe an evolutionary algorithm to directly optimize an arbitrary metric of covariate balance. We demonstrate the performance of the proposed method using a simulated dataset of 275,000 patients and 10 matching covariates. We further apply the method to match 250 patients from a recently completed clinical trial to a pool of more than 160,000 patients identified from electronic health records on 101 covariates. In all cases, we find that the proposed method outperforms PS matching as measured by the specified balance criterion. We additionally find that the evolutionary approach can perform comparably to another popular direct optimization technique based on linear integer programming, while having the additional advantage of supporting arbitrary balance metrics. We demonstrate how the chosen balance metric impacts the statistical properties of the resulting matched populations, emphasizing the potential impact of using nonlinear balance functions in constructing an external control arm. We release our implementation of the considered algorithms in Python.

Efficient algorithms for building representative matched pairs with enhanced generalizability

Many recent efforts center on assessing the ability of real-world evidence (RWE) generated from non-randomized, observational data to produce results compatible with those from randomized controlled trials (RCTs). One noticeable endeavor is the RCT DUPLICATE initiative. To better reconcile findings from an observational study and an RCT, or two observational studies based on different databases, it is desirable to eliminate differences between study populations. We outline an efficient, network-flow-based statistical matching algorithm that designs well-matched pairs from observational data that resemble the covariate distributions of a target population, for instance, the target-RCT-eligible population in the RCT DUPLICATE initiative studies or a generic population of scientific interest. We demonstrate the usefulness of the method by revisiting the inconsistency regarding a cardioprotective effect of the hormone replacement therapy (HRT) in the Women's Health Initiative (WHI) clinical trial and corresponding observational study. We found that the discrepancy between the trial and observational study persisted in a design that adjusted for the difference in study populations' cardiovascular risk profile, but seemed to disappear in a study design that further adjusted for the difference in HRT initiation age and previous estrogen-plus-progestin use. The proposed method is integrated into the R package match2C.

Firearm Injuries In Children And Adolescents: Health And Economic Consequences Among Survivors And Family Members

More US children and adolescents today die from firearms than any other cause, and many more sustain firearm injuries and survive. The clinical and economic impact of these firearm injuries on survivors and family members remains poorly understood. Using 2007-21 commercial health insurance claims data, we studied 2,052 child and adolescent survivors compared to 9,983 matched controls who did not incur firearm injuries, along with 6,209 family members of survivors compared to 29,877 matched controls, and 265 family members of decedents compared to 1,263 matched controls. Through one year after firearm injury, child and adolescent survivors experienced a 117 percent increase in pain disorders, a 68 percent increase in psychiatric disorders, and a 144 percent increase in substance use disorders relative to the controls. Survivors' health care spending increased by an average of $34,884-a 17.1-fold increase-with 95 percent paid by insurers or employers. Parents of survivors experienced a 30-31 percent increase in psychiatric disorders, with 75 percent more mental health visits by mothers, and 5-14 percent reductions in mothers' and siblings' routine medical care. Family members of decedents experienced substantially larger 2.3- to 5.3-fold increases in psychiatric disorders, with at least 15.3-fold more mental health visits among parents. Firearm injuries in youth have notable health implications for the whole family, along with large effects on societal spending.

How well can fine balance work for covariate balancing

Fine balance is a matching technique to improve covariate balance in observational studies. It constrains a match to have identical distributions for some covariates without restricting who is matched to whom. However, despite its wide application and excellent performance in practice, there is very little theory indicating when the method is likely to succeed or fail and to what extent it can remove covariate imbalance. In order to answer these questions, this paper studies the limits of what is possible for covariate balancing using fine balance and near-fine balance. The investigations suggest that given the distributions of the treated and control groups, in large samples, the maximum achievable balance by using fine balance only depends on the matching ratio (ie, the ratio of the sample size of the control group to that of the treated group). In addition, the results indicate how to estimate this matching ratio threshold without knowledge of the true distributions in finite samples. The findings are also illustrated by numerical studies in this paper.

Sensitivity analyses informed by tests for bias in observational studies

In an observational study, the treatment received and the outcome exhibited may be associated in the absence of an effect caused by the treatment, even after controlling for observed covariates. Two tactics are common: (i) a test for unmeasured bias may be obtained using a secondary outcome for which the effect is known and (ii) a sensitivity analysis may explore the magnitude of unmeasured bias that would need to be present to explain the observed association as something other than an effect caused by the treatment. Can such a test for unmeasured bias inform the sensitivity analysis? If the test for bias does not discover evidence of unmeasured bias, then ask: Are conclusions therefore insensitive to larger unmeasured biases? Conversely, if the test for bias does find evidence of bias, then ask: What does that imply about sensitivity to biases? This problem is formulated in a new way as a convex quadratically constrained quadratic program and solved on a large scale using interior point methods by a modern solver. That is, a convex quadratic function of N variables is minimized subject to constraints on linear and convex quadratic functions of these variables. The quadratic function that is minimized is a statistic for the primary outcome that is a function of the unknown treatment assignment probabilities. The quadratic function that constrains this minimization is a statistic for subsidiary outcome that is also a function of these same unknown treatment assignment probabilities. In effect, the first statistic is minimized over a confidence set for the unknown treatment assignment probabilities supplied by the unaffected outcome. This process avoids the mistake of interpreting the failure to reject a hypothesis as support for the truth of that hypothesis. The method is illustrated by a study of the effects of light daily alcohol consumption on high-density lipoprotein (HDL) cholesterol levels. In this study, the method quickly optimizes a nonlinear function of N = 800 $N=800$ variables subject to linear and quadratic constraints. In the example, strong evidence of unmeasured bias is found using the subsidiary outcome, but, perhaps surprisingly, this finding makes the primary comparison insensitive to larger biases.

Matching on Generalized Propensity Scores with Continuous Exposures

In the context of a binary treatment, matching is a well-established approach in causal inference. However, in the context of a continuous treatment or exposure, matching is still underdeveloped. We propose an innovative matching approach to estimate an average causal exposure-response function under the setting of continuous exposures that relies on the generalized propensity score (GPS). Our approach maintains the following attractive features of matching: a) clear separation between the design and the analysis; b) robustness to model misspecification or to the presence of extreme values of the estimated GPS; c) straightforward assessments of covariate balance. We first introduce an assumption of identifiability, called local weak unconfoundedness. Under this assumption and mild smoothness conditions, we provide theoretical guarantees that our proposed matching estimator attains point-wise consistency and asymptotic normality. In simulations, our proposed matching approach outperforms existing methods under settings with model misspecification or in the presence of extreme values of the estimated GPS. We apply our proposed method to estimate the average causal exposure-response function between long-term PM2.5 exposure and all-cause mortality among 68.5 million Medicare enrollees, 2000-2016. We found strong evidence of a harmful effect of long-term PM2.5 exposure on mortality. Code for the proposed matching approach is provided in the CausalGPS R package, which is available on CRAN and provides a computationally efficient implementation.

Testing Biased Randomization Assumptions and Quantifying Imperfect Matching and Residual Confounding in Matched Observational Studies

One central goal of design of observational studies is to embed non-experimental data into an approximate randomized controlled trial using statistical matching. Despite empirical researchers' best intention and effort to create high-quality matched samples, residual imbalance due to observed covariates not being well matched often persists. Although statistical tests have been developed to test the randomization assumption and its implications, few provide a means to quantify the level of residual confounding due to observed covariates not being well matched in matched samples. In this article, we develop two generic classes of exact statistical tests for a biased randomization assumption. One important by-product of our testing framework is a quantity called residual sensitivity value (RSV), which provides a means to quantify the level of residual confounding due to imperfect matching of observed covariates in a matched sample. We advocate taking into account RSV in the downstream primary analysis. The proposed methodology is illustrated by re-examining a famous observational study concerning the effect of right heart catheterization (RHC) in the initial care of critically ill patients. Code implementing the method can be found in the supplementary materials.

Survival outcomes following pregnancy or assisted reproductive technologies after breast cancer: A population-based study

BACKGROUND

This study sought to determine the impact of pregnancy or assisted reproductive technologies (ART) on breast-cancer-specific survival among breast cancer survivors.

METHODS

The authors performed a cohort study using a novel data linkage from the California Cancer Registry, the California birth cohort, and the Society for Assisted Reproductive Technology Clinic Outcome Reporting System data sets. They performed risk-set matching in women with stages I-III breast cancer diagnosed between 2000 and 2012. For each pregnant woman, comparable women who were not pregnant at that point but were otherwise similar based on observed characteristics were matched at the time of pregnancy. After matching, Cox proportional hazards models were used to estimate hazard ratios (HRs) and 95% confidence intervals (CIs) for the association of pregnancy with breast-cancer-specific survival. We repeated these analyses for women who received ART.

RESULTS

Among 30,021 women with breast cancer, 553 had a pregnancy and 189 attempted at least one cycle of ART. In Cox proportional hazards modeling, the pregnancy group had a higher 5-year disease-specific survival rate; 95.6% in the pregnancy group and 90.6% in the nonpregnant group (HR, 0.43; 95% CI, 0.24-0.77). In women with hormone receptor-positive cancer, we found similar results (HR, 0.43; 95% CI, 0.2-0.91). In the ART analysis, there was no difference in survival between groups; the 5-year disease-specific survival rate was 96.9% in the ART group and 94.1% in the non-ART group (HR, 0.44; 95% CI, 0.17-1.13).

CONCLUSION

Pregnancy and ART are not associated with worse survival in women with breast cancer.

LAY SUMMARY

We sought to determine the impact of pregnancy or assisted reproductive technologies (ART) among breast cancer survivors. We performed a study of 30,021 women by linking available data from California and the Society for Assisted Reproductive Technology Clinic Outcome Reporting System. For each pregnant woman, we matched at the time of pregnancy comparable women who were not pregnant at that point but were otherwise similar based on observed characteristics. We repeated these analyses for women who received ART. We found that pregnancy and ART were not associated with worse survival.

Indirect Treatment Comparison of Daratumumab, Pomalidomide, and Dexamethasone Versus Standard of Care in Patients with Difficult-to-Treat Relapsed/Refractory Multiple Myeloma

INTRODUCTION

The phase 3 APOLLO study demonstrated significantly better progression-free survival (PFS) and clinical responses with daratumumab, pomalidomide, and dexamethasone (D-Pd) versus pomalidomide and dexamethasone (Pd) in patients with relapsed/refractory multiple myeloma (RRMM). On the basis of these results and those from the phase 1b EQUULEUS trial, D-Pd was approved in this patient population. In the absence of head-to-head data comparing D-Pd with further standard of care (SOC) therapies, indirect treatment comparisons (ITCs) can provide important information to help optimize treatment selection. The objective of this study was to indirectly compare PFS improvement with D-Pd versus daratumumab, bortezomib, and dexamethasone (D-Vd) and D-Pd versus bortezomib and dexamethasone (Vd) in patients with RRMM.

METHODS

Patient-level data were from APOLLO, EQUULEUS, and CASTOR. Three methods of adjusting imbalances in baseline characteristics including stabilized inverse probability of treatment weighting (sIPTW), cardinality matching (CM), and propensity score matching (PSM) were initially considered. CM offers mathematically guaranteed largest matched sample meeting pre-specified maximum standardized mean difference criteria for matching covariates. sIPTW and PSM were based on propensity scores derived from logistic regression. Feasibility assessment of the PSM method returned too low effective sample size to support a meaningful comparison. CM was chosen as the base case and sIPTW as a sensitivity analysis.

RESULTS

After harmonized eligibility criteria were applied, 253, 104, and 122 patients from the D-Pd, D-Vd, and Vd cohorts, respectively, were included in the ITC analyses. Some imbalances in baseline characteristics were identified between D-Pd and D-Vd/Vd cohorts that remained after adjustment. PFS hazard ratios showed significant improvement for D-Pd over D-Vd and Vd for CM and sIPTW analyses.

CONCLUSIONS

Results showed consistent PFS benefit for D-Pd versus D-Vd and Vd regardless of the adjustment technique used. These findings support the use of D-Pd versus D-Vd or Vd in patients with difficult-to-treat RRMM.

TRIAL REGISTRATION

NCT03180736; NCT02136134, NCT01998971.

Profile Matching for the Generalization and Personalization of Causal Inferences

We introduce profile matching, a multivariate matching method for randomized experiments and observational studies that finds the largest possible unweighted samples across multiple treatment groups that are balanced relative to a covariate profile. This covariate profile can represent a specific population or a target individual, facilitating the generalization and personalization of causal inferences. For generalization, because the profile often amounts to summary statistics for a target population, profile matching does not always require accessing individual-level data, which may be unavailable for confidentiality reasons. For personalization, the profile comprises the characteristics of a single individual. Profile matching achieves covariate balance by construction, but unlike existing approaches to matching, it does not require specifying a matching ratio, as this is implicitly optimized for the data. The method can also be used for the selection of units for study follow-up, and it readily applies to multivalued treatments with many treatment categories. We evaluate the performance of profile matching in a simulation study of the generalization of a randomized trial to a target population. We further illustrate this method in an exploratory observational study of the relationship between opioid use and mental health outcomes. We analyze these relationships for three covariate profiles representing: (i) sexual minorities, (ii) the Appalachian United States, and (iii) the characteristics of a hypothetical vulnerable patient. The method can be implemented via the new function profmatch in the designmatch package for R, for which we provide a step-by-step tutorial.

Changes in Health Care Spending, Use, and Clinical Outcomes After Nonfatal Firearm Injuries Among Survivors and Family Members : A Cohort Study

BACKGROUND

Despite increasing awareness of firearm-related deaths, evidence on the clinical and economic implications of nonfatal firearm injuries is limited.

OBJECTIVE

To measure changes in clinical and economic outcomes after nonfatal firearm injuries among survivors and their family members.

DESIGN

Cohort study.

SETTING

MarketScan Medicare and commercial claims data, 2008 to 2018.

PARTICIPANTS

6498 survivors of firearm injuries matched to 32 490 control participants and 12 489 family members of survivors matched to 62 445 control participants.

INTERVENTION

Exposure to nonfatal firearm injury as a survivor or family member of a survivor.

MEASUREMENTS

Changes in health care spending, use, and morbidity from preinjury through 1 year postinjury relative to control participants, on average and by type and severity of firearm injury.

RESULTS

After nonfatal firearm injury, medical spending increased $2495 per person per month (402%) and cost sharing increased $102 per person per month (176%) among survivors relative to control participants (P < 0.001) in the first year after injury, driven by an increase in the first month of $25 554 (4122%) in spending and $1112 (1917%) in cost sharing per survivor (P < 0.001). All categories of health care use increased relative to the control group. Survivors had a 40% increase in pain diagnoses, a 51% increase in psychiatric disorders, and an 85% increase in substance use disorders after firearm injury relative to control participants (P < 0.001), accompanied by increased pain and psychiatric medications. Family members had a 12% increase in psychiatric disorders relative to their control participants (P = 0.003). These overall clinical and economic changes were driven by intentional firearm injuries and more severe firearm injuries.

LIMITATION

Precision of diagnostic codes and generalizability to other patient populations, including Medicaid and uninsured patients.

CONCLUSION

In survivors, nonfatal firearm injuries led to increases in psychiatric disorders, substance use disorders, and pain diagnoses, alongside substantial increases in health care spending and use. In addition, mental health worsened among family members.

PRIMARY FUNDING SOURCE

National Institutes of Health.

Association Between Telepsychiatry Capability and Treatment of Patients With Mental Illness in the Emergency Department

OBJECTIVE

Because of limited access to psychiatrists, patients with acute mental illness in some emergency departments (EDs) may wait days for a consultation in the ED or as a medical-surgical admission. The study assessed whether telepsychiatry improves access to care and decreases ED wait times and hospital admissions.

METHODS

ED visits with a primary diagnosis of mental illness were identified from 2010-2018 Medicare claims. A total of 134 EDs across 22 states that implemented telepsychiatry between 2013 and 2016 were matched 1:1 with control EDs without telepsychiatry on several characteristics, including availability of in-person psychiatrist consultations. Outcomes included patients' likelihood of admission to a medical-surgical or psychiatric bed, mental illness spending, prolonged ED length of stay (LOS) (two or more midnights in the ED), 90-day mortality, and outpatient follow-up care. Using a difference-in-difference design, changes in outcomes between the 3 years before telepsychiatry adoption and the 2 years after were examined.

RESULTS

There were 172,708 ED mental illness visits across the 134 matched ED pairs in the study period. Telepsychiatry adoption was associated with increased admissions to a psychiatric bed (differential increase, 4.3 percentage points; p<0.001), decreased admissions to a medical-surgical bed (differential decrease, 2.0 percentage points; p<0.001), increased likelihood of a prolonged ED LOS (differential increase, 3.0 percentage points; p<0.001), and increased mental illness spending (differential increase, $292; p<0.01).

CONCLUSIONS

Telepsychiatry adoption was associated with a lower likelihood of admission to a medical-surgical bed but an increased likelihood of admission to a psychiatric bed and a prolonged ED LOS.

Matching One Sample According to Two Criteria in Observational Studies

Multivariate matching has two goals: (i) to construct treated and control groups that have similar distributions of observed covariates, and (ii) to produce matched pairs or sets that are homogeneous in a few key covariates. When there are only a few binary covariates, both goals may be achieved by matching exactly for these few covariates. Commonly, however, there are many covariates, so goals (i) and (ii) come apart, and must be achieved by different means. As is also true in a randomized experiment, similar distributions can be achieved for a high-dimensional covariate, but close pairs can be achieved for only a few covariates. We introduce a new polynomial-time method for achieving both goals that substantially generalizes several existing methods; in particular, it can minimize the earthmover distance between two marginal distributions. The method involves minimum cost flow optimization in a network built around a tripartite graph, unlike the usual network built around a bipartite graph. In the tripartite graph, treated subjects appear twice, on the far left and the far right, with controls sandwiched between them, and efforts to balance covariates are represented on the right, while efforts to find close individual pairs are represented on the left. In this way, the two efforts may be pursued simultaneously without conflict. The method is applied to our on-going study in the Medicare population of the relationship between superior nursing and sepsis mortality. The match2C package in R implements the method.

Scientific prizes and the extraordinary growth of scientific topics

Fast growing scientific topics have famously been key harbingers of the new frontiers of science, yet, large-scale analyses of their genesis and impact are rare. We investigated one possible factor connected with a topic’s extraordinary growth: scientific prizes. Our longitudinal analysis of nearly all recognized prizes worldwide and over 11,000 scientific topics from 19 disciplines indicates that topics associated with a scientific prize experience extraordinary growth in productivity, impact, and new entrants. Relative to matched non-prizewinning topics, prizewinning topics produce 40% more papers and 33% more citations, retain 55% more scientists, and gain 37 and 47% more new entrants and star scientists, respectively, in the first five-to-ten years after the prize. Funding do not account for a prizewinning topic’s growth. Rather, growth is positively related to the degree to which the prize is discipline-specific, conferred for recent research, or has prize money. These findings reveal new dynamics behind scientific innovation and investment.

Scientific revolutions have famously inspired scientists and innovation but large-scale analyses of scientific revolutions in modern science are rare. Here, the authors investigate one possible factor connected with a topic’s extraordinary growth—scientific prizes.

Relationship between changing malaria burden and low birth weight in sub-Saharan Africa: A difference-in-differences study via a pair-of-pairs approach

Background

According to the World Health Organization (WHO), in 2018, an estimated 228 million malaria cases occurred worldwide with most cases occurring in sub-Saharan Africa. Scale-up of vector control tools coupled with increased access to diagnosis and effective treatment has resulted in a large decline in malaria prevalence in some areas, but other areas have seen little change. Although interventional studies demonstrate that preventing malaria during pregnancy can reduce the rate of low birth weight (i.e. child's birth weight <2500 g), it remains unknown whether natural changes in parasite transmission and malaria burden can improve birth outcomes.

Methods

We conducted an observational study of the effect of changing malaria burden on low birth weight using data from 18,112 births in 19 countries in sub-Saharan African countries during the years 2000-2015. Specifically, we conducted a difference-in-differences study via a pair-of-pairs matching approach using the fact that some sub-Saharan areas experienced sharp drops in malaria prevalence and some experienced little change.

Results

A malaria prevalence decline from a high rate (Plasmodium falciparum parasite rate in children aged 2-up-to-10 (i.e. PfPR2-10) > 0.4) to a low rate (PfPR2-10 < 0.2) is estimated to reduce the rate of low birth weight by 1.48 percentage points (95% confidence interval: 3.70 percentage points reduction, 0.74 percentage points increase), which is a 17% reduction in the low birth weight rate compared to the average (8.6%) in our study population with observed birth weight records (1.48/8.6 ≈ 17%). When focusing on first pregnancies, a decline in malaria prevalence from high to low is estimated to have a greater impact on the low birth weight rate than for all births: 3.73 percentage points (95% confidence interval: 9.11 percentage points reduction, 1.64 percentage points increase).

Conclusions

Although the confidence intervals cannot rule out the possibility of no effect at the 95% confidence level, the concurrence between our primary analysis, secondary analyses, and sensitivity analyses, and the magnitude of the effect size, contribute to the weight of the evidence suggesting that declining malaria burden can potentially substantially reduce the low birth weight rate at the community level in sub-Saharan Africa, particularly among firstborns. The novel statistical methodology developed in this article-a pair-of-pairs approach to a difference-in-differences study-could be useful for many settings in which different units are observed at different times.

Funding

Ryan A. Simmons is supported by National Center for Advancing Translational Sciences (UL1TR002553). The funder had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Adversarial balancing-based representation learning for causal effect inference with observational data

Learning causal effects from observational data greatly benefits a variety of domains such as health care, education, and sociology. For instance, one could estimate the impact of a new drug on specific individuals to assist clinical planning and improve the survival rate. In this paper, we focus on studying the problem of estimating the Conditional Average Treatment Effect (CATE) from observational data. The challenges for this problem are two-fold: on the one hand, we have to derive a causal estimator to estimate the causal quantity from observational data, in the presence of confounding bias; on the other hand, we have to deal with the identification of the CATE when the distributions of covariates over the treatment group units and the control units are imbalanced. To overcome these challenges, we propose a neural network framework called Adversarial Balancing-based representation learning for Causal Effect Inference (ABCEI), based on recent advances in representation learning. To ensure the identification of the CATE, ABCEI uses adversarial learning to balance the distributions of covariates in the treatment and the control group in the latent representation space, without any assumptions on the form of the treatment selection/assignment function. In addition, during the representation learning and balancing process, highly predictive information from the original covariate space might be lost. ABCEI can tackle this information loss problem by preserving useful information for predicting causal effects under the regularization of a mutual information estimator. The experimental results show that ABCEI is robust against treatment selection bias, and matches/outperforms the state-of-the-art approaches. Our experiments show promising results on several datasets, encompassing several health care (and other) domains.

Assessment of the Utility of Kidney Histology as a Basis for Discarding Organs in the United States: A Comparison of International Transplant Practices and Outcomes

BACKGROUND

Many kidneys donated for transplant in the United States are discarded because of abnormal histology. Whether histology adds incremental value beyond usual donor attributes in assessing allograft quality is unknown.

METHODS

This population-based study included patients who received a deceased donor kidney that had been biopsied before implantation according to a prespecified protocol in France and Belgium, where preimplantation biopsy findings are generally not used for decision making in the allocation process. We also studied kidneys that had been acquired from deceased United States donors for transplantation that were biopsied during allocation and discarded because of low organ quality. Using donor and recipient characteristics, we fit multivariable Cox models for death-censored graft failure and examined whether predictive accuracy (C index) improved after adding donor histology. We matched the discarded United States kidneys to similar kidneys transplanted in Europe and calculated predicted allograft survival.

RESULTS

In the development cohort of 1629 kidney recipients at two French centers, adding donor histology to the model did not significantly improve prediction of long-term allograft failure. Analyses using an external validation cohort from two Belgian centers confirmed the lack of improved accuracy from adding histology. About 45% of 1103 United States kidneys discarded because of histologic findings could be accurately matched to very similar kidneys that had been transplanted in France; these discarded kidneys would be expected to have allograft survival of 93.1% at 1 year, 80.7% at 5 years, and 68.9% at 10 years.

CONCLUSIONS

In this multicenter study, donor kidney histology assessment during allocation did not provide substantial incremental value in ascertaining organ quality. Many kidneys discarded on the basis of biopsy findings would likely benefit United States patients who are wait listed.

Comparing the Performance of Statistical Adjustment Methods by Recovering the Experimental Benchmark from the REFLUX Trial

Much evidence in comparative effectiveness research is based on observational studies. Researchers who conduct observational studies typically assume that there are no unobservable differences between the treatment groups under comparison. Treatment effectiveness is estimated after adjusting for observed differences between comparison groups. However, estimates of treatment effectiveness may be biased because of misspecification of the statistical model. That is, if the method of treatment effect estimation imposes unduly strong functional form assumptions, treatment effect estimates may be inaccurate, leading to inappropriate recommendations about treatment decisions. We compare the performance of a wide variety of treatment effect estimation methods for the average treatment effect. We do so within the context of the REFLUX study from the United Kingdom. In REFLUX, participants were enrolled in either an randomized controlled trial (RCT) or an observational study arm. In the RCT, patients were randomly assigned to either surgery or medical management. In the patient preference arm, participants selected to either have surgery or medical management. We attempt to recover the treatment effect estimate from the RCT using the data from the patient preference arms of the study. We vary the method of treatment effect estimation and record which methods are successful and which are not. We apply more than 20 different methods, including standard regression models as well as advanced machine learning methods. We find that simple propensity score matching methods provide the least accurate estimates versus the RCT benchmark. We find variation in performance across the other methods, with some, but not all recovering the experimental benchmark. We conclude that future studies should use multiple methods of estimation to fully represent uncertainty according to the choice of estimation approach.

Evaluating the Costs and Outcomes of Hospital Nursing Resources: a Matched Cohort Study of Patients with Common Medical Conditions

BACKGROUND

Nursing resources, such as staffing ratios and skill mix, vary across hospitals. Better nursing resources have been linked to better patient outcomes but are assumed to increase costs. The value of investments in nursing resources, in terms of clinical benefits relative to costs, is unclear.

OBJECTIVE

To determine whether there are differential clinical outcomes, costs, and value among medical patients at hospitals characterized by better or worse nursing resources.

DESIGN

Matched cohort study of patients in 306 acute care hospitals.

PATIENTS

A total of 74,045 matched pairs of fee-for-service Medicare beneficiaries admitted for common medical conditions (25,446 sepsis pairs; 16,332 congestive heart failure pairs; 12,811 pneumonia pairs; 10,598 stroke pairs; 8858 acute myocardial infarction pairs). Patients were also matched on hospital size, technology, and teaching status.

MAIN MEASURES

Better (n = 76) and worse (n = 230) nursing resourced hospitals were defined by patient-to-nurse ratios, skill mix, proportions of bachelors-degree nurses, and nurse work environments. Outcomes included 30-day mortality, readmission, and resource utilization-based costs.

KEY RESULTS

Patients in hospitals with better nursing resources had significantly lower 30-day mortality (16.1% vs 17.1%, p < 0.0001) and fewer readmissions (32.3% vs 33.6%, p < 0.0001) yet costs were not significantly different ($18,848 vs 18,671, p = 0.133). The greatest outcomes and cost advantage of better nursing resourced hospitals were in patients with sepsis who had lower mortality (25.3% vs 27.6%, p < 0.0001). Overall, patients with the highest risk of mortality on admission experienced the greatest reductions in mortality and readmission from better nursing at no difference in cost.

CONCLUSIONS

Medicare beneficiaries with common medical conditions admitted to hospitals with better nursing resources experienced more favorable outcomes at almost no difference in cost.

Multivariate one-sided testing in matched observational studies as an adversarial game

We present a multivariate one-sided sensitivity analysis for matched observational studies, appropriate when the researcher has specified that a given causal mechanism should manifest itself in effects on multiple outcome variables in a known direction. The test statistic can be thought of as the solution to an adversarial game, where the researcher determines the best linear combination of test statistics to combat nature’s presentation of the worst-case pattern of hidden bias. The corresponding optimization problem is convex, and can be solved efficiently even for reasonably sized observational studies. Asymptotically, the test statistic converges to a chi-bar-squared distribution under the null, a common distribution in order-restricted statistical inference. The test attains the largest possible design sensitivity over a class of coherent test statistics, and facilitates one-sided sensitivity analyses for individual outcome variables while maintaining familywise error control through its incorporation into closed testing procedures.

Confounder selection strategies targeting stable treatment effect estimators

Inferring the causal effect of a treatment on an outcome in an observational study requires adjusting for observed baseline confounders to avoid bias. However, adjusting for all observed baseline covariates, when only a subset are confounders of the effect of interest, is known to yield potentially inefficient and unstable estimators of the treatment effect. Furthermore, it raises the risk of finite-sample bias and bias due to model misspecification. For these stated reasons, confounder (or covariate) selection is commonly used to determine a subset of the available covariates that is sufficient for confounding adjustment. In this article, we propose a confounder selection strategy that focuses on stable estimation of the treatment effect. In particular, when the propensity score (PS) model already includes covariates that are sufficient to adjust for confounding, then the addition of covariates that are associated with either treatment or outcome alone, but not both, should not systematically change the effect estimator. The proposal, therefore, entails first prioritizing covariates for inclusion in the PS model, then using a change-in-estimate approach to select the smallest adjustment set that yields a stable effect estimate. The ability of the proposal to correctly select confounders, and to ensure valid inference of the treatment effect following data-driven covariate selection, is assessed empirically and compared with existing methods using simulation studies. We demonstrate the procedure using three different publicly available datasets commonly used for causal inference.

Evaluating a Key Instrumental Variable Assumption Using Randomization Tests

Instrumental variable (IV) analyses are becoming common in health services research and epidemiology. Most IV analyses use naturally occurring instruments, such as distance to a hospital. In these analyses, investigators must assume that the instrument is as-if randomly assigned. This assumption cannot be tested directly, but it can be falsified. Most IV falsification tests compare relative prevalence or bias in observed covariates between the instrument and exposure. These tests require investigators to make covariate-by-covariate judgments about the validity of the IV design. Often, only some covariates are well-balanced, making it unclear whether as-if randomization can be assumed for the instrument. We propose an alternative falsification test that compares IV balance or bias with the balance or bias that would have been produced under randomization. A key advantage of our test is that it allows for global balance measures as well as easily interpretable graphical comparisons. Furthermore, our test does not rely on parametric assumptions and can be used to validly assess whether the instrument is significantly closer to being as-if randomized than the exposure. We demonstrate our approach using data from (SPOT)light, a prospective cohort study carried out in 48 National Health Service hospitals in the United Kingdom between November 1, 2010, and December 31, 2011. This study used bed availability in the intensive care unit as an instrument for admission to the intensive care unit.

Balancing vs modeling approaches to weighting in practice

There are two seemingly unrelated approaches to weighting in observational studies. One of them maximizes the fit of a model for treatment assignment to then derive weights-we call this the modeling approach. The other directly optimizes certain features of the weights-we call this the balancing approach. The implementations of these two approaches are related: the balancing approach implicitly models the propensity score, while instances of the modeling approach impose balance conditions on the covariates used to estimate the propensity score. In this article, we review and compare these two approaches to weighting. Previous review papers have focused on the modeling approach, emphasizing the importance of checking covariate balance. However, as we discuss, the dispersion of the weights is another important aspect of the weights to consider, in addition to the representativeness of the weighted sample and the sample boundedness of the weighted estimator. In particular, the dispersion of the weights is important because it translates into a measure of effective sample size, which can be used to select between alternative weighting schemes. In this article, we examine the balancing approach to weighting, discuss recent methodological developments, and compare instances of the balancing and modeling approaches in a simulation study and an empirical study. In practice, unless the treatment assignment model is known, we recommend using the balancing approach to weighting, as it systematically results in better covariate balance with weights that are minimally dispersed. As a result, effect estimates tend to be more accurate and stable.

Evaluating and improving a matched comparison of antidepressants and bone density

Matching is a common approach to covariate adjustment in estimating causal effects in observational studies. It is important to assess covariate balance of the matched samples. This is usually done informally, in ways that have a number of limitations. First, there are many diagnostics, even if covariates are assessed one at a time, which raises multiplicity issues. In addition, joint distributions of covariates, even bivariate distributions, are often ignored. Further, it is an open question whether diagnostics identify the major problems. To address these issues, a formal assessment of covariate balance is developed in the current paper. Unlike the common informal diagnostics, the proposed method compares both marginal distributions and joint distributions of the matched sample with those of the benchmark, complete randomizations. The method controls the probability of falsely identifying a covariate imbalance among many comparisons, yet it has a high probability of correctly detecting and identifying a major problem. An R package met implementing the proposed method is available on CRAN.

Optimal subset selection for causal inference using machine learning ensembles and particle swarm optimization

We suggest and evaluate a method for optimal construction of synthetic treatment and control samples for the purpose of drawing causal inference. The balance optimization subset selection problem, which formulates minimization of aggregate imbalance in covariate distributions to reduce bias in data, is a new area of study in operations research. We investigate a novel metric, cross-validated area under the receiver operating characteristic curve (AUC) as a measure of balance between treatment and control groups. The proposed approach provides direct and automatic balancing of covariate distributions. In addition, the AUC-based approach is able to detect subtler distributional differences than existing measures, such as simple empirical mean/variance and count-based metrics. Thus, optimizing AUCs achieves a greater balance than the existing methods. Using 5 widely used real data sets and 7 synthetic data sets, we show that optimization of samples using existing methods (Chi-square, mean variance differences, Kolmogorov–Smirnov, and Mahalanobis) results in samples containing imbalance that is detectable using machine learning ensembles. We minimize covariate imbalance by minimizing the absolute value of the distance of the maximum cross-validated AUC on $$ M $$ M folds from 0.50, using evolutionary optimization. We demonstrate that particle swarm optimization (PSO) outperforms modified cuckoo swarm (MCS) for a gradient-free, non-linear noisy cost function. To compute AUCs, we use supervised binary classification approaches from the machine learning and credit scoring literature. Using superscore ensembles adds to the classifier-based two-sample testing literature. If the mean cross-validated AUC based on machine learning is 0.50, the two groups are indistinguishable and suitable for causal inference.

A conditional test with demonstrated insensitivity to unmeasured bias in matched observational studies

In an observational study matched for observed covariates, an association between treatment received and outcome exhibited may indicate not an effect caused by the treatment, but merely some bias in the allocation of treatments to individuals within matched pairs. The evidence that distinguishes moderate biases from causal effects is unevenly dispersed among possible comparisons in an observational study: some comparisons are insensitive to larger biases than others. Intuitively, larger treatment effects tend to be insensitive to larger unmeasured biases, and perhaps matched pairs can be grouped using covariates, doses or response patterns so that groups of pairs with larger treatment effects may be identified. Even if an investigator has a reasoned conjecture about where to look for insensitive comparisons, that conjecture might prove mistaken, or, when not mistaken, it might be received sceptically by other scientists who doubt the conjecture or judge it to be too convenient in light of its success with the data at hand. In this article a test is proposed that searches for insensitive findings over many comparisons, but controls the probability of falsely rejecting a true null hypothesis of no treatment effect in the presence of a bias of specified magnitude. An example is studied in which the test considers many comparisons and locates an interpretable comparison that is insensitive to larger biases than a conventional comparison based on Wilcoxon’s signed rank statistic applied to all pairs. A simulation examines the power of the proposed test. The method is implemented in the R package dstat, which contains the example and reproduces the analysis.

Valuing hospital investments in nursing: multistate matched-cohort study of surgical patients

BACKGROUND

There are known clinical benefits associated with investments in nursing. Less is known about their value.

AIMS

To compare surgical patient outcomes and costs in hospitals with better versus worse nursing resources and to determine if value differs across these hospitals for patients with different mortality risks.

METHODS

Retrospective matched-cohort design of patient outcomes at hospitals with better versus worse nursing resources, defined by patient-to-nurse ratios, skill mix, proportions of bachelors-degree nurses and nurse work environments. The sample included 62 715 pairs of surgical patients in 76 better nursing resourced hospitals and 230 worse nursing resourced hospitals from 2013 to 2015. Patients were exactly matched on principal procedures and their hospital's size category, teaching and technology status, and were closely matched on comorbidities and other risk factors.

RESULTS

Patients in hospitals with better nursing resources had lower 30-day mortality: 2.7% vs 3.1% (p<0.001), lower failure-to-rescue: 5.4% vs 6.2% (p<0.001), lower readmissions: 12.6% vs 13.5% (p<0.001), shorter lengths of stay: 4.70 days vs 4.76 days (p<0.001), more intensive care unit admissions: 17.2% vs 15.4% (p<0.001) and marginally higher nurse-adjusted costs (which account for the costs of better nursing resources): $20 096 vs $19 358 (p<0.001), as compared with patients in worse nursing resourced hospitals. The nurse-adjusted cost associated with a 1% improvement in mortality at better nursing hospitals was $2035. Patients with the highest mortality risk realised the greatest value from nursing resources.

CONCLUSION

Hospitals with better nursing resources provided better clinical outcomes for surgical patients at a small additional cost. Generally, the sicker the patient, the greater the value at better nursing resourced hospitals.

Family-Based Treatment for Anorexia Nervosa Symptoms in High-Risk Youth: A Partially-Randomized Preference-Design Study

This pilot study adapted family-based treatment (FBT) for youth with potentially prodromal anorexia nervosa (AN). Fifty-nine youth with clinically significant AN symptom constellations, but who never met full Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.) (DSM-IV) criteria for AN, were enrolled in a partially randomized preference design study. Participants were offered randomization to FBT or supportive psychotherapy (SPT); those who declined to be randomized because of a strong treatment preference were entered into a parallel, non-randomized self-selected intervention study. Without accessing outcome data, an observational analysis with three diagnostic subclasses was designed based on AN symptom severity profiles, combining randomized and non-randomized participants, such that participants receiving FBT and SPT within each subclass were similar on key baseline characteristics. Outcomes of this pilot study were explored by calculating effect sizes for end-of-treatment values within each subclass, and also with a longitudinal mixed effect model that accounted for subclass. Weight trajectory was measured by percent expected body weight. Psychological outcomes were fear of weight gain, feeling fat, importance of weight, and importance of shape. Results show that the pattern of symptom observations over time was dependent on subclass of SAN (least symptomatic, moderately symptomatic, or most symptomatic) and on the target outcome variable category (weight or psychological). Results from this study, which should be considered in the context of the small sample sizes overall and within groups, can generate hypotheses for future, larger research trials on early treatment strategies. Feasibility findings illustrate how the innovative partially randomized preference design has potential broader application for AN intervention research.

CLINICAL TRIAL REGISTRATION

ClinicalTrials.gov, identifier NCT00418977.

Minimal dispersion approximately balancing weights: asymptotic properties and practical considerations

Weighting methods are widely used to adjust for covariates in observational studies, sample surveys, and regression settings. In this paper, we study a class of recently proposed weighting methods, which find the weights of minimum dispersion that approximately balance the covariates. We call these weights ‘minimal weights’ and study them under a common optimization framework. Our key observation is that finding weights which achieve approximate covariate balance is equivalent to performing shrinkage estimation of the inverse propensity score. This connection leads to both theoretical and practical developments. From a theoretical standpoint, we characterize the asymptotic properties of minimal weights and show that, under standard smoothness conditions on the propensity score function, minimal weights are consistent estimates of the true inverse probability weights. In addition, we show that the resulting weighting estimator is consistent, asymptotically normal and semiparametrically efficient. From a practical standpoint, we give a finite-sample oracle inequality that bounds the loss incurred by balancing more functions of the covariates than strictly needed. This inequality shows that minimal weights implicitly bound the number of active covariate balance constraints. Finally, we provide a tuning algorithm for choosing the degree of approximate balance in minimal weights. The paper concludes with an empirical study which suggests that approximate balance is preferable to exact balance, especially when there is limited overlap in covariate distributions. Further studies show that the root mean squared error of the weighting estimator can be reduced by as much as a half with approximate balance.

Robust policy evaluation from large-scale observational studies

Under the current policy decision making paradigm we make or evaluate a policy decision by intervening different socio-economic parameters and analyzing the impact of those interventions. This process involves identifying the causal relation between interventions and outcomes. Matching method is one of the popular techniques to identify such causal relations. However, in one-to-one matching, when a treatment or control unit has multiple pair assignment options with similar match quality, different matching algorithms often assign different pairs. Since all the matching algorithms assign pairs without considering the outcomes, it is possible that with the same data and same hypothesis, different experimenters can reach different conclusions creating an uncertainty in policy decision making. This problem becomes more prominent in the case of large-scale observational studies as there are more pair assignment options. Recently, a robust approach has been proposed to tackle the uncertainty that uses an integer programming model to explore all possible assignments. Though the proposed integer programming model is very efficient in making robust causal inference, it is not scalable to big data observational studies. With the current approach, an observational study with 50,000 samples will generate hundreds of thousands binary variables. Solving such integer programming problem is computationally expensive and becomes even worse with the increase of sample size. In this work, we consider causal inference testing with binary outcomes and propose computationally efficient algorithms that are adaptable for large-scale observational studies. By leveraging the structure of the optimization model, we propose a robustness condition that further reduces the computational burden. We validate the efficiency of the proposed algorithms by testing the causal relation between the Medicare Hospital Readmission Reduction Program (HRRP) and non-index readmissions (i.e., readmission to a hospital that is different from the hospital that discharged the patient) from the State of California Patient Discharge Database from 2010 to 2014. Our result shows that HRRP has a causal relation with the increase in non-index readmissions. The proposed algorithms proved to be highly scalable in testing causal relations from large-scale observational studies.

Matching estimators for causal effects of multiple treatments

Matching estimators for average treatment effects are widely used in the binary treatment setting, in which missing potential outcomes are imputed as the average of observed outcomes of all matches for each unit. With more than two treatment groups, however, estimation using matching requires additional techniques. In this paper, we propose a nearest-neighbors matching estimator for use with multiple, nominal treatments, and use simulations to show that this method is precise and has coverage levels that are close to nominal. In addition, we implement the proposed inference methods to examine the effects of different medication regimens on long-term pain for patients experiencing motor vehicle collision.

Machine learning methods for developing precision treatment rules with observational data

Clinical trials have identified a variety of predictor variables for use in precision treatment protocols, ranging from clinical biomarkers and symptom profiles to self-report measures of various sorts. Although such variables are informative collectively, none has proven sufficiently powerful to guide optimal treatment selection individually. This has prompted growing interest in the development of composite precision treatment rules (PTRs) that are constructed by combining information across a range of predictors. But this work has been hampered by the generally small samples in randomized clinical trials and the use of suboptimal analysis methods to analyze the resulting data. In this paper, we propose to address the sample size problem by: working with large observational electronic medical record databases rather than controlled clinical trials to develop preliminary PTRs; validating these preliminary PTRs in subsequent pragmatic trials; and using ensemble machine learning methods rather than individual algorithms to carry out statistical analyses to develop the PTRs. The major challenges in this proposed approach are that treatment are not randomly assigned in observational databases and that these databases often lack measures of key prescriptive predictors and mental disorder treatment outcomes. We proposed a tiered case-cohort design approach that uses innovative methods for measuring and balancing baseline covariates and estimating PTRs to address these challenges.