An adaptive Mantel-Haenszel test for sensitivity analysis in observational studies

In a sensitivity analysis in an observational study with a binary outcome, is it better to use all of the data or to focus on subgroups that are expected to experience the largest treatment effects? The answer depends on features of the data that may be difficult to anticipate, a trade-off between unknown effect-sizes and known sample sizes. We propose a sensitivity analysis for an adaptive test similar to the Mantel-Haenszel test. The adaptive test performs two highly correlated analyses, one focused analysis using a subgroup, one combined analysis using all of the data, correcting for multiple testing using the joint distribution of the two test statistics. Because the two component tests are highly correlated, this correction for multiple testing is small compared with, for instance, the Bonferroni inequality. The test has the maximum design sensitivity of two component tests. A simulation evaluates the power of a sensitivity analysis using the adaptive test. Two examples are presented. An R package, sensitivity2x2xk, implements the procedure.

Auditing Practice Style Variation in Pediatric Inpatient Asthma Care

IMPORTANCE

Asthma is the most prevalent chronic illness among children, remaining a leading cause of pediatric hospitalizations and representing a major financial burden to many health care systems.

OBJECTIVE

To implement a new auditing process examining whether differences in hospital practice style may be associated with potential resource savings or inefficiencies in treating pediatric asthma admissions.

DESIGN, SETTING, AND PARTICIPANTS

A retrospective matched-cohort design study, matched for asthma severity, compared practice patterns for patients admitted to Children's Hospital Association hospitals contributing data to the Pediatric Hospital Information System (PHIS) database. With 3 years of PHIS data on 48 887 children, an asthma template was constructed consisting of representative children hospitalized for asthma between April 1, 2011, and March 31, 2014. The template was matched with either a 1:1, 2:1, or 3:1 ratio at each of 37 tertiary care children's hospitals, depending on available sample size.

EXPOSURE

Treatment at each PHIS hospital.

MAIN OUTCOMESS AND MEASURES

Cost, length of stay, and intensive care unit (ICU) utilization.

RESULTS

After matching patients (n = 9100; mean [SD] age, 7.1 [3.6] years; 3418 [37.6%] females) to the template (n = 100, mean [SD] age, 7.2 [3.7] years; 37 [37.0%] females), there was no significant difference in observable patient characteristics at the 37 hospitals meeting the matching criteria. Despite similar characteristics of the patients, we observed large and significant variation in use of the ICUs as well as in length of stay and cost. For the same template-matched populations, comparing utilization between the 12.5th percentile (lower eighth) and 87.5th percentile (upper eighth) of hospitals, median cost varied by 87% ($3157 vs $5912 per patient; P < .001); total hospital length of stay varied by 47% (1.5 vs 2.2 days; P < .001); and ICU utilization was 254% higher (6.5% vs 23.0%; P < .001). Furthermore, the patterns of resource utilization by patient risk differed significantly across hospitals. For example, as patient risk increased one hospital displayed significantly increasing costs compared with their matched controls (comparative cost difference: lowest risk, -34.21%; highest risk, 53.27%; P < .001). In contrast, another hospital displayed significantly decreasing costs relative to their matched controls as patient risk increased (comparative cost difference: lowest risk, -10.12%; highest risk, -16.85%; P = .01).

CONCLUSIONS AND RELEVANCE

For children with asthma who had similar characteristics, we observed different hospital resource utilization; some values differed greatly, with important differences by initial patient risk. Through the template matching audit, hospitals and stakeholders can better understand where this excess variation occurs and can help to pinpoint practice styles that should be emulated or avoided.

Practice Patterns in Medicaid and Non-Medicaid Asthma Admissions

BACKGROUND AND OBJECTIVES

With American children experiencing increased Medicaid coverage, it has become especially important to determine if practice patterns differ between Medicaid and non-Medicaid patients. Auditing such potential differences must carefully compare like patients to avoid falsely identifying suspicious practice patterns. We asked if we could observe differences in practice patterns between Medicaid and non-Medicaid patients admitted for asthma inside major children's hospitals.

METHODS

A matched cohort design, studying 17 739 matched pairs of children (Medicaid to non-Medicaid) admitted for asthma in the same hospital between April 1, 2011 and March 31, 2014 in 40 Children's Hospital Association hospitals contributing data to the Pediatric Hospital Information System database. Patients were matched on age, sex, asthma severity, and other patient characteristics.

RESULTS

Medicaid patient median cost was $4263 versus $4160 for non-Medicaid patients (P < .001). Additionally, the median cost difference (Medicaid minus non-Medicaid) between individual pairs was only $84 (95% confidence interval: 44 to 124), and the mean cost difference was only $49 (95% confidence interval: -72 to 170). The 90th percentile costs were also similar between groups ($10 710 vs $10 948; P < .07). Length of stay (LOS) was also very similar; both groups had a median stay of 1 day, with a similar percentage of patients exceeding the 90th percentile of individual hospital LOS (7.1% vs 6.7%; P = .14). ICU use was also similar (10.1% vs 10.6%; P = .12).

CONCLUSIONS

For closely matched patients within the same hospital, Medicaid status did not importantly influence costs, LOS, or ICU use.

Retraction Notice

[This retracts the article DOI: 10.1093/biomet/asv053..].

Improving Medicare's Hospital Compare Mortality Model

OBJECTIVE

To improve the predictions provided by Medicare's Hospital Compare (HC) to facilitate better informed decisions regarding hospital choice by the public.

DATA SOURCES/SETTING

Medicare claims on all patients admitted for Acute Myocardial Infarction between 2009 through 2011.

STUDY DESIGN

Cohort analysis using a Bayesian approach, comparing the present assumptions of HC (using a constant mean and constant variance for all hospital random effects), versus an expanded model that allows for the inclusion of hospital characteristics to permit the data to determine whether they vary with attributes of hospitals, such as volume, capabilities, and staffing. Hospital predictions are then created using directly standardized estimates to facilitate comparisons between hospitals.

DATA COLLECTION/EXTRACTION METHODS

Medicare fee-for-service claims.

PRINCIPAL FINDINGS

Our model that included hospital characteristics produces very different predictions from the current HC model, with higher predicted mortality rates at hospitals with lower volume and worse characteristics. Using Chicago as an example, the expanded model would advise patients against seeking treatment at the smallest hospitals with worse technology and staffing.

CONCLUSION

To aid patients when selecting between hospitals, the Centers for Medicare and Medicaid Services (CMS) should improve the HC model by permitting its predictions to vary systematically with hospital attributes such as volume, capabilities, and staffing.

Indirect Standardization Matching: Assessing Specific Advantage and Risk Synergy

OBJECTIVE

To develop a method to allow a hospital to compare its performance using its entire patient population to the outcomes of very similar patients treated elsewhere.

DATA SOURCES/SETTING

Medicare claims in orthopedics and common general, gynecologic, and urologic surgery from Illinois, New York, and Texas from 2004 to 2006.

STUDY DESIGN

Using two example "focal" hospitals, each hospital's patients were matched to 10 very similar patients selected from 619 other hospitals.

DATA COLLECTION/EXTRACTION METHODS

All patients were used at each focal hospital, and we found the 10 closest matched patients from control hospitals with exactly the same principal procedure as each focal patient.

PRINCIPAL FINDINGS

We achieved exact matches on all procedures and very close matches for other patient characteristics for both hospitals. There were few to no differences between each hospital's patients and their matched control patients on most patient characteristics, yet large and significant differences were observed for mortality, failure-to-rescue, and cost.

CONCLUSION

Indirect standardization matching can produce fair audits of quality and cost, allowing for a comprehensive, transparent, and relevant assessment of all patients at a focal hospital. With this approach, hospitals will be better able to benchmark their performance and determine where quality improvement is most needed.

The cross-cut statistic and its sensitivity to bias in observational studies with ordered doses of treatment

A common practice with ordered doses of treatment and ordered responses, perhaps recorded in a contingency table with ordered rows and columns, is to cut or remove a cross from the table, leaving the outer corners--that is, the high-versus-low dose, high-versus-low response corners--and from these corners to compute a risk or odds ratio. This little remarked but common practice seems to be motivated by the oldest and most familiar method of sensitivity analysis in observational studies, proposed by Cornfield et al. (1959), which says that to explain a population risk ratio purely as bias from an unobserved binary covariate, the prevalence ratio of the covariate must exceed the risk ratio. Quite often, the largest risk ratio, hence the one least sensitive to bias by this standard, is derived from the corners of the ordered table with the central cross removed. Obviously, the corners use only a portion of the data, so a focus on the corners has consequences for the standard error as well as for bias, but sampling variability was not a consideration in this early and familiar form of sensitivity analysis, where point estimates replaced population parameters. Here, this cross-cut analysis is examined with the aid of design sensitivity and the power of a sensitivity analysis.

Large, Sparse Optimal Matching with Refined Covariate Balance in an Observational Study of the Health Outcomes Produced by New Surgeons

Every newly trained surgeon performs her first unsupervised operation. How do the health outcomes of her patients compare with the patients of experienced surgeons? Using data from 498 hospitals, we compare 1252 pairs comprised of a new surgeon and an experienced surgeon working at the same hospital. We introduce a new form of matching that matches patients of each new surgeon to patients of an otherwise similar experienced surgeon at the same hospital, perfectly balancing 176 surgical procedures and closely balancing a total of 2.9 million categories of patients; additionally, the individual patient pairs are as close as possible. A new goal for matching is introduced, called "refined covariate balance," in which a sequence of nested, ever more refined, nominal covariates is balanced as closely as possible, emphasizing the first or coarsest covariate in that sequence. A new algorithm for matching is proposed and the main new results prove that the algorithm finds the closest match in terms of the total within-pair covariate distances among all matches that achieve refined covariate balance. Unlike previous approaches to forcing balance on covariates, the new algorithm creates multiple paths to a match in a network, where paths that introduce imbalances are penalized and hence avoided to the extent possible. The algorithm exploits a sparse network to quickly optimize a match that is about two orders of magnitude larger than is typical in statistical matching problems, thereby permitting much more extensive use of fine and near-fine balance constraints. The match was constructed in a few minutes using a network optimization algorithm implemented in R. An R package called rcbalance implementing the method is available from CRAN.

Racial disparities in colon cancer survival: a matched cohort study

BACKGROUND

Differences in colon cancer survival by race are a recognized problem among Medicare beneficiaries.

OBJECTIVE

To determine to what extent the racial disparity in survival is due to disparity in presentation characteristics at diagnosis or disparity in subsequent treatment.

DESIGN

Black patients with colon cancer were matched with 3 groups of white patients: a "demographic characteristics" match controlling for age, sex, diagnosis year, and Survey, Epidemiology, and End Results (SEER) site; a "presentation" match controlling for demographic characteristics plus comorbid conditions and tumor characteristics, including stage and grade; and a "treatment" match, including presentation variables plus details of surgery, radiation, and chemotherapy.

SETTING

16 U.S. SEER sites.

PATIENTS

7677 black patients aged 65 years or older diagnosed between 1991 and 2005 in the SEER-Medicare database and 3 sets of 7677 matched white patients, followed until 31 December 2009.

MEASUREMENTS

5-year survival.

RESULTS

The absolute difference in 5-year survival between black and white patients was 9.9% (95% CI, 8.3% to 11.4%; P<0.001) in the demographic characteristics match. This disparity remained unchanged between 1991 and 2005. After matching for presentation characteristics, the difference decreased to 4.9% (CI, 3.6% to 6.1%; P<0.001). After additional matching for treatment, this difference decreased to 4.3% (CI, 2.9% to 5.5%; P<0.001). The disparity in survival attributed to treatment differences made up only an absolute 0.6% of the overall 9.9% survival disparity.

LIMITATION

An observational study limited to elderly Medicare fee-for-service beneficiaries living in selected geographic areas.

CONCLUSION

Racial disparities in colon cancer survival did not decrease among patients diagnosed between 1991 and 2005. This persistent disparity seemed to be more related to presentation characteristics at diagnosis than to subsequent treatment differences.

PRIMARY FUNDING SOURCE

Agency for Healthcare Research and Quality and National Science Foundation.

A hospital-specific template for benchmarking its cost and quality

OBJECTIVE

Develop an improved method for auditing hospital cost and quality tailored to a specific hospital's patient population.

DATA SOURCES/SETTING

Medicare claims in general, gynecologic and urologic surgery, and orthopedics from Illinois, New York, and Texas between 2004 and 2006.

STUDY DESIGN

A template of 300 representative patients from a single index hospital was constructed and used to match 300 patients at 43 hospitals that had a minimum of 500 patients over a 3-year study period.

DATA COLLECTION/EXTRACTION METHODS

From each of 43 hospitals we chose 300 patients most resembling the template using multivariate matching.

PRINCIPAL FINDINGS

We found close matches on procedures and patient characteristics, far more balanced than would be expected in a randomized trial. There were little to no differences between the index hospital's template and the 43 hospitals on most patient characteristics yet large and significant differences in mortality, failure-to-rescue, and cost.

CONCLUSION

Matching can produce fair, directly standardized audits. From the perspective of the index hospital, "hospital-specific" template matching provides the fairness of direct standardization with the specific institutional relevance of indirect standardization. Using this approach, hospitals will be better able to examine their performance, and better determine why they are achieving the results they observe.

Anesthesia technique, mortality, and length of stay after hip fracture surgery

IMPORTANCE

More than 300,000 hip fractures occur each year in the United States. Recent practice guidelines have advocated greater use of regional anesthesia for hip fracture surgery.

OBJECTIVE

To test the association of regional (ie, spinal or epidural) anesthesia vs general anesthesia with 30-day mortality and hospital length of stay after hip fracture.

DESIGN, SETTING, AND PATIENTS

We conducted a matched retrospective cohort study involving patients 50 years or older who were undergoing surgery for hip fracture at general acute care hospitals in New York State between July 1, 2004, and December 31, 2011. Our main analysis was a near-far instrumental variable match that paired patients who lived at different distances from hospitals that specialized in regional or general anesthesia. Supplementary analyses included a within-hospital match that paired patients within the same hospital and an across-hospital match that paired patients at different hospitals.

EXPOSURES

Spinal or epidural anesthesia; general anesthesia.

MAIN OUTCOMES AND MEASURES

Thirty-day mortality and hospital length of stay. Because the distribution of length of stay had long tails, we characterized this outcome using the Huber M estimate with Huber weights, a robust estimator similar to a trimmed mean.

RESULTS

Of 56,729 patients, 15,904 (28%) received regional anesthesia and 40,825 (72%) received general anesthesia. Overall, 3032 patients (5.3%) died. The M estimate of the length of stay was 6.2 days (95% CI, 6.2 to 6.2). The near-far matched analysis showed no significant difference in 30-day mortality by anesthesia type among the 21,514 patients included in this match: 583 of 10,757 matched patients (5.4%) who lived near a regional anesthesia-specialized hospital died vs 629 of 10,757 matched patients (5.8%) who lived near a general anesthesia-specialized hospital (instrumental variable estimate of risk difference, -1.1%; 95% CI, -2.8 to 0.5; P = .20). Supplementary analyses of within and across hospital patient matches yielded mortality findings to be similar to the main analysis. In the near-far match, regional anesthesia was associated with a 0.6-day shorter length of stay than general anesthesia (95% CI, -0.8 to -0.4, P < .001). Supplementary analyses also showed regional anesthesia to be associated with shorter length of stay, although the observed association was smaller in magnitude than in the main analysis.

CONCLUSIONS AND RELEVANCE

Among adults in acute care hospitals in New York State undergoing hip repair, the use of regional anesthesia compared with general anesthesia was not associated with lower 30-day mortality but was associated with a modestly shorter length of stay. These findings do not support a mortality benefit for regional anesthesia in this setting.

Template matching for auditing hospital cost and quality

OBJECTIVE

Develop an improved method for auditing hospital cost and quality.

DATA SOURCES/SETTING

Medicare claims in general, gynecologic and urologic surgery, and orthopedics from Illinois, Texas, and New York between 2004 and 2006.

STUDY DESIGN

A template of 300 representative patients was constructed and then used to match 300 patients at hospitals that had a minimum of 500 patients over a 3-year study period.

DATA COLLECTION/EXTRACTION METHODS

From each of 217 hospitals we chose 300 patients most resembling the template using multivariate matching.

PRINCIPAL FINDINGS

The matching algorithm found close matches on procedures and patient characteristics, far more balanced than measured covariates would be in a randomized clinical trial. These matched samples displayed little to no differences across hospitals in common patient characteristics yet found large and statistically significant hospital variation in mortality, complications, failure-to-rescue, readmissions, length of stay, ICU days, cost, and surgical procedure length. Similar patients at different hospitals had substantially different outcomes.

CONCLUSION

The template-matched sample can produce fair, directly standardized audits that evaluate hospitals on patients with similar characteristics, thereby making benchmarking more believable. Through examining matched samples of individual patients, administrators can better detect poor performance at their hospitals and better understand why these problems are occurring.

Case Definition and Design Sensitivity

In a case-referent study, cases of disease are compared to non-cases with respect to their antecedent exposure to a treatment in an effort to determine whether exposure causes some cases of the disease. Because exposure is not randomly assigned in the population, as it would be if the population were a vast randomized trial, exposed and unexposed subjects may differ prior to exposure with respect to covariates that may or may not have been measured. After controlling for measured pre-exposure differences, for instance by matching, a sensitivity analysis asks about the magnitude of bias from unmeasured covariates that would need to be present to alter the conclusions of a study that presumed matching for observed covariates removes all bias. The definition of a case of disease affects sensitivity to unmeasured bias. We explore this issue using: (i) an asymptotic tool, the design sensitivity, (ii) a simulation for finite samples, and (iii) an example. Under favorable circumstances, a narrower case definition can yield an increase in the design sensitivity, and hence an increase in the power of a sensitivity analysis. Also, we discuss an adaptive method that seeks to discover the best case definition from the data at hand while controlling for multiple testing. An implementation in R is available as SensitivityCaseControl.