Combining planned and discovered comparisons in observational studies

In observational studies of treatment effects, it is common to have several outcomes, perhaps of uncertain quality and relevance, each purporting to measure the effect of the treatment. A single planned combination of several outcomes may increase both power and insensitivity to unmeasured bias when the plan is wisely chosen, but it may miss opportunities in other cases. A method is proposed that uses one planned combination with only a mild correction for multiple testing and exhaustive consideration of all possible combinations fully correcting for multiple testing. The method works with the joint distribution of $\kappa^{T}\left( \mathbf{T}-\boldsymbol{\mu}\right) /\sqrt {\boldsymbol{\kappa}^{T}\boldsymbol{\Sigma\boldsymbol{\kappa}}}$ and $max_{\boldsymbol{\lambda}\neq\mathbf{0}}$$\,\lambda^{T}\left( \mathbf{T} -\boldsymbol{\mu}\right) /$$\sqrt{\boldsymbol{\lambda}^{T}\boldsymbol{\Sigma \lambda}}$ where $\kappa$ is chosen a priori and the test statistic $\mathbf{T}$ is asymptotically $N_{L}\left( \boldsymbol{\mu},\boldsymbol{\Sigma}\right) $. The correction for multiple testing has a smaller effect on the power of $\kappa^{T}\left( \mathbf{T}-\boldsymbol{\mu }\right) /\sqrt{\boldsymbol{\kappa}^{T}\boldsymbol{\Sigma\boldsymbol{\kappa} }}$ than does switching to a two-tailed test, even though the opposite tail does receive consideration when $\lambda=-\kappa$. In the application, there are three measures of cognitive decline, and the a priori comparison $\kappa$ is their first principal component, computed without reference to treatment assignments. The method is implemented in an R package sensitivitymult.

A conditional test with demonstrated insensitivity to unmeasured bias in matched observational studies

In an observational study matched for observed covariates, an association between treatment received and outcome exhibited may indicate not an effect caused by the treatment, but merely some bias in the allocation of treatments to individuals within matched pairs. The evidence that distinguishes moderate biases from causal effects is unevenly dispersed among possible comparisons in an observational study: some comparisons are insensitive to larger biases than others. Intuitively, larger treatment effects tend to be insensitive to larger unmeasured biases, and perhaps matched pairs can be grouped using covariates, doses or response patterns so that groups of pairs with larger treatment effects may be identified. Even if an investigator has a reasoned conjecture about where to look for insensitive comparisons, that conjecture might prove mistaken, or, when not mistaken, it might be received sceptically by other scientists who doubt the conjecture or judge it to be too convenient in light of its success with the data at hand. In this article a test is proposed that searches for insensitive findings over many comparisons, but controls the probability of falsely rejecting a true null hypothesis of no treatment effect in the presence of a bias of specified magnitude. An example is studied in which the test considers many comparisons and locates an interpretable comparison that is insensitive to larger biases than a conventional comparison based on Wilcoxon’s signed rank statistic applied to all pairs. A simulation examines the power of the proposed test. The method is implemented in the R package dstat, which contains the example and reproduces the analysis.

Valuing hospital investments in nursing: multistate matched-cohort study of surgical patients

BACKGROUND

There are known clinical benefits associated with investments in nursing. Less is known about their value.

AIMS

To compare surgical patient outcomes and costs in hospitals with better versus worse nursing resources and to determine if value differs across these hospitals for patients with different mortality risks.

METHODS

Retrospective matched-cohort design of patient outcomes at hospitals with better versus worse nursing resources, defined by patient-to-nurse ratios, skill mix, proportions of bachelors-degree nurses and nurse work environments. The sample included 62 715 pairs of surgical patients in 76 better nursing resourced hospitals and 230 worse nursing resourced hospitals from 2013 to 2015. Patients were exactly matched on principal procedures and their hospital's size category, teaching and technology status, and were closely matched on comorbidities and other risk factors.

RESULTS

Patients in hospitals with better nursing resources had lower 30-day mortality: 2.7% vs 3.1% (p<0.001), lower failure-to-rescue: 5.4% vs 6.2% (p<0.001), lower readmissions: 12.6% vs 13.5% (p<0.001), shorter lengths of stay: 4.70 days vs 4.76 days (p<0.001), more intensive care unit admissions: 17.2% vs 15.4% (p<0.001) and marginally higher nurse-adjusted costs (which account for the costs of better nursing resources): $20 096 vs $19 358 (p<0.001), as compared with patients in worse nursing resourced hospitals. The nurse-adjusted cost associated with a 1% improvement in mortality at better nursing hospitals was $2035. Patients with the highest mortality risk realised the greatest value from nursing resources.

CONCLUSION

Hospitals with better nursing resources provided better clinical outcomes for surgical patients at a small additional cost. Generally, the sicker the patient, the greater the value at better nursing resourced hospitals.

Using Evidence Factors to Clarify Exposure Biomarkers

A study has 2 evidence factors if it permits 2 statistically independent inferences about 1 treatment effect such that each factor is immune to some bias that would invalidate the other factor. Because the 2 factors are statistically independent, the evidence they provide can be combined using methods associated with meta-analysis for independent studies, despite using the same data twice in different ways. We illustrate evidence factors, applying them in a new way in investigations that have both an exposure biomarker and a coarse external measure of exposure to a treatment. To illustrate, we consider the possible effects of cigarette smoking on homocysteine levels, with self-reported smoking and a cotinine biomarker. We examine joint sensitivity of 2 factors to bias from confounding, a central aspect of any observational study.

Comparing Resource Use in Medical Admissions of Children With Complex Chronic Conditions

BACKGROUND

Children with complex chronic conditions (CCCs) utilize a disproportionate share of hospital resources.

OBJECTIVE

We asked whether some hospitals display a significantly different pattern of resource utilization than others when caring for similar children with CCCs admitted for medical diagnoses.

RESEARCH DESIGN

Using Pediatric Health Information System data from 2009 to 2013, we constructed an inpatient Template of 300 children with CCCs, matching these to 300 patients at each hospital, thereby performing a type of direct standardization.

SUBJECTS

Children with CCCs were drawn from a list of the 40 most common medical principal diagnoses, then matched to patients across 40 Children's Hospitals.

MEASURES

Rate of intensive care unit admission, length of stay, resource cost.

RESULTS

For the Template-matched patients, when comparing resource use at the lower 12.5-percentile and upper 87.5-percentile of hospitals, we found: intensive care unit utilization was 111% higher (6.6% vs. 13.9%, P<0.001); hospital length of stay was 25% higher (2.4 vs. 3.0 d/admission, P<0.001); and finally, total cost per patient varied by 47% ($6856 vs. $10,047, P<0.001). Furthermore, some hospitals, compared with their peers, were more efficient with low-risk patients and less efficient with high-risk patients, whereas other hospitals displayed the opposite pattern.

CONCLUSIONS

Hospitals treating similar patients with CCCs admitted for similar medical diagnoses, varied greatly in resource utilization. Template Matching can aid chief quality officers benchmarking their hospitals to peer institutions and can help determine types of their patients having the most aberrant outcomes, facilitating quality initiatives to target these patients.

Disparities in Breast Cancer Survival by Socioeconomic Status Despite Medicare and Medicaid Insurance

Policy Points Patients with low socioeconomic status (SES) experience poorer survival rates after diagnosis of breast cancer, even when enrolled in Medicare and Medicaid. Most of the difference in survival is due to more advanced cancer on presentation and the general poor health of lower SES patients, while only a very small fraction of the SES disparity is due to differences in cancer treatment. Even when comparing only low- versus not-low-SES whites (without confounding by race) the survival disparity between disparate white SES populations is very large and is associated with lower use of preventive care, despite having insurance.

CONTEXT

Disparities in breast cancer survival by socioeconomic status (SES) exist despite the "safety net" programs Medicare and Medicaid. What is less clear is the extent to which SES disparities affect various racial and ethnic groups and whether causes differ across populations.

METHODS

We conducted a tapered matching study comparing 1,890 low-SES (LSES) non-Hispanic white, 1,824 black, and 723 Hispanic white women to 60,307 not-low-SES (NLSES) non-Hispanic white women, all in Medicare and diagnosed with invasive breast cancer between 1992 and 2010 in 17 US Surveillance, Epidemiology, and End Results (SEER) regions. LSES Medicare patients were Medicaid dual-eligible and resided in neighborhoods with both high poverty and low education. NLSES Medicare patients had none of these factors.

MEASUREMENTS

5-year and median survival.

FINDINGS

LSES non-Hispanic white patients were diagnosed with more stage IV disease (6.6% vs 3.6%; p < 0.0001), larger tumors (24.6 mm vs 20.2 mm; p < 0.0001), and more chronic diseases such as diabetes (37.8% vs 19.0%; p < 0.0001) than NLSES non-Hispanic white patients. Disparity in 5-year survival (NLSES - LSES) was 13.7% (p < 0.0001) when matched for age, year, and SEER site (a 42-month difference in median survival). Additionally, matching 55 presentation factors, including stage, reduced the disparity to 4.9% (p = 0.0012), but further matching on treatments yielded little further change in disparity: 4.6% (p = 0.0014). Survival disparities among LSES blacks and Hispanics, also versus NLSES whites, were significantly associated with presentation factors, though black patients also displayed disparities related to initial treatment. Before being diagnosed, all LSES populations used significantly less preventive care services than matched NLSES controls.

CONCLUSIONS

In Medicare, SES disparities in breast cancer survival were large (even among non-Hispanic whites) and predominantly related to differences of presentation characteristics at diagnosis rather than differences in treatment. Preventive care was less frequent in LSES patients, which may help explain disparities at presentation.

Adjustment for Atherosclerosis Diagnosis Distorts the Effects of Percutaneous Coronary Intervention and the Ranking of Hospital Performance

BACKGROUND

Coronary atherosclerosis raises the risk of acute myocardial infarction (AMI), and is usually included in AMI risk-adjustment models. Percutaneous coronary intervention (PCI) does not cause atherosclerosis, but may contribute to the notation of atherosclerosis in administrative claims. We investigated how adjustment for atherosclerosis affects rankings of hospitals that perform PCI.

METHODS AND RESULTS

This was a retrospective cohort study of 414 715 Medicare beneficiaries hospitalized for AMI between 2009 and 2011. The outcome was 30-day mortality. Regression models determined the association between patient characteristics and mortality. Rankings of the 100 largest PCI and non-PCI hospitals were assessed with and without atherosclerosis adjustment. Patients admitted to PCI hospitals or receiving interventional cardiology more frequently had an atherosclerosis diagnosis. In adjustment models, atherosclerosis was associated, implausibly, with a 42% reduction in odds of mortality (odds ratio=0.58, P<0.0001). Without adjustment for atherosclerosis, the number of expected lives saved by PCI hospitals increased by 62% (P<0.001). Hospital rankings also changed: 72 of the 100 largest PCI hospitals had better ranks without atherosclerosis adjustment, while 77 of the largest non-PCI hospitals had worse ranks (P<0.001).

CONCLUSIONS

Atherosclerosis is almost always noted in patients with AMI who undergo interventional cardiology but less often in medically managed patients, so adjustment for its notation likely removes part of the effect of interventional treatment. Therefore, hospitals performing more extensive imaging and more PCIs have higher atherosclerosis diagnosis rates, making their patients appear healthier and artificially reducing the expected mortality rate against which they are benchmarked. Thus, atherosclerosis adjustment is detrimental to hospitals providing more thorough AMI care.

A powerful approach to the study of moderate effect modification in observational studies

Effect modification means the magnitude or stability of a treatment effect varies as a function of an observed covariate. Generally, larger and more stable treatment effects are insensitive to larger biases from unmeasured covariates, so a causal conclusion may be considerably firmer if this pattern is noted if it occurs. We propose a new strategy, called the submax-method, that combines exploratory, and confirmatory efforts to determine whether there is stronger evidence of causality-that is, greater insensitivity to unmeasured confounding-in some subgroups of individuals. It uses the joint distribution of test statistics that split the data in various ways based on certain observed covariates. For L binary covariates, the method splits the population L times into two subpopulations, perhaps first men and women, perhaps then smokers and nonsmokers, computing a test statistic from each subpopulation, and appends the test statistic for the whole population, making <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>2</mml:mn> <mml:mi>L</mml:mi> <mml:mo>+</mml:mo> <mml:mn>1</mml:mn></mml:math> test statistics in total. Although L binary covariates define <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msup><mml:mn>2</mml:mn> <mml:mi>L</mml:mi></mml:msup> </mml:math> interaction groups, only <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>2</mml:mn> <mml:mi>L</mml:mi> <mml:mo>+</mml:mo> <mml:mn>1</mml:mn></mml:math> tests are performed, and at least <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>L</mml:mi> <mml:mo>+</mml:mo> <mml:mn>1</mml:mn></mml:math> of these tests use at least half of the data. The submax-method achieves the highest design sensitivity and the highest Bahadur efficiency of its component tests. Moreover, the form of the test is sufficiently tractable that its large sample power may be studied analytically. The simulation suggests that the submax method exhibits superior performance, in comparison with an approach using CART, when there is effect modification of moderate size. Using data from the NHANES I epidemiologic follow-up survey, an observational study of the effects of physical activity on survival is used to illustrate the method. The method is implemented in the <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>R</mml:mi></mml:math> package <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>submax</mml:mi></mml:math> which contains the NHANES example. An online Appendix provides simulation results and further analysis of the example.

Comparison of the Value of Nursing Work Environments in Hospitals Across Different Levels of Patient Risk

IMPORTANCE

The literature suggests that hospitals with better nursing work environments provide better quality of care. Less is known about value (cost vs quality).

OBJECTIVES

To test whether hospitals with better nursing work environments displayed better value than those with worse nursing environments and to determine patient risk groups associated with the greatest value.

DESIGN, SETTING, AND PARTICIPANTS

A retrospective matched-cohort design, comparing the outcomes and cost of patients at focal hospitals recognized nationally as having good nurse working environments and nurse-to-bed ratios of 1 or greater with patients at control group hospitals without such recognition and with nurse-to-bed ratios less than 1. This study included 25 752 elderly Medicare general surgery patients treated at focal hospitals and 62 882 patients treated at control hospitals during 2004-2006 in Illinois, New York, and Texas. The study was conducted between January 1, 2004, and November 30, 2006; this analysis was conducted from April to August 2015.

EXPOSURES

Focal vs control hospitals (better vs worse nursing environment).

MAIN OUTCOMES AND MEASURES

Thirty-day mortality and costs reflecting resource utilization.

RESULTS

This study was conducted at 35 focal hospitals (mean nurse-to-bed ratio, 1.51) and 293 control hospitals (mean nurse-to-bed ratio, 0.69). Focal hospitals were larger and more teaching and technology intensive than control hospitals. Thirty-day mortality in focal hospitals was 4.8% vs 5.8% in control hospitals (P < .001), while the cost per patient was similar: the focal-control was -$163 (95% CI = -$542 to $215; P = .40), suggesting better value in the focal group. For the focal vs control hospitals, the greatest mortality benefit (17.3% vs 19.9%; P < .001) occurred in patients in the highest risk quintile, with a nonsignificant cost difference of $941 per patient ($53 701 vs $52 760; P = .25). The greatest difference in value between focal and control hospitals appeared in patients in the second-highest risk quintile, with mortality of 4.2% vs 5.8% (P < .001), with a nonsignificant cost difference of -$862 ($33 513 vs $34 375; P = .12).

CONCLUSIONS AND RELEVANCE

Hospitals with better nursing environments and above-average staffing levels were associated with better value (lower mortality with similar costs) compared with hospitals without nursing environment recognition and with below-average staffing, especially for higher-risk patients. These results do not suggest that improving any specific hospital's nursing environment will necessarily improve its value, but they do show that patients undergoing general surgery at hospitals with better nursing environments generally receive care of higher value.

Comparing Outcomes between Antibody Induction Therapies in Kidney Transplantation

Kidney transplant recipients often receive antibody induction. Previous studies of induction therapy were often limited by short follow-up and/or absence of information about complications. After linking Organ Procurement and Transplantation Network data with Medicare claims, we compared outcomes between three induction therapies for kidney recipients. Using novel matching techniques developed on the basis of 15 clinical and demographic characteristics, we generated 1:1 pairs of alemtuzumab-rabbit antithymocyte globulin (rATG) (5330 pairs) and basiliximab-rATG (9378 pairs) recipients. We used paired Cox regression to analyze the primary outcomes of death and death or allograft failure. Secondary outcomes included death or sepsis, death or lymphoma, death or melanoma, and healthcare resource utilization within 1 year. Compared with rATG recipients, alemtuzumab recipients had higher risk of death (hazard ratio [HR], 1.14; 95% confidence interval [95% CI], 1.03 to 1.26; P<0.01) and death or allograft failure (HR, 1.18; 95% CI, 1.09 to 1.28; P<0.001). Results for death as well as death or allograft failure were generally consistent among elderly and nonelderly subgroups and among pairs receiving oral prednisone. Compared with rATG recipients, basiliximab recipients had higher risk of death (HR, 1.08; 95% CI, 1.01 to 1.16; P=0.03) and death or lymphoma (HR, 1.12; 95% CI, 1.01 to 1.23; P=0.03), although these differences were not confirmed in subgroup analyses. One-year resource utilization was slightly lower among alemtuzumab recipients than among rATG recipients, but did not differ between basiliximab and rATG recipients. This observational evidence indicates that, compared with alemtuzumab and basiliximab, rATG associates with lower risk of adverse outcomes, including mortality.

Outcomes, ICU Use, and Length of Stay in Chronically Ill Black and White Children on Medicaid and Hospitalized for Surgery

BACKGROUND

With increasing Medicaid coverage, it has become especially important to determine whether racial differences exist within the Medicaid system. We asked whether disparities exist in hospital practice and patient outcomes between matched black and white Medicaid children with chronic conditions undergoing surgery.

STUDY DESIGN

We conducted a matched cohort study, matching 6,398 pairs within states on detailed patient characteristics using data from 25 states contributing adequate Medicaid Analytic eXtract claims for admissions of children with chronic conditions undergoing the same surgical procedures between January 1, 2009 and November 30, 2010 for ages 1 to 18 years.

RESULTS

The black patient 30-day revisit rate was 19.3% vs 19.8% in matched white patients (p = 0.61), 30-day readmission rates were 7.0% vs 6.9% (p = 0.43), and 30-day mortality rates were 0.38% vs 0.19% (p = 0.06), respectively. A higher percentage of black patients exceeded their own state's individual median length of stay (44.0% vs 39.6%; p < 0.001) and median ICU length of stay (25.9% vs 23.8%; p < 0.001). Intensive care unit use was higher in black patients (25.9% vs 23.8%; p < 0.001). After adjusting for multiple testing, only 2 states were found to differ significantly by race (New York for length of stay and New Jersey for ICU use).

CONCLUSIONS

We did not observe disparities in 30-day revisits and readmissions for chronically ill children in Medicaid undergoing surgery, and only slight differences in length of stay, ICU length of stay, and use of the ICU, where blacks displayed somewhat elevated rates compared with white controls.

Racial Disparities in Medicaid Asthma Hospitalizations

BACKGROUND AND OBJECTIVES

Black children with asthma comprise one-third of all asthma patients in Medicaid. With increasing Medicaid coverage, it has become especially important to monitor Medicaid for differences in hospital practice and patient outcomes by race.

METHODS

A multivariate matched cohort design, studying 11 079 matched pairs of children in Medicaid (black versus white matched pairs from inside the same state) admitted for asthma between January 1, 2009 and November 30, 2010 in 33 states contributing adequate Medicaid Analytic eXtract claims.

RESULTS

Ten-day revisit rates were 3.8% in black patients versus 4.2% in white patients (P = .12); 30-day revisit and readmission rates were also not significantly different by race (10.5% in black patients versus 10.8% in white patients; P = .49). Length of stay (LOS) was also similar; both groups had a median stay of 2.0 days, with a slightly lower percentage of black patients exceeding their own state's median LOS (30.2% in black patients versus 31.8% in white patients; P = .01). The mean paired difference in LOS was 0.00 days (95% confidence interval, -0.08 to 0.08). However, ICU use was higher in black patients than white patients (22.2% versus 17.5%; P < .001). After adjusting for multiple testing, only 4 states were found to differ significantly, but only in ICU use, where blacks had higher rates of use.

CONCLUSIONS

For closely matched black and white patients, racial disparities concerning asthma admission outcomes and style of practice are small and generally nonsignificant, except for ICU use, where we observed higher rates in black patients.

Dropping out of High School in the United States: An Observational Study

Using data from a two-stage probability sample of U.S. high school students, an attempt is made to estimate the effect that dropping out has on cognitive achievement test scores. Each sampled dropout from a school is matched by a multivariate procedure to a student who remained in the same school. The matched pair differences are then adjusted using analysis of covariance. The possibility that important covariates have been omitted from the analysis is addressed through tests of ignorable treatment assignment and through sensitivity analyses.