The Use of Fixed-and Random-Effects Models for Classifying Hospitals as Mortality Outliers: A Monte Carlo Assessment

Prognostic score-based methods for estimating center effects based on survival probability: Application to post-kidney transplant survival

In evaluating the performance of different facilities or centers on survival outcomes, the standardized mortality ratio (SMR), which compares the observed to expected mortality has been widely used, particularly in the evaluation of kidney transplant centers. Despite its utility, the SMR may exaggerate center effects in settings where survival probability is relatively high. An example is one-year graft survival among U.S. kidney transplant recipients. We propose a novel approach to estimate center effects in terms of differences in survival probability (ie, each center versus a reference population). An essential component of the method is a prognostic score weighting technique, which permits accurately evaluating centers without necessarily specifying a correct survival model. Advantages of our approach over existing facility-profiling methods include a metric based on survival probability (greater clinical relevance than ratios of counts/rates); direct standardization (valid to compare between centers, unlike indirect standardization based methods, such as the SMR); and less reliance on correct model specification (since the assumed model is used to generate risk classes as opposed to fitted-value based 'expected' counts). We establish the asymptotic properties of the proposed weighted estimator and evaluate its finite-sample performance under a diverse set of simulation settings. The method is then applied to evaluate U.S. kidney transplant centers with respect to graft survival probability.

Predictive Validity of Anesthesiologists' Quality of Clinical Supervision and Nurse Anesthetists' Work Habits Assessed by Their Associations With Operating Room Times

BACKGROUND

At all Joint Commission-accredited hospitals, the anesthesia department chair must report quantitative assessments of anesthesiologists' and nurse anesthetists' (CRNAs') clinical performance at least annually. Most metrics lack evidence of usefulness, cost-effectiveness, reliability, or validity. Earlier studies showed that anesthesiologists' clinical supervision quality and CRNAs' work habits have content, convergent, discriminant, and construct validity. We evaluated predictive validity by testing for (expected) small but statistically significant associations between higher quality of supervision (work habits) and reduced probabilities of cases taking longer than estimated.

METHODS

Supervision quality of each anesthesiologist was evaluated daily by assigned trainees using the 9-item de Oliveira Filho scale. The work habits of each CRNA were evaluated daily by assigned anesthesiologists using a 6-item scale. Both are scored binary, 1 if all items are rated the maximum, 0 otherwise. From 40,718 supervision evaluations and 53,722 work habit evaluations over 8 fiscal years, 16 mixed-effects logistic regression models were estimated, with raters as fixed effects and ratees (anesthesiologists or CRNAs) as random effects. Empirical Bayes means in the logit scale were obtained for 561 anesthesiologist-years and 605 CRNA-years. The binary-dependent variable was whether the case took longer than estimated from the historical mean time for combinations of scheduled procedures and surgeons. From 264,060 cases, 8 mixed-effects logistic regression models were fitted, 1 per fiscal year, using ratees as random effects. Predictive validity was tested by pairing the 8 one-year analyses of clinical supervision, and the 8 one-year analyses of work habits, by ratee, with the 8 one-year analyses of whether OR time was longer than estimated. Bivariate errors in variable linear least squares linear regressions minimized total variances.

RESULTS

Among anesthesiologists, 8.2% (46/561) had below-average supervision quality, and 17.7% (99/561), above-average. Among CRNAs, 6.3% (38/605) had below-average work habits, and 10.9% (66/605) above-average. Increases in the logits of the quality of clinical supervision were associated with decreases in the logits of the probabilities of cases taking longer than estimated, unitless slope = -0.0361 (SE, 0.0053), P < .00001. Increases in the logits of CRNAs' work habits were associated with decreases in the logits of probabilities of cases taking longer than estimated, slope = -0.0238 (SE, 0.0054), P < .00001.

CONCLUSIONS

Predictive validity was confirmed, providing further evidence for using supervision and work habits scales for ongoing professional practice evaluations. Specifically, OR times were briefer when anesthesiologists supervised residents more closely, and when CRNAs had better work habits.

Evaluations of statistical methods for outlier detection when benchmarking in clinical registries: a systematic review

OBJECTIVES

Benchmarking is common in clinical registries to support the improvement of health outcomes by identifying underperforming clinician or health service providers. Despite the rise in clinical registries and interest in publicly reporting benchmarking results, appropriate methods for benchmarking and outlier detection within clinical registries are not well established, and the current application of methods is inconsistent. The aim of this review was to determine the current statistical methods of outlier detection that have been evaluated in the context of clinical registry benchmarking.

DESIGN

A systematic search for studies evaluating the performance of methods to detect outliers when benchmarking in clinical registries was conducted in five databases: EMBASE, ProQuest, Scopus, Web of Science and Google Scholar. A modified healthcare modelling evaluation tool was used to assess quality; data extracted from each study were summarised and presented in a narrative synthesis.

RESULTS

Nineteen studies evaluating a variety of statistical methods in 20 clinical registries were included. The majority of studies conducted application studies comparing outliers without statistical performance assessment (79%), while only few studies used simulations to conduct more rigorous evaluations (21%). A common comparison was between random effects and fixed effects regression, which provided mixed results. Registry population coverage, provider case volume minimum and missing data handling were all poorly reported.

CONCLUSIONS

The optimal methods for detecting outliers when benchmarking clinical registry data remains unclear, and the use of different models may provide vastly different results. Further research is needed to address the unresolved methodological considerations and evaluate methods across a range of registry conditions.

PROSPERO REGISTRATION NUMBER

CRD42022296520.

Outlier identification and monitoring of institutional or clinician performance: an overview of statistical methods and application to national audit data

BACKGROUND

Institutions or clinicians (units) are often compared according to a performance indicator such as in-hospital mortality. Several approaches have been proposed for the detection of outlying units, whose performance deviates from the overall performance.

METHODS

We provide an overview of three approaches commonly used to monitor institutional performances for outlier detection. These are the common-mean model, the 'Normal-Poisson' random effects model and the 'Logistic' random effects model. For the latter we also propose a visualisation technique. The common-mean model assumes that the underlying true performance of all units is equal and that any observed variation between units is due to chance. Even after applying case-mix adjustment, this assumption is often violated due to overdispersion and a post-hoc correction may need to be applied. The random effects models relax this assumption and explicitly allow the true performance to differ between units, thus offering a more flexible approach. We discuss the strengths and weaknesses of each approach and illustrate their application using audit data from England and Wales on Adult Cardiac Surgery (ACS) and Percutaneous Coronary Intervention (PCI).

RESULTS

In general, the overdispersion-corrected common-mean model and the random effects approaches produced similar p-values for the detection of outliers. For the ACS dataset (41 hospitals) three outliers were identified in total but only one was identified by all methods above. For the PCI dataset (88 hospitals), seven outliers were identified in total but only two were identified by all methods. The common-mean model uncorrected for overdispersion produced several more outliers. The reason for observing similar p-values for all three approaches could be attributed to the fact that the between-hospital variance was relatively small in both datasets, resulting only in a mild violation of the common-mean assumption; in this situation, the overdispersion correction worked well.

CONCLUSION

If the common-mean assumption is likely to hold, all three methods are appropriate to use for outlier detection and their results should be similar. Random effect methods may be the preferred approach when the common-mean assumption is likely to be violated.

Variation in hospital morbidities in an Australian neonatal intensive care unit network

OBJECTIVE

There is an expectation among the public and within the profession that the performance and outcome of neonatal intensive care units (NICUs) should be comparable between centres with a similar setting. This study aims to benchmark and audit performance variation in a regional Australian network of eight NICUs.

DESIGN

Cohort study using prospectively collected data.

SETTING

All eight perinatal centres in New South Wales and the Australian Capital Territory, Australia.

PATIENTS

All live-born infants born between 23⁺⁰ and 31⁺⁶ weeks gestation admitted to one of the tertiary perinatal centres from 2007 to 2020 (n=12 608).

MAIN OUTCOME MEASURES

Early and late confirmed sepsis, intraventricular haemorrhage, medically and surgically treated patent ductus arteriosus, chronic lung disease (CLD), postnatal steroid for CLD, necrotising enterocolitis, retinopathy of prematurity (ROP), surgery for ROP, hospital mortality and home oxygen.

RESULTS

NICUs showed variations in maternal and neonatal characteristics and resources. The unadjusted funnel plots for neonatal outcomes showed apparent variation with multiple centres outside the 99.8% control limits of the network values. The hierarchical model-based risk-adjustment accounting for differences in patient characteristics showed that discharged home with oxygen is the only outcome above the 99.8% control limits.

CONCLUSIONS

Hierarchical model-based risk-adjusted estimates of morbidity rates plotted on funnel plots provide a robust and straightforward visual graphical tool for presenting variations in outcome performance to detect aberrations in healthcare delivery and guide timely intervention. We propose using hierarchical model-based risk adjustment and funnel plots in real or near real-time to detect aberrations and start timely intervention.

Hospital variation in admissions for low back pain following an emergency department presentation: a retrospective study

Background

One in 6 patients with low back pain (LBP) presenting to emergency departments (EDs) are subsequently admitted to hospital each year, making LBP the ninth most common reason for hospital admission in Australia. No studies have investigated and quantified the extent of clinical variation in hospital admission following an ED presentation for LBP.

Methods

We used routinely collected ED data from public hospitals within the state of New South Wales, Australia, to identify presentations of patients aged between 18 and 111 with a discharge diagnosis of LBP. We fitted a series of random effects multilevel logistic regression models adjusted by case-mix and hospital variables. The main outcome was the hospital-adjusted admission rate (HAAR). Data were presented as funnel plots with 95% and 99.8% confidence limits. Hospitals with a HAAR outside the 95% confidence limit were considered to have a HAAR significantly different to the state average.

Results

We identified 176,729 LBP presentations across 177 public hospital EDs and 44,549 hospital admissions (25.2%). The mean (SD) age was 51.8 (19.5) and 52% were female. Hospital factors explained 10% of the variation (ICC = 0.10), and the median odds ratio (MOR) was 2.03. We identified marked variation across hospitals, with HAAR ranging from 6.9 to 65.9%. After adjusting for hospital variables, there was still marked variation between hospitals with similar characteristics.

Conclusion

We found substantial variation in hospital admissions following a presentation to the ED due to LBP even after controlling by case-mix and hospital characteristics. Given the substantial costs associated with these admissions, our findings indicate the need to investigate sources of variation and to determine instances where the observed variation is warranted or unwarranted.

Supplementary information

The online version contains supplementary material available at 10.1186/s12913-022-08134-8.

Emulating Three Clinical Trials that Compare Stroke Rehabilitation at Inpatient Rehabilitation Facilities to Skilled Nursing Facilities

OBJECTIVE

To inform the design of a potential future randomized controlled trial, we emulated three trials where patient-level outcomes were compared following stroke rehabilitation at Inpatient Rehabilitation Facilities (IRFs) to Skilled Nursing Facilities (SNFs).

DESIGN

Trials were emulated using a 1:1 matched propensity score analysis. The three trials differed as facilities from rehabilitation networks with different case-volumes were compared. Rehabilitation network case-volumes were based on the number of stroke patients that each hospital discharged to each specific IRF or SNF. Trial 1 included 60,529 patients from all networks, trial 2 included 34,444 patients from networks with medium- and large case-volumes (i.e., ≥5 patients), trial 3 included 19,161 patients from networks with large case-volumes (i.e., ≥10 patients). E-values were calculated to estimate the minimum strength that an unmeasured confounder would need to be to nullify the results.

SETTING

A national sample of IRFs and SNFs from across the United States.

PARTICIPANTS

Acute Fee-for-service Medicare stroke patients who received IRF or SNF based rehabilitation.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURE(S)

1-year successful community discharge (home for >30 consecutive days) and all-cause mortality.

RESULTS

Overall, 29,500, 15,156, and 7,450 patients were matched for trials 1, 2 and 3. For 1-year successful community discharge, absolute risk differences for IRF patients were 0.21 (95% CI: 0.20, 0.22), 0.17 (95% CI: 0.16, 0.19), and 0.12 (95% CI: 0.10, 0.14) in trials 1, 2 and 3, respectively. For 1-year all-cause mortality, corresponding risk differences were -0.11 (95% CI: -0.12, -0.11), -0.11 (95% CI: -0.12, -0.09), and -0.08 (95% CI: -0.10, -0.06). E-values indicated that a moderately sized unmeasured confounder, with a relative risk of 1.6 to 2.0 would nullify differences in successful community discharge.

CONCLUSION(S)

IRF patients had superior outcomes, but differences were attenuated when IRFs and SNFs from larger rehabilitation networks were compared. The vulnerability of the findings to unmeasured confounding supports the need for an RCT.

Sex differences in vascular risk factors, in-hospital management, and outcomes of patients with acute ischemic stroke in China

BACKGROUND

Previous assessments of sex differences for patients with acute ischemic stroke were limited in a specific region or population, narrow scope, or small sample size.

METHODS

Patients with acute ischemic stroke hospitalized in the China Stroke Center Alliance hospitals were analyzed. Absolute standardized differences (ASDs) were used to assess sex differences in vascular risk factors, guideline-recommended in-hospital management measures and outcomes, including stroke severity (National Institutes of Health Stroke Scale≥16), death/discharge against medical advice, major adverse cardiovascular events, pneumonia, and disability (modified Rankin Scale≥3).

RESULTS

Of 838,229 patients analyzed, 524351 (62.6%) were men and 313,878 (37.4%) were women. Compared with men, women were older (68.6 vs. 64.7 years), had higher prevalence of hypertension (67.7% vs. 62.4%), diabetes (24.7% vs. 19.5%), and atrial fibrillation (7.1% vs. 4.3%), but lower prevalence of smoking (4.5% vs. 56.6%) and drinking (2.6% vs 35.8%) (ASDs >10%). No sex differences were seen in guideline-directed management measures, indicated by risk-adjusted individual measures and the all-or-null summary measure (34.5% vs 34.9%, ASD = 1.0%). Compared to men, women tended to have strokes that were more severe at presentation (6.5% vs. 4.5%, ASD = 8.8%) and more disabilities at discharge (34.9% vs 30.5%, ASD =9.4%). However, all sex-related differences in outcomes were attenuated to null after risk adjustments (ASDs<2%).

CONCLUSIONS

Compared to male patients, female patients had more vascular risk factors and received similar in-hospital care. They had strokes that were more severe at presentation and more disabilities at discharge, both of which may be explained by worse vascular risk profiles.

Relationship between primary stroke center volume and time to endovascular thrombectomy in acute ischemic stroke

Background and purpose

We investigated whether the annual volume of patients with acute ischemic stroke referred from a primary stroke center (PSC) for endovascular treatment (EVT) is associated with treatment times and functional outcome.

Methods

We used data from the Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands (MR CLEAN) registry (2014–2017). We included patients with acute ischemic stroke of the anterior circulation who were transferred from a PSC to a comprehensive stroke center (CSC) for EVT. We examined the association between EVT referral volume of PSCs and treatment times and functional outcome using multivariable regression modeling. The main outcomes were time from arrival at the PSC to groin puncture (PSC‐door‐to‐groin time), adjusted for estimated ambulance travel times, time from arrival at the CSC to groin puncture (CSC‐door‐to‐groin time), and modified Rankin Scale (mRS) score at 90 days after stroke.

Results

Of the 3637 patients in the registry, 1541 patients (42%) from 65 PSCs were included. Mean age was 71 years (SD ± 13.3), median National Institutes of Health Stroke Scale score was 16 (interquartile range [IQR]: 12–19), and median time from stroke onset to arrival at the PSC was 53 min (IQR: 38–90). Eighty‐three percent had received intravenous thrombolysis. EVT referral volume was not associated with PSC‐door‐to‐groin time (adjusted coefficient: −0.49 min/annual referral, 95% confidence interval [CI]: −1.27 to 0.29), CSC‐door‐to‐groin time (adjusted coefficient: −0.34 min/annual referral, 95% CI: −0.69 to 0.01) or 90‐day mRS score (adjusted common odds ratio: 0.99, 95% CI: 0.96–1.01).

Conclusions

In patients transferred from a PSC for EVT, higher PSC volumes do not seem to translate into better workflow metrics or patient outcome.

Variability in outcomes and quality-of-care indicators across clinics participating in a large smoking-cessation program

BACKGROUND

The effectiveness of care for substance-related problems varies across providers. Best-known treatments are rarely universally applied, and various process differences can affect participant outcomes. Measuring and understanding this variability can suggest changes that will improve system performance.

METHODS

We measure variability in 7-day cigarette abstinence at a six-month follow-up; return for a second clinical visit; and receipt of combination nicotine replacement therapy across 223 primary care clinics participating in the Smoking Treatment for Ontario Patients program, a large smoking cessation initiative in Ontario, Canada. We include 41,992 enrolments from April 11, 2016 and May 31, 2019. We risk adjust for demographic and clinical case-mix differences and characterize variability using funnel plots and measures based on clinic-level variance explained. The abstinence outcome is missing for 38% of participants. We adjust for missingness using multiple imputation and inverse probability weighting.

RESULTS

Abstinence was achieved by 28.0% (95% CI = 27.5%-28.5%) of participants, 63.2% (62.8%-63.7%) received combination NRT, and 72.9% (72.4%-73.3%) returned for a second clinical visit. Variability was moderate for abstinence (median odds ratio (MOR) = 1.16) and pronounced for return visit (MOR = 1.29) and combination therapy (MOR = 1.89).

CONCLUSION

Outcomes and processes vary significantly across clinics within a program with shared guidelines and standards. Differences across providers may be greater in other contexts. Results underscore the importance of measuring and understanding variability, and of ongoing maintenance and improvement. The existence of high outliers holds out the possibility of identifying practices that might be more widely adopted.

Surgeon experience contributes to improved outcomes in pancreatoduodenectomies at high risk for fistula development

BACKGROUND

Pancreatoduodenectomies at high risk for clinically relevant pancreatic fistula are uncommon, yet intimidating, situations. In such scenarios, the impact of individual surgeon experience on outcomes is poorly understood.

METHODS

The fistula risk score was applied to identify high-risk patients (fistula risk score 7-10) from 7,706 pancreatoduodenectomies performed at 18 international institutions (2003-2020). For each case, surgeon pancreatoduodenectomy career volume and years of practice were linked to intraoperative fistula mitigation strategy adoption and outcomes. Consequently, best operative approaches for clinically relevant pancreatic fistula prevention and best performer profiles were identified through multivariable analysis models.

RESULTS

Eight hundred and thirty high-risk pancreatoduodenectomies, performed by 64 surgeons, displayed an overall clinically relevant pancreatic fistula rate of 33.7%. Clinically relevant pancreatic fistula rates decreased with escalating surgeon career pancreatoduodenectomy (-49.7%) and career length (-41.2%; both P < .001), as did transfusion and reoperation rates, postoperative morbidity index, and duration of stay. Great experience (≥400 pancreatoduodenectomies performed or ≥21-year-long career) was a significant predictor of clinically relevant pancreatic fistula prevention (odds ratio 0.52, 95% confidence interval 0.35-0.76) and was more often associated with pancreatojejunostomy reconstruction and prophylactic octreotide omission, which were both independently associated with clinically relevant pancreatic fistula reduction. A risk-adjusted performance analysis also correlated with experience. Moreover, minimizing blood loss (≤400 mL) significantly contributed to clinically relevant pancreatic fistula prevention (odds ratio 0.40, 95% confidence interval 0.22-0.74).

CONCLUSION

Surgeon experience is a key contributor to achieve better outcomes after high-risk pancreatoduodenectomy. Surgeons can improve their performance in these challenging situations by employing pancreatojejunostomy reconstruction, omitting prophylactic octreotide, and minimizing blood loss.

A pilot study to assess the learning environment and use of reliability enhancing work practices in VHA cardiac catheterization laboratories

INTRODUCTION

A learning health system (LHS) harnesses data and analytics to learn from clinical encounters to implement the best care with high reliability. The 81 Veterans Health Administration (VHA) cardiac catheterization laboratories (cath lab) are a model LHS. The quality and safety of coronary procedures are monitored and reported by the Clinical Assessment, Reporting and Tracking (CART) Program, which has identified variation in care across cath labs. This variation may be due to underappreciated aspects of LHSs, the learning environment and reliability enhancing work practices (REWPs). Learning environments are the educational approaches, context, and settings in which learning occurs. REWPs are the organizational practices found in high reliability organizations. High learning environments and use of REWPs are associated with improved outcomes. This study assessed the learning environments and use of REWPs in VHA cath labs to examine factors supportive of learning and high reliability.

METHODS

In 2018, the learning organization survey-27 and the REWP survey were administered to 732 cath lab staff. Factor analysis and linear models were computed. Unit-level analyses and site ranking (high, low) were conducted on cath labs with >40% response rate using Bayesian methods.

RESULTS

Surveys from 40% of cath lab staff (n = 294) at 84% of cath labs (n = 68) were included. Learning environment and REWP strengths across cath labs include the presence of training programs, openness to new ideas, and respectful interaction. Learning environment and REWP gaps include lack of structured knowledge transfer (eg, checklists) and low use of forums for improvement. Survey dimensions matched established factor structures and demonstrated high reliability (Cronbach's alpha >.76). Unit-level analyses were conducted for 29 cath labs. One ranked as high and four as low learning environments.

CONCLUSIONS

This work demonstrates an approach to assess local learning environments and use of REWPs, providing insights for systems working to become a LHS.

Bayesian profiling for cost with zeros to decompose total cost into probability of cost and mean nonzero cost

Cost of health care can vary substantially across hospitals, centers, or providers. Data from electronic health records provide information for studying patterns of cost variation and identifying high or low cost centers. Cost data often include zero values when patients receive no care, and joint two-part models have been developed for clustered cost data with zeros. Standard methods for center comparisons, sometimes called profiling, can use these methods to incorporate zero values into total cost. However, zero costs also provide opportunities to further examine sources of cost variation and outliers. For example, a hospital may have high (or low) cost due to frequency of nonzero cost, amount of nonzero cost, or a combination of those. We give methods for decomposing hospital differences in total cost with zeros into components for probability of use (i.e., of nonzero cost) and for cost of use (mean of nonzero cost). The components multiply to total cost and quantify components on the same easily interpreted multiplicative scales. The methods are based on Bayesian hierarchical models and counterfactual arguments, with Markov chain Monte Carlo estimation. We used simulated data to illustrate use, interpretation, and visualization of the methods in diverse situations, and applied the methods to 30,024 patients at 57 US Veterans Administration hospitals to characterize outlier hospitals in one year cost of inpatient care following a cardiac procedure. Twenty eight percent of patients had zero cost. These methods are useful in providing insight into cost variation and outliers for planning future studies or interventions.

Impact of Missing Stroke Severity Data on the Accuracy of Hospital Ischemic Stroke Mortality Profiling

BACKGROUND

The Centers for Medicare and Medicaid Services have proposed 30-day ischemic stroke risk-standardized mortality rates that include adjustment for stroke severity using the National Institute of Health Stroke Scale (NIHSS), which is often undocumented. We used simulations to quantify the effect of missing NIHSS data on the accuracy of hospital-level ischemic stroke risk-standardized mortality rate profiling for 100 hypothetical hospitals with different case volumes.

METHODS AND RESULTS

We generated simulated data sets of patients with NIHSS scores and other predictors of 30-day mortality based on empirical analysis of data from 7654 patients with ischemic stroke in the Michigan Stroke Registry. We assigned and rank-ordered a true (known) 30-day mortality rate to each hospital in the simulated data sets of N=100 hospitals with either low (100 patients with stroke), medium (300), or high (500) case volumes. We then estimated and rank-ordered 30-day risk-standardized mortality rates for the N=100 hospitals in each simulated data set using hierarchical logistic regression models. In each data set, we systematically varied the rate of missing NIHSS data and whether missing NIHSS data was independent (missing completely at random) or dependent (missing not at random) on the NIHSS score. With no missing NIHSS data, the Spearman correlation between the true hospital performance rank order assigned during data set generation and the estimated 30-day risk-standardized mortality rate rank order was 0.72, 0.88, and 0.91 in low, medium, and high volume hospitals, respectively. However, this fell to as low as 0.50, 0.71, and 0.79 as missing NIHSS data reached 70%.

CONCLUSIONS

Missing NIHSS data had substantial detrimental effects on the accuracy of profiling of ischemic stroke mortality, especially in lower volume hospitals. Even with complete NIHSS documentation, significant limitations in ischemic stroke mortality profiling remain.

Causal Mediation Analysis for Standardized Mortality Ratios

Indirectly standardized mortality ratios (SMR) are often used to compare patient outcomes between health care providers as indicators of quality of care. Observed differences in the outcomes raise the question of whether these could be causally attributable to earlier processes or outcomes in the pathway of care that the patients received. Such pathways can be naturally addressed in a causal mediation analysis framework. Adopting causal mediation models allows the total provider effect on outcome to be decomposed into direct and indirect (mediated) effects. This in turn enables quantification of the improvement in patient outcomes due to a hypothetical intervention on the mediator. We formulate the effect decomposition for the indirectly standardized SMR when comparing to a health care system-wide average performance, propose novel model-based and semiparametric estimators for the decomposition, study the properties of these through simulations, and demonstrate their use through application to Ontario kidney cancer data.

Performance Characteristics of Profiling Methods and the Impact of Inadequate Case-mix Adjustment

Profiling or evaluation of health care providers involves the application of statistical models to compare each provider's performance with respect to a patient outcome, such as unplanned 30-day hospital readmission, adjusted for patient case-mix characteristics. The nationally adopted method is based on random effects (RE) hierarchical logistic regression models. Although RE models are sensible for modeling hierarchical data, novel high dimensional fixed effects (FE) models have been proposed which may be well-suited for the objective of identifying sub-standard performance. However, there are limited comparative studies. Thus, we examine their relative performance, including the impact of inadequate case-mix adjustment.

Impact of Case-Mix Measurement Error on Estimation and Inference in Profiling of Health Care Providers

Profiling analysis aims to evaluate health care providers by modeling each provider's performance with respect to a patient outcome, such as unplanned hospital readmission. High-dimensional regression models are used in profiling to risk-adjust for patient case-mix covariates. Case-mix covariates typically ascertained from administrative databases are inherently error-prone. We examine the impact of case-mix measurement error (ME) on profiling models. The results show that even though the models' coefficient estimates are biased, this does not affect the estimation of standardized readmission ratio (SRR). However, ME leads to increased variation in SRR estimates and degrades the ability to identify under-performing providers.

Investigating Risk Adjustment Methods for Health Care Provider Profiling When Observations are Scarce or Events Rare

Background:

When profiling health care providers, adjustment for case-mix is essential. However, conventional risk adjustment methods may perform poorly, especially when provider volumes are small or events rare. Propensity score (PS) methods, commonly used in observational studies of binary treatments, have been shown to perform well when the amount of observations and/or events are low and can be extended to a multiple provider setting. The objective of this study was to evaluate the performance of different risk adjustment methods when profiling multiple health care providers that perform highly protocolized procedures, such as coronary artery bypass grafting.

Methods:

In a simulation study, provider effects estimated using PS adjustment, PS weighting, PS matching, and multivariable logistic regression were compared in terms of bias, coverage and mean squared error (MSE) when varying the event rate, sample size, provider volumes, and number of providers. An empirical example from the field of cardiac surgery was used to demonstrate the different methods.

Results:

Overall, PS adjustment, PS weighting, and logistic regression resulted in provider effects with low amounts of bias and good coverage. The PS matching and PS weighting with trimming led to biased effects and high MSE across several scenarios. Moreover, PS matching is not practical to implement when the number of providers surpasses three.

Conclusions:

None of the PS methods clearly outperformed logistic regression, except when sample sizes were relatively small. Propensity score matching performed worse than the other PS methods considered.

Factors associated with hospital emergency readmission and mortality rates in patients with heart failure or chronic obstructive pulmonary disease: a national observational study

Background

Heart failure (HF) and chronic obstructive pulmonary disease (COPD) lead to unplanned hospital activity, but our understanding of what drives this is incomplete.

Objectives

To model patient, primary care and hospital factors associated with readmission and mortality for patients with HF and COPD, to assess the statistical performance of post-discharge emergency department (ED) attendance compared with readmission metrics and to compare all the results for the two conditions.

Design

Observational study.

Setting

English NHS.

Participants

All patients admitted to acute non-specialist hospitals as an emergency for HF or COPD.

Interventions

None.

Main outcome measures

One-year mortality and 30-day emergency readmission following the patient’s first unplanned admission (‘index admission’) for HF or COPD.

Data sources

Patient-level data from Hospital Episodes Statistics were combined with publicly available practice- and hospital-level data on performance, patient and staff experience and rehabilitation programme website information.

Results

One-year mortality rates were 39.6% for HF and 24.1% for COPD and 30-day readmission rates were 19.8% for HF and 16.5% for COPD. Most patients were elderly with multiple comorbidities. Patient factors predicting mortality included older age, male sex, white ethnicity, prior missed outpatient appointments, (long) index length of hospital stay (LOS) and several comorbidities. Older age, missed appointments, (short) LOS and comorbidities also predicted readmission. Of the practice and hospital factors we considered, only more doctors per 10 beds [odds ratio (OR) 0.95 per doctor;p < 0.001] was significant for both cohorts for mortality, with staff recommending to friends and family (OR 0.80 per unit increase;p < 0.001) and number of general practitioners (GPs) per 1000 patients (OR 0.89 per extra GP;p = 0.004) important for COPD. For readmission, only hospital size [OR per 100 beds = 2.16, 95% confidence interval (CI) 1.34 to 3.48 for HF, and 2.27, 95% CI 1.40 to 3.66 for COPD] and doctors per 10 beds (OR 0.98;p < 0.001) were significantly associated. Some factors, such as comorbidities, varied in importance depending on the readmission diagnosis. ED visits were common after the index discharge, with 75% resulting in admission. Many predictors of admission at this visit were as for readmission minus comorbidities and plus attendance outside the day shift and numbers of admissions that hour. Hospital-level rates for ED attendance varied much more than those for readmission, but the omega statistics favoured them as a performance indicator.

Limitations

Data lacked direct information on disease severity and ED attendance reasons; NHS surveys were not specific to HF or COPD patients; and some data sets were aggregated.

Conclusions

Following an index admission for HF or COPD, older age, prior missed outpatient appointments, LOS and many comorbidities predict both mortality and readmission. Of the aggregated practice and hospital information, only doctors per bed and numbers of hospital beds were strongly associated with either outcome (both negatively). The 30-day ED visits and diagnosis-specific readmission rates seem to be useful performance indicators.

Future work

Hospital variations in ED visits could be investigated using existing data despite coding limitations. Primary care management could be explored using individual-level linked databases.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Outlier classification performance of risk adjustment methods when profiling multiple providers

BACKGROUND

When profiling multiple health care providers, adjustment for case-mix is essential to accurately classify the quality of providers. Unfortunately, misclassification of provider performance is not uncommon and can have grave implications. Propensity score (PS) methods have been proposed as viable alternatives to conventional multivariable regression. The objective was to assess the outlier classification performance of risk adjustment methods when profiling multiple providers.

METHODS

In a simulation study based on empirical data, the classification performance of logistic regression (fixed and random effects), PS adjustment, and three PS weighting methods was evaluated when varying parameters such as the number of providers, the average incidence of the outcome, and the percentage of outliers. Traditional classification accuracy measures were considered, including sensitivity and specificity.

RESULTS

Fixed effects logistic regression consistently had the highest sensitivity and negative predictive value, yet a low specificity and positive predictive value. Of the random effects methods, PS adjustment and random effects logistic regression performed equally well or better than all the remaining PS methods for all classification accuracy measures across the studied scenarios.

CONCLUSIONS

Of the evaluated PS methods, only PS adjustment can be considered a viable alternative to random effects logistic regression when profiling multiple providers in different scenarios.

Should interventions to reduce variation in care quality target doctors or hospitals?

•

Performance management initiatives are increasingly targeting individual doctors as well as hospitals.

•

Less than 25% of variation in clinical outcomes is attributable to providers.

•

More variation in clinical outcomes is associated with doctors than with hospitals.

•

Performance estimates for individual doctors are unreliable due to small samples.

Interventions to reduce variation in care quality are increasingly targeted at both individual doctors and the organisations in which they work. Concerns remain about the scope and consequences for such performance management, the relative contribution of individuals and organisations to observed variation, and whether performance can be measured reliably.

This study explores these issues in the context of the English National Health Service by analysing comprehensive administrative data for all patients treated for four clinical conditions (acute myocardial infarction, hip fracture, pneumonia, ischemic stroke) and two surgical procedures (coronary artery bypass, hip replacement) during April 2010–February 2013. Performance indicators are defined as 30-day mortality, 28-day emergency readmission and inpatient length of stay. Three-level hierarchical generalised linear mixed models are estimated to attribute variation in case-mix adjusted indicators to individual doctors and hospital organisations.

Except for length of stay after hip replacement, no more than 11% of variation in case-mix adjusted performance indicators can be attributed to doctors and organisations with the rest reflecting random chance and unobserved patient factors. Doctor variation exceeds hospital variation by a factor of 1.2 or more. However, identifying poor performance amongst doctors is hampered by insufficient numbers of cases per doctor to reliably estimate their individual performances. Policy makers and regulators should therefore be cautious when targeting individual doctors in performance improvement initiatives.

Multidimensional performance assessment of public sector organisations using dominance criteria

Public sector organisations pursue multiple objectives and serve a number of stakeholders. But stakeholders are rarely explicit about the valuations they attach to different objectives, nor are these valuations likely to be identical. This complicates the assessment of their performance because no single set of weights can be chosen legitimately to aggregate outputs into unidimensional composite scores. We propose the use of dominance criteria in a multidimensional performance assessment framework to identify best practice and poor performance under relatively weak assumptions about stakeholders' preferences. We use as an example providers of hip replacement surgery in the English National Health Service and estimate multivariate multilevel models to study their performance in terms of length of stay, readmission rates, post-operative patient-reported health status and waiting time. We find substantial correlation between objectives and demonstrate that ignoring the correlation can lead to incorrect assessments of performance.

Elective hospital admissions: secondary data analysis and modelling with an emphasis on policies to moderate growth

Background

The English NHS faces financial pressures that may render the growth rates of elective admissions seen between 2001/2 and 2011/12 unsustainable. A better understanding of admissions growth, and the influence of policy, are needed to minimise the impact on health gain for patients.

Objectives

This project had several objectives: (1) to better understand the determinants of elective activity and policy to moderate growth at minimum health loss for patients; (2) to build a rich data set integrating health, practice and local area data to study general practitioner (GP) referrals and resulting admissions; (3) to predict patients whose treatment is unlikely to be cost-effective using patient-reported outcomes and to examine variation in provider performance; and (4) to study how policies that aim to reduce elective admissions may change demand for emergency care. The main drivers of elective admissions growth have increased either supply of or demand for care, and could include, for example, technical innovations or increased awareness of treatment benefits. Of the factors studied, neither system reform nor population ageing appears to be a key driver. The introduction of the prospective payment tariff ‘Payment by Results’ appears to have led to primary care trusts (PCTs) having increasingly similar lengths of stay. In deprived areas, increasing GP supply appears to moderate elective admissions. Reducing the incidence of single-handed practices tends to reduce referrals and admissions. Policies to reduce referrals are likely to reduce admissions but treatments may be particularly reduced in the lowest referring practices, in which resulting health loss may be greatest. In this model, per full-time equivalent, female and highly experienced GPs identify more patients admitted by specialists.

Results

It appears from our studies that some patient characteristics are associated with not achieving sufficient patient gain to warrant cost-effective treatment. The introduction of independent sector treatment centres is estimated to have caused an increase in emergency activity rates at local PCTs. The explanations offered for increasing elective admissions indicate that they are manageable by health policy.

Conclusions

Further work is required to understand some of the results identified, such as whether or not high-volume Clinical Commissioning Groups are fulfilling unmet need; why some practices refer at low rates relative to admissions; why the period effect, which results from factors that equally affect all in the study at a point in time, dominates in the age–period–cohort analysis; and exactly how the emergency and elective sections of hospital treatment interact. This project relies on the analysis of secondary data. This type of research does not easily facilitate the important input of clinical experts or service users. It would be beneficial if other methods, including surveys and consultation with key stakeholders, could be incorporated into future research now that we have uncovered important questions.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Quality of Quality Measurement

Background

The validity of basing healthcare reimbursement policy on pay-for-performance is grounded in the accuracy of performance measurement.

Methods

Monte Carlo simulation was used to examine the accuracy of performance profiling as a function of statistical methodology, case volume, and the extent to which hospital or physician performance deviates from the average.

Results

There is extensive variation in the true-positive rate and false discovery rate as a function of model specification, hospital quality, and hospital case volume. Hierarchical and nonhierarchical modeling are both highly accurate at very high case volumes for very low-quality hospitals. At equivalent case volumes and hospital effect sizes, the true-positive rate is higher for nonhierarchical modeling than for hierarchical modeling, but the false discovery rate is generally much lower for hierarchical modeling than for nonhierarchical modeling. At low hospital case volumes (200) that are typical for many procedures, and for hospitals with twice the rate of death or major complications for patients undergoing isolated coronary artery bypass graft surgery at the average hospital, hierarchical modeling missed 90.6% of low-quality hospitals, whereas nonhierarchical modeling missed 65.3%. However, at low case volumes, 38.9% of hospitals classified as low-quality outliers using nonhierarchical modeling were actually average quality, compared to 5.3% using hierarchical modeling.

Conclusions

Nonhierarchical modeling frequently misclassified average-quality hospitals as low quality. Hierarchical modeling commonly misclassified low-quality hospitals as average. Assuming that the consequences of misclassifying an average-quality hospital as low quality outweigh the consequences of misclassifying a low-quality hospital as average, hierarchical modeling may be the better choice for quality measurement.

Optimal Statistical Decisions for Hospital Report Cards

Purpose. Hospital report cards provide information designed to help patients and providers to make decisions. The purpose of this study was to place the design of hospital report cards into a decision-theoretic framework. The authors’ objectives were 2-fold: 1st, to determine what the choice of significance level implies about the relative value of the different types of misclassifications that can arise. Second, to determine optimal significance levels for specific cost functions describing the relative costs associated with different types of misclassifications. Methods. Using a previously published theoretical model for hospital mortality, the authors computed false positive (i.e., falsely classified as providing poor-quality care) and false negative (falsely classified as providing good-quality care) rates. First, they determined the cost functions for false negatives and false positives that are implicitly associated with the use of significance levels of 0.05 and 0.01 for identifying hospitals with higher than average mortality. Second, they determined the levels of statistical significance that should be chosen to minimize predefined cost functions, thus minimizing costs associated with misclassifying hospitals. Results. The lower the statistical significance level required for identifying hospitals with higher than average mortality, the lower the implicit cost of false negatives compared to false positives. For a given significance level, the greater the number of patients treated at each hospital or the greater the proportion of truly poorly performing hospitals, the lower the value of the implicit cost incurred by a false negative compared to that for a false positive. For cost functions that put a high relative penalty on false negatives compared to false positives, the use of significance levels of 0.05 or 0.01 does not result in optimal decisions across expected number of patients treated at each hospital or proportions of truly poor-quality care. Conclusions. Hospital report cards that use significance levels of either 0.05 or 0.01 to identify hospitals that have statistically significantly higher than average mortality make implicit assumptions about cost functions, and the values of the optimal cost function vary across scenarios.

The Importance of Integrating Clinical Relevance and Statistical Significance in the Assessment of Quality of Care--Illustrated Using the Swedish Stroke Register

Background

When profiling hospital performance, quality inicators are commonly evaluated through hospital-specific adjusted means with confidence intervals. When identifying deviations from a norm, large hospitals can have statistically significant results even for clinically irrelevant deviations while important deviations in small hospitals can remain undiscovered. We have used data from the Swedish Stroke Register (Riksstroke) to illustrate the properties of a benchmarking method that integrates considerations of both clinical relevance and level of statistical significance.

Methods

The performance measure used was case-mix adjusted risk of death or dependency in activities of daily living within 3 months after stroke. A hospital was labeled as having outlying performance if its case-mix adjusted risk exceeded a benchmark value with a specified statistical confidence level. The benchmark was expressed relative to the population risk and should reflect the clinically relevant deviation that is to be detected. A simulation study based on Riksstroke patient data from 2008–2009 was performed to investigate the effect of the choice of the statistical confidence level and benchmark value on the diagnostic properties of the method.

Results

Simulations were based on 18,309 patients in 76 hospitals. The widely used setting, comparing 95% confidence intervals to the national average, resulted in low sensitivity (0.252) and high specificity (0.991). There were large variations in sensitivity and specificity for different requirements of statistical confidence. Lowering statistical confidence improved sensitivity with a relatively smaller loss of specificity. Variations due to different benchmark values were smaller, especially for sensitivity. This allows the choice of a clinically relevant benchmark to be driven by clinical factors without major concerns about sufficiently reliable evidence.

Conclusions

The study emphasizes the importance of combining clinical relevance and level of statistical confidence when profiling hospital performance. To guide the decision process a web-based tool that gives ROC-curves for different scenarios is provided.

On the practice of ignoring center-patient interactions in evaluating hospital performance

We evaluate the performance of medical centers based on a continuous or binary patient outcome (e.g., 30‐day mortality). Common practice adjusts for differences in patient mix through outcome regression models, which include patient‐specific baseline covariates (e.g., age and disease stage) besides center effects. Because a large number of centers may need to be evaluated, the typical model postulates that the effect of a center on outcome is constant over patient characteristics. This may be violated, for example, when some centers are specialized in children or geriatric patients. Including interactions between certain patient characteristics and the many fixed center effects in the model increases the risk for overfitting, however, and could imply a loss of power for detecting centers with deviating mortality. Therefore, we assess how the common practice of ignoring such interactions impacts the bias and precision of directly and indirectly standardized risks. The reassuring conclusion is that the common practice of working with the main effects of a center has minor impact on hospital evaluation, unless some centers actually perform substantially better on a specific group of patients and there is strong confounding through the corresponding patient characteristic. The bias is then driven by an interplay of the relative center size, the overlap between covariate distributions, and the magnitude of the interaction effect. Interestingly, the bias on indirectly standardized risks is smaller than on directly standardized risks. We illustrate our findings by simulation and in an analysis of 30‐day mortality on Riksstroke. © 2015 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

Provider Differences in Use of Implanted Ports in Older Adults With Cancer

BACKGROUND

Identifying unwarranted variation in health care can highlight opportunities to reduce harm. One often discretionary process in oncology is use of implanted ports to administer intravenous chemotherapy. While there are benefits, ports carry risks. This study's objective was to assess provider-driven variation in port use among cancer patients receiving chemotherapy.

RESEARCH DESIGN

Retrospective assessment using population-based SEER-Medicare data to assess differences in port use across health care providers of older adults with cancer. Participants included over 18,000 patients ages 66 and older diagnosed with breast, colorectal, lung, or pancreatic cancer in 2005-2007, treated by approximately 2900 providers. We identified port use for patients receiving treatment from hospital outpatient facilities versus physicians' offices. Our main analysis assessed the likelihood of a patient receiving a port given port use by the provider's last patient. For a subset of high-use providers, we examined individual provider-level variation by estimating the risk-adjusted likelihood of insertion.

RESULTS

Patients receiving chemotherapy in hospital outpatient facilities were significantly less likely to receive a port than those treated in physicians' offices, with adjusted odds ratios (AOR) varying from 0.50 to 0.75 across cancer sites. Implanting a port was associated with increased likelihood of port insertion in the provider's next patient (AOR varied from 1.71 to 2.25). Significant between-provider variation was found among providers with at least 10 patients.

CONCLUSIONS

Our findings support the idea that there is provider-driven variation in the use of ports for chemotherapy administration. This variation highlights an opportunity to standardize practice and reduce unnecessary use.

Stress Testing After Percutaneous Coronary Intervention in the Veterans Affairs HealthCare System: Insights from the Veterans Affairs Clinical Assessment, Reporting, and Tracking Program

BACKGROUND

Stress testing after percutaneous coronary intervention (PCI) in fee-for-service settings is common and rates vary by hospital. Rates of stress testing after PCI within integrated healthcare systems, such as the Veterans Affairs (VA) are unknown.

METHODS AND RESULTS

We evaluated all VA patients who underwent PCI from October 2007 through June 2010. To avoid the influence of Medicare eligibility on rates of stress testing use in the VA, we excluded Medicare eligible patients during the follow-up period. Hospital-level variation in risk-standardized rates of stress testing and the association with 1-year mortality and myocardial infarction was determined from Markov chain Monte Carlo methods. Among 10 293 patients undergoing PCI at 55 VA hospitals, 2239 (21.8%) had a stress test performed within 1 year of PCI and 3902 (37.9%) within 2 years. Most stress tests after PCI were performed with nuclear imaging (79.8%). The hospital-level risk-standardized rate of stress testing differed significantly from the average at 14 hospitals, with 8 (14.5%) hospitals significantly below and 6 (10.9%) hospitals significantly above the average stress testing rate. Hospital-level risk-standardized stress testing rates were not significantly correlated with risk-standardized mortality (Spearman ρ=-0.24; P=0.08) or myocardial infarction rates (Spearman ρ=0.20; P=0.14).

CONCLUSIONS

In the VA, nearly 40% of patients underwent stress testing in the 2 years after PCI, which is a third less than published studies from other healthcare systems. However, stress testing rates varied across VA hospitals, suggesting opportunities to optimize the use of stress testing are still present in integrated healthcare systems.

Facility-Level Variation in Hospitalization, Mortality, and Costs in the 30 Days After Percutaneous Coronary Intervention

Background—

Policies to reduce unnecessary hospitalizations after percutaneous coronary intervention (PCI) are intended to improve healthcare value by reducing costs while maintaining patient outcomes. Whether facility-level hospitalization rates after PCI are associated with cost of care is unknown.

Methods and Results—

We studied 32 080 patients who received PCI at any 1 of 62 Veterans Affairs hospitals from 2008 to 2011. We identified facility outliers for 30-day risk-standardized hospitalization, mortality, and cost. Compared with the risk-standardized average, 2 hospitals (3.2%) had a lower-than-expected hospitalization rate, and 2 hospitals (3.2%) had a higher-than-expected hospitalization rate. We observed no statistically significant variation in facility-level risk-standardized mortality. The facility-level unadjusted median per patient 30-day total cost was $23 820 (interquartile range, $19 604–$29 958). Compared with the risk-standardized average, 17 hospitals (27.4%) had lower-than-expected costs, and 14 hospitals (22.6%) had higher-than-expected costs. At the facility level, the index PCI accounted for 83.1% of the total cost (range, 60.3%–92.2%), whereas hospitalization after PCI accounted for only 5.8% (range, 2.0%–12.7%) of the 30-day total cost. Facilities with higher hospitalization rates were not more expensive (Spearman ρ=0.16; 95% confidence interval, −0.09 to 0.39; P =0.21).

Conclusions—

In this national study, hospitalizations in the 30 day after PCI accounted for only 5.8% of 30-day cost, and facility-level cost was not correlated with hospitalization rates. This challenges the focus on reducing hospitalizations after PCI as an effective means of improving healthcare value. Opportunities remain to improve PCI value by reducing the variation in total cost of PCI without compromising patient outcomes.

A primer on using shrinkage to compare in-hospital mortality between centers

Outcomes of cardiothoracic surgery are usually compared among hospitals or physicians by reporting the frequency of in-hospital mortality. Although there is agreement that these frequencies should be adjusted for case mix, there remains uncertainty about the value of using a statistical model that represents hospitals as random effects as opposed to the conventional approach of fixed effects. For years, the Northern New England Cardiovascular Disease Study Group has compared in-hospital mortality after coronary artery bypass graft surgery among centers using a fixed effects approach. An alternative method using random effects has become increasingly popular, and is the method used by cardiothoracic surgery registries such as the Massachusetts Data Analysis Center. The purpose of this report is to provide a short background on fixed versus random effects modeling, describe the use of shrinkage estimators including empirical Bayes, and illustrate them using data from the Northern New England Cardiovascular Disease Study Group. We conclude that both are acceptable approaches to hospital profiling if done in combination with appropriate risk adjustment.

Comparing colon cancer outcomes: The impact of low hospital case volume and case-mix adjustment

OBJECTIVE

When comparing performance across hospitals it is essential to consider the noise caused by low hospital case volume and to perform adequate case-mix adjustment. We aimed to quantify the role of noise and case-mix adjustment on standardized postoperative mortality and anastomotic leakage (AL) rates.

METHODS

We studied 13,120 patients who underwent colon cancer resection in 85 Dutch hospitals. We addressed differences between hospitals in postoperative mortality and AL, using fixed (ignoring noise) and random effects (incorporating noise) logistic regression models with general and additional, disease specific, case-mix adjustment.

RESULTS

Adding disease specific variables improved the performance of the case-mix adjustment models for postoperative mortality (c-statistic increased from 0.77 to 0.81). The overall variation in standardized mortality ratios was similar, but some individual hospitals changed considerably. For the standardized AL rates the performance of the adjustment models was poor (c-statistic 0.59 and 0.60) and overall variation was small. Most of the observed variation between hospitals was actually noise.

CONCLUSION

Noise had a larger effect on hospital performance than extended case-mix adjustment, although some individual hospital outcome rates were affected by more detailed case-mix adjustment. To compare outcomes between hospitals it is crucial to consider noise due to low hospital case volume with a random effects model.

Clinical outcomes with alternative dosing strategies for piperacillin/tazobactam: a systematic review and meta-analysis

Objectives

A better dosing strategy can improve clinical outcomes for patients. We sought to compare the extended or continuous infusion with conventional intermittent infusion of piperacillin/tazobactam, investigating which approach is better and worthy of recommendation for clinical use.

Methods

Articles were gathered from PubMed, Web of Science, ProQuest, Science Direct, Cochrane, two Chinese literature databases (CNKI, Wan Fang Data) and related ICAAC and ACCP conferences. Randomized controlled and observational studies that compared extended or continuous infusion with conventional intermittent infusion of piperacillin/tazobactam were identified from the databases above and analyzed. Two reviewers independently extracted and investigated the data. A meta-analysis was performed using Revman 5.2 software. The quality of each study was assessed. Sensitivity analysis and publication bias were evaluated.

Results

Five randomized controlled trials and nine observational studies were included in this study. All included studies had high quality and no publication bias was found. Compared to the conventional intermittent infusion approach, the extended or continuous infusion group had a significantly higher clinical cure rate (OR 1.88, 95% CI 1.29-2.73, P = 0.0009) and a lower mortality rate (OR 0.67, 95% CI 0.50-0.89, P = 0.005). No statistical difference was observed for bacteriologic cure (OR 1.40, 95% CI 0.82-2.37, P = 0.22) between the two dosing regimens. The sensitivity analysis showed the results were stable.

Conclusions

Our systematic review and meta-analysis suggested that the extended or continuous infusion strategy of piperacillin/tazobactam should be recommended for clinical use considering its higher clinical cure rate and lower mortality rate in comparison with conventional intermittent strategy. Data from this study could be extrapolated for other β-lactam antimicrobials. Therefore, this dosing strategy could be considered in clinical practice.

Can valid and practical risk-prediction or casemix adjustment models, including adjustment for comorbidity, be generated from English hospital administrative data (Hospital Episode Statistics)? A national observational study

Background

NHS hospitals collect a wealth of administrative data covering accident and emergency (A&E) department attendances, inpatient and day case activity, and outpatient appointments. Such data are increasingly being used to compare units and services, but adjusting for risk is difficult.

Objectives

To derive robust risk-adjustment models for various patient groups, including those admitted for heart failure (HF), acute myocardial infarction, colorectal and orthopaedic surgery, and outcomes adjusting for available patient factors such as comorbidity, using England’s Hospital Episode Statistics (HES) data. To assess if more sophisticated statistical methods based on machine learning such as artificial neural networks (ANNs) outperform traditional logistic regression (LR) for risk prediction. To update and assess for the NHS the Charlson index for comorbidity. To assess the usefulness of outpatient data for these models.

Main outcome measures

Mortality, readmission, return to theatre, outpatient non-attendance. For HF patients we considered various readmission measures such as diagnosis-specific and total within a year.

Methods

We systematically reviewed studies comparing two or more comorbidity indices. Logistic regression, ANNs, support vector machines and random forests were compared for mortality and readmission. Models were assessed using discrimination and calibration statistics. Competing risks proportional hazards regression and various count models were used for future admissions and bed-days.

Results

Our systematic review and empirical analysis suggested that for general purposes comorbidity is currently best described by the set of 30 Elixhauser comorbidities plus dementia. Model discrimination was often high for mortality and poor, or at best moderate, for other outcomes, for examplec = 0.62 for readmission andc = 0.73 for death following stroke. Calibration was often good for procedure groups but poorer for diagnosis groups, with overprediction of low risk a common cause. The machine learning methods we investigated offered little beyond LR for their greater complexity and implementation difficulties. For HF, some patient-level predictors differed by primary diagnosis of readmission but not by length of follow-up. Prior non-attendance at outpatient appointments was a useful, strong predictor of readmission. Hospital-level readmission rates for HF did not correlate with readmission rates for non-HF; hospital performance on national audit process measures largely correlated only with HF readmission rates.

Conclusions

Many practical risk-prediction or casemix adjustment models can be generated from HES data using LR, though an extra step is often required for accurate calibration. Including outpatient data in readmission models is useful. The three machine learning methods we assessed added little with these data. Readmission rates for HF patients should be divided by diagnosis on readmission when used for quality improvement.

Future work

As HES data continue to develop and improve in scope and accuracy, they can be used more, for instance A&E records. The return to theatre metric appears promising and could be extended to other index procedures and specialties. While our data did not warrant the testing of a larger number of machine learning methods, databases augmented with physiological and pathology information, for example, might benefit from methods such as boosted trees. Finally, one could apply the HF readmissions analysis to other chronic conditions.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Classification accuracy of claims-based methods for identifying providers failing to meet performance targets

Quality assessment is critical for healthcare reform, but data sources are lacking for measurement of many important healthcare outcomes. With over 49 million people covered by Medicare as of 2010, Medicare claims data offer a potentially valuable source that could be used in targeted health care quality improvement efforts. However, little is known about the operating characteristics of provider profiling methods using claims-based outcome measures that may estimate provider performance with error. Motivated by the example of screening mammography performance, we compared approaches to identifying providers failing to meet guideline targets using Medicare claims data. We used data from the Breast Cancer Surveillance Consortium and linked Medicare claims to compare claims-based and clinical estimates of cancer detection rate. We then demonstrated the performance of claim-based estimates across a broad range of operating characteristics using simulation studies. We found that identification of poor performing providers was extremely sensitive to algorithm specificity, with no approach identifying more than 65% of poor performing providers when claims-based measures had specificity of 0.995 or less. We conclude that claims have the potential to contribute important information on healthcare outcomes to quality improvement efforts. However, to achieve this potential, development of highly accurate claims-based outcome measures should remain a priority.

Fixed effects modelling for provider mortality outcomes: Analysis of the Australia and New Zealand Intensive Care Society (ANZICS) Adult Patient Data-base

BACKGROUND

Risk adjusted mortality for intensive care units (ICU) is usually estimated via logistic regression. Random effects (RE) or hierarchical models have been advocated to estimate provider risk-adjusted mortality on the basis that standard estimators increase false outlier classification. The utility of fixed effects (FE) estimators (separate ICU-specific intercepts) has not been fully explored.

METHODS

Using a cohort from the Australian and New Zealand Intensive Care Society Adult Patient Database, 2009-2010, the model fit of different logistic estimators (FE, random-intercept and random-coefficient) was characterised: Bayesian Information Criterion (BIC; lower values better), receiver-operator characteristic curve area (AUC) and Hosmer-Lemeshow (H-L) statistic. ICU standardised hospital mortality ratios (SMR) and 95%CI were compared between models. ICU site performance (FE), relative to the grand observation-weighted mean (GO-WM) on odds ratio (OR), risk ratio (RR) and probability scales were assessed using model-based average marginal effects (AME).

RESULTS

The data set consisted of 145355 patients in 128 ICUs, years 2009 (47.5%) & 2010 (52.5%), with mean(SD) age 60.9(18.8) years, 56% male and ICU and hospital mortalities of 7.0% and 10.9% respectively. The FE model had a BIC = 64058, AUC = 0.90 and an H-L statistic P-value = 0.22. The best-fitting random-intercept model had a BIC = 64457, AUC = 0.90 and H-L statistic P-value = 0.32 and random-coefficient model, BIC = 64556, AUC = 0.90 and H-L statistic P-value = 0.28. Across ICUs and over years no outliers (SMR 95% CI excluding null-value = 1) were identified and no model difference in SMR spread or 95%CI span was demonstrated. Using AME (OR and RR scale), ICU site-specific estimates diverged from the GO-WM, and the effect spread decreased over calendar years. On the probability scale, a majority of ICUs demonstrated calendar year decrease, but in the for-profit sector, this trend was reversed.

CONCLUSIONS

The FE estimator had model advantage compared with conventional RE models. Using AME, between and over-year ICU site-effects were easily characterised.

HIV quality report cards: impact of case-mix adjustment and statistical methods

BACKGROUND

There will be increasing pressure to publicly report and rank the performance of healthcare systems on human immunodeficiency virus (HIV) quality measures. To inform discussion of public reporting, we evaluated the influence of case-mix adjustment when ranking individual care systems on the viral control quality measure.

METHODS

We used data from the Veterans Health Administration (VHA) HIV Clinical Case Registry and administrative databases to estimate case-mix adjusted viral control for 91 local systems caring for 12 368 patients. We compared results using 2 adjustment methods, the observed-to-expected estimator and the risk-standardized ratio.

RESULTS

Overall, 10 913 patients (88.2%) achieved viral control (viral load ≤400 copies/mL). Prior to case-mix adjustment, system-level viral control ranged from 51% to 100%. Seventeen (19%) systems were labeled as low outliers (performance significantly below the overall mean) and 11 (12%) as high outliers. Adjustment for case mix (patient demographics, comorbidity, CD4 nadir, time on therapy, and income from VHA administrative databases) reduced the number of low outliers by approximately one-third, but results differed by method. The adjustment model had moderate discrimination (c statistic = 0.66), suggesting potential for unadjusted risk when using administrative data to measure case mix.

CONCLUSIONS

Case-mix adjustment affects rankings of care systems on the viral control quality measure. Given the sensitivity of rankings to selection of case-mix adjustment methods-and potential for unadjusted risk when using variables limited to current administrative databases-the HIV care community should explore optimal methods for case-mix adjustment before moving forward with public reporting.

On shrinkage and model extrapolation in the evaluation of clinical center performance

We consider statistical methods for benchmarking clinical centers based on a dichotomous outcome indicator. Borrowing ideas from the causal inference literature, we aim to reveal how the entire study population would have fared under the current care level of each center. To this end, we evaluate direct standardization based on fixed versus random center effects outcome models that incorporate patient-specific baseline covariates to adjust for differential case-mix. We explore fixed effects (FE) regression with Firth correction and normal mixed effects (ME) regression to maintain convergence in the presence of very small centers. Moreover, we study doubly robust FE regression to avoid outcome model extrapolation. Simulation studies show that shrinkage following standard ME modeling can result in substantial power loss relative to the considered alternatives, especially for small centers. Results are consistent with findings in the analysis of 30-day mortality risk following acute stroke across 90 centers in the Swedish Stroke Register.

Classifying hospitals as mortality outliers: logistic versus hierarchical logistic models

The use of hierarchical logistic regression for provider profiling has been recommended due to the clustering of patients within hospitals, but has some associated difficulties. We assess changes in hospital outlier status based on standard logistic versus hierarchical logistic modelling of mortality. The study population consisted of all patients admitted to acute, non-specialist hospitals in England between 2007 and 2011 with a primary diagnosis of acute myocardial infarction, acute cerebrovascular disease or fracture of neck of femur or a primary procedure of coronary artery bypass graft or repair of abdominal aortic aneurysm. We compared standardised mortality ratios (SMRs) from non-hierarchical models with SMRs from hierarchical models, without and with shrinkage estimates of the predicted probabilities (Model 1 and Model 2). The SMRs from standard logistic and hierarchical models were highly statistically significantly correlated (r > 0.91, p = 0.01). More outliers were recorded in the standard logistic regression than hierarchical modelling only when using shrinkage estimates (Model 2): 21 hospitals (out of a cumulative number of 565 pairs of hospitals under study) changed from a low outlier and 8 hospitals changed from a high outlier based on the logistic regression to a not-an-outlier based on shrinkage estimates. Both standard logistic and hierarchical modelling have identified nearly the same hospitals as mortality outliers. The choice of methodological approach should, however, also consider whether the modelling aim is judgment or improvement, as shrinkage may be more appropriate for the former than the latter.

Analytical Problems and Suggestions in the Analysis of Behavioral Economic Demand Curves

Behavioral economic demand curves (Hursh, Raslear, Shurtleff, Bauman, & Simmons, 1988) are innovative approaches to characterize the relationships between consumption of a substance and its price. In this article, we investigate common analytical issues in the use of behavioral economic demand curves, which can cause inconsistent interpretations of demand curves, and then we provide methodological suggestions to address those analytical issues. We first demonstrate that log transformation with different added values for handling zeros changes model parameter estimates dramatically. Second, demand curves are often analyzed using an overparameterized model that results in an inefficient use of the available data and a lack of assessment of the variability among individuals. To address these issues, we apply a nonlinear mixed effects model based on multivariate error structures that has not been used previously to analyze behavioral economic demand curves in the literature. We also propose analytical formulas for the relevant standard errors of derived values such as P max, O max, and elasticity. The proposed model stabilizes the derived values regardless of using different added increments and provides substantially smaller standard errors. We illustrate the data analysis procedure using data from a relative reinforcement efficacy study of simulated marijuana purchasing.

Risk adjustment of ischemic stroke outcomes for comparing hospital performance: a statement for healthcare professionals from the American Heart Association/American Stroke Association

BACKGROUND AND PURPOSE

Stroke is the fourth-leading cause of death and a leading cause of long-term major disability in the United States. Measuring outcomes after stroke has important policy implications. The primary goals of this consensus statement are to (1) review statistical considerations when evaluating models that define hospital performance in providing stroke care; (2) discuss the benefits, limitations, and potential unintended consequences of using various outcome measures when evaluating the quality of ischemic stroke care at the hospital level; (3) summarize the evidence on the role of specific clinical and administrative variables, including patient preferences, in risk-adjusted models of ischemic stroke outcomes; (4) provide recommendations on the minimum list of variables that should be included in risk adjustment of ischemic stroke outcomes for comparisons of quality at the hospital level; and (5) provide recommendations for further research.

METHODS AND RESULTS

This statement gives an overview of statistical considerations for the evaluation of hospital-level outcomes after stroke and provides a systematic review of the literature for the following outcome measures for ischemic stroke at 30 days: functional outcomes, mortality, and readmissions. Data on outcomes after stroke have primarily involved studies conducted at an individual patient level rather than a hospital level. On the basis of the available information, the following factors should be included in all hospital-level risk-adjustment models: age, sex, stroke severity, comorbid conditions, and vascular risk factors. Because stroke severity is the most important prognostic factor for individual patients and appears to be a significant predictor of hospital-level performance for 30-day mortality, inclusion of a stroke severity measure in risk-adjustment models for 30-day outcome measures is recommended. Risk-adjustment models that do not include stroke severity or other recommended variables must provide comparable classification of hospital performance as models that include these variables. Stroke severity and other variables that are included in risk-adjustment models should be standardized across sites, so that their reliability and accuracy are equivalent. There is a pressing need for research in multiple areas to better identify methods and metrics to evaluate outcomes of stroke care.

CONCLUSIONS

There are a number of important methodological challenges in undertaking risk-adjusted outcome comparisons to assess the quality of stroke care in different hospitals. It is important for stakeholders to recognize these challenges and for there to be a concerted approach to improving the methods for quality assessment and improvement.

Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models

In this work we propose the use of a graphical diagnostic tool (the funnel plot) to detect outliers among hospitals that treat patients affected by Acute Myocardial Infarction (AMI). We consider an application to data on AMI hospitalizations recorded in the administrative databases of our regional district. The outcome of interest is the in-hospital mortality, a variable indicating if the patient has been discharged dead or alive. We then compare the results obtained by graphical diagnostic tools with those arising from fitting parametric mixed effects models to the same data.