Impact of the choice of benchmark on the conclusions of hospital report cards

Hospital variation in admissions for low back pain following an emergency department presentation: a retrospective study

Background

One in 6 patients with low back pain (LBP) presenting to emergency departments (EDs) are subsequently admitted to hospital each year, making LBP the ninth most common reason for hospital admission in Australia. No studies have investigated and quantified the extent of clinical variation in hospital admission following an ED presentation for LBP.

Methods

We used routinely collected ED data from public hospitals within the state of New South Wales, Australia, to identify presentations of patients aged between 18 and 111 with a discharge diagnosis of LBP. We fitted a series of random effects multilevel logistic regression models adjusted by case-mix and hospital variables. The main outcome was the hospital-adjusted admission rate (HAAR). Data were presented as funnel plots with 95% and 99.8% confidence limits. Hospitals with a HAAR outside the 95% confidence limit were considered to have a HAAR significantly different to the state average.

Results

We identified 176,729 LBP presentations across 177 public hospital EDs and 44,549 hospital admissions (25.2%). The mean (SD) age was 51.8 (19.5) and 52% were female. Hospital factors explained 10% of the variation (ICC = 0.10), and the median odds ratio (MOR) was 2.03. We identified marked variation across hospitals, with HAAR ranging from 6.9 to 65.9%. After adjusting for hospital variables, there was still marked variation between hospitals with similar characteristics.

Conclusion

We found substantial variation in hospital admissions following a presentation to the ED due to LBP even after controlling by case-mix and hospital characteristics. Given the substantial costs associated with these admissions, our findings indicate the need to investigate sources of variation and to determine instances where the observed variation is warranted or unwarranted.

Supplementary information

The online version contains supplementary material available at 10.1186/s12913-022-08134-8.

Improving Medicare's Hospital Compare Mortality Model

OBJECTIVE

To improve the predictions provided by Medicare's Hospital Compare (HC) to facilitate better informed decisions regarding hospital choice by the public.

DATA SOURCES/SETTING

Medicare claims on all patients admitted for Acute Myocardial Infarction between 2009 through 2011.

STUDY DESIGN

Cohort analysis using a Bayesian approach, comparing the present assumptions of HC (using a constant mean and constant variance for all hospital random effects), versus an expanded model that allows for the inclusion of hospital characteristics to permit the data to determine whether they vary with attributes of hospitals, such as volume, capabilities, and staffing. Hospital predictions are then created using directly standardized estimates to facilitate comparisons between hospitals.

DATA COLLECTION/EXTRACTION METHODS

Medicare fee-for-service claims.

PRINCIPAL FINDINGS

Our model that included hospital characteristics produces very different predictions from the current HC model, with higher predicted mortality rates at hospitals with lower volume and worse characteristics. Using Chicago as an example, the expanded model would advise patients against seeking treatment at the smallest hospitals with worse technology and staffing.

CONCLUSION

To aid patients when selecting between hospitals, the Centers for Medicare and Medicaid Services (CMS) should improve the HC model by permitting its predictions to vary systematically with hospital attributes such as volume, capabilities, and staffing.

Performance profiling in primary care: does the choice of statistical model matter?

BACKGROUND

Profiling is increasingly being used to generate input for improvement efforts in health care. For these efforts to be successful, profiles must reflect true provider performance, requiring an appropriate statistical model. Sophisticated models are available to account for the specific features of performance data, but they may be difficult to use and explain to providers.

OBJECTIVE

To assess the influence of the statistical model on the performance profiles of primary care providers. Data Source. Administrative data (2006–2008) on 2.8 million members of a Dutch health insurer who were registered with 1 of 4396 general practitioners.

METHODS

Profiles are constructed for 6 quality measures and 5 resource use measures, controlling for differences in case mix. Models include ordinary least squares, generalized linear models, and multilevel models. Separately for each model, providers are ranked on z scores and classified as outlier if belonging to the 10% with the worst or best performance. The impact of the model is evaluated using the weighted kappa for rankings overall, percentage agreement on outlier designation, and changes in rankings over time.

RESULTS

Agreement among models was relatively high overall (kappa typically .0.85). Agreement on outlier designation was more variable and often below 80%, especially for high outliers. Rankings were more similar for processes than for outcomes and expenses. Agreement among annual rankings per model was low for all models.

CONCLUSIONS

Differences among models were relatively small, but the choice of statistical model did affect the rankings. In addition, most measures appear to be driven largely by chance, regardless of the model that is used. Profilers should pay careful attention to the choice of both the statistical model and the performance measures.

Development of peer-group-classification criteria for the comparison of cost efficiency among general hospitals under the Korean NHI program

OBJECTIVES

To classify general hospitals into homogeneous systematic-risk groups in order to compare cost efficiency and propose peer-group-classification criteria.

DATA SOURCES

Health care institution registration data and inpatient-episode-based claims data submitted by the Korea National Health Insurance system to the Health Insurance Review and Assessment Service from July 2007 to December 2009.

STUDY DESIGN

Cluster analysis was performed to classify general hospitals into peer groups based on similarities in hospital characteristics, case mix complexity, and service-distribution characteristics. Classification criteria reflecting clustering were developed. To test whether the new peer groups better adjusted for differences in systematic risks among peer groups, we compared the R(2) statistics of the current and proposed peer groups according to total variations in medical costs per episode and case mix indices influencing the cost efficiency.

DATA COLLECTION

A total of 1,236,471 inpatient episodes were constructed for 222 general hospitals in 2008.

PRINCIPAL FINDINGS

New criteria were developed to classify general hospitals into three peer groups (large general hospitals, small and medium general hospitals treating severe cases, and small and medium general hospitals) according to size and case mix index.

CONCLUSIONS

This study provides information about using peer grouping to enhance fairness in the performance assessment of health care providers.

Hospital-wide mortality as a quality metric: conceptual and methodological challenges

Hospital-wide mortality rates are used as a measure of overall hospital quality. However, their parsimony and apparent simplicity belie significant conceptual and methodological concerns. For many diagnoses included in hospital-wide mortality, the association between short-term mortality and quality of care is not well established. Furthermore, compared with condition-specific or procedure-specific mortality, hospital-wide mortality rates pose greater methodological challenges (ie, eligibility and exclusion criteria, risk adjustment, statistical techniques for aggregating across diagnoses, usability). Many of these result from substantial interprovider heterogeneity in diagnosis frequency, sample sizes, and patient severity. Hospital-wide mortality is problematic as a quality metric for public reporting, although hospitals may elect to use such measures for other purposes. Potential alternative approaches include multidimensional composite metrics or mortality measurement limited to selected conditions and procedures for which the link between hospital mortality and quality is clear, legitimate exclusions are uncommon, and sample sizes, end points, and risk adjustment are adequate.

Prediction of random effects in linear and generalized linear models under model misspecification

Statistical models that include random effects are commonly used to analyze longitudinal and correlated data, often with the assumption that the random effects follow a Gaussian distribution. Via theoretical and numerical calculations and simulation, we investigate the impact of misspecification of this distribution on both how well the predicted values recover the true underlying distribution and the accuracy of prediction of the realized values of the random effects. We show that, although the predicted values can vary with the assumed distribution, the prediction accuracy, as measured by mean square error, is little affected for mild-to-moderate violations of the assumptions. Thus, standard approaches, readily available in statistical software, will often suffice. The results are illustrated using data from the Heart and Estrogen/Progestin Replacement Study using models to predict future blood pressure values.

Are (the log-odds of) hospital mortality rates normally distributed? Implications for studying variations in outcomes of medical care

RATIONALE

Hierarchical regression models are increasingly being used to examine variations in outcomes following the provision of medical care across providers. These models frequently assume a normal distribution for the provider-specific random effects. The appropriateness of this assumption for examining variations in health care outcomes has never been explicitly tested.

AIMS AND OBJECTIVES

To compare hierarchical logistic regression models in which the provider-specific random effects were either a normal distribution or a mixture of three normal distributions.

METHODS

We used data on 18,825 patients admitted to 109 hospitals in Ontario with a diagnosis of acute myocardial infarction. We used the Deviance Information Criterion, Bayes factors and predictive distributions to compare the evidence between the two competing models.

RESULTS

There was strong evidence that the distribution of hospital-specific log-odds of mortality was a mixture of three normal distributions compared to the evidence that it was normal. In some scenarios, the hospital-specific posterior tail probabilities of unacceptably high mortality were lower when a logistic-normal model was fit compared to when a logistic-mixture of normal distributions model was fit. Additionally, in these same scenarios, fewer hospitals were classified as having higher than acceptable mortality when the logistic-mixture of three normal distributions was used.

CONCLUSIONS

These findings have important consequences for those who use hierarchical models to examine variations in outcomes of medical care across providers since the mixture of three normal distributions model indicated that variations in outcomes across providers was greater than indicated by the logistic-normal model.

Method to develop health care peer groups for quality and financial comparisons across hospitals

OBJECTIVE

To develop and explore the characteristics of a novel "nearest neighbor" methodology for creating peer groups for health care facilities.

DATA SOURCES

Data were obtained from the Department of Veterans Affairs (VA) databases. STATISTICAL METHODS AND FINDINGS: Peer groups are developed by first calculating the multidimensional Euclidean distance between each of 133 VA medical centers based on 16 facility characteristics. Each medical center then serves as the center for its own peer group, and the nearest neighbor facilities in terms of Euclidean distance comprise the peer facilities. We explore the attributes and characteristics of the nearest neighbor peer groupings. In addition, we construct standard cluster analysis-derived peer groups and compare the characteristics of groupings from the two methodologies.

CONCLUSIONS

The novel peer group methodology presented here results in groups where each medical center is at the center of its own peer group. Possible advantages over other peer group methodologies are that facilities are never on the "edge" of a group and group size-and thus group dispersion-is determined by the researcher. Peer groups with these characteristics may be more appealing to some researchers and administrators than standard cluster analysis and may thus strengthen organizational buy-in for financial and quality comparisons.

Massachusetts cardiac surgery report card: implications of statistical methodology

BACKGROUND

Choice of statistical methodology may significantly impact the results of provider profiling, including cardiac surgery report cards. Because of sample size and clustering issues, logistic regression may overestimate systematic interprovider variability, leading to false outlier classification. Theoretically, the use of hierarchical models should result in more accurate representation of provider performance.

METHODS

Extensively validated and audited data were available for all 4,603 isolated coronary artery bypass grafting procedures performed at 13 Massachusetts hospitals during 2002. To produce the official Massachusetts cardiac surgery report card, a 19-variable predictor set and a hierarchical generalized linear model were employed. For the current study, this same analysis was repeated with the 14 predictors used in the New York Cardiac Surgery Reporting System. Two additional analyses were conducted using each set of predictor variables and applying standard logistic regression. For each of the four combinations of predictors and models, the point estimates of risk-adjusted 30-day mortality, 95% confidence or probability intervals, and outlier status were determined for each hospital.

RESULTS

Overall unadjusted mortality for coronary bypass operations was 2.19%. For most hospitals, there was wide variability in the point estimates and confidence or probability intervals of risk-adjusted mortality depending on statistical model, but little variability relative to the choice of predictors. There were no hospital outliers using hierarchical models, but there was one outlier using logistic regression with either predictor set.

CONCLUSIONS

When used to compare provider performance, logistic regression increases the possibility of false outlier classification. The use of hierarchical models is recommended.