Navigating by Stars: Using CMS Star Ratings to Choose Hospitals for Complex Cancer Surgery

Variation in Postoperative Outcomes Across Federally Designated Hospital Star Ratings

Importance

Despite widespread use to guide patients to hospitals providing the best care, it remains unknown whether Centers for Medicare & Medicaid Services (CMS) hospital star ratings are a reliable measure of hospital surgical quality.

Objective

To examine the CMS hospital star ratings and hospital surgical quality measured by 30-day postoperative mortality, serious complications, and readmission rates for Medicare beneficiaries undergoing colectomy, coronary artery bypass graft, cholecystectomy, appendectomy, and incisional hernia repair.

Design, Setting, and Participants

This cohort study evaluated 100% Medicare administrative claims for nonfederal acute care hospitals with a CMS hospital star rating for calendar years 2014-2018. Data analysis was performed from January 15, 2022, to April 30, 2023. Participants included fee-for-service Medicare beneficiaries aged 66 to 99 years who underwent colectomy, coronary artery bypass graft, cholecystectomy, appendectomy, or incisional hernia repair with continuous Medicare coverage for 3 months before and 6 months after surgery.

Exposure

Centers for Medicare & Medicaid Services hospital star rating.

Main Outcomes and Measures

Risk- and reliability-adjusted hospital rates of 30-day postoperative mortality, serious complications, and 30-day readmissions were measured and compared across hospitals and star ratings.

Results

A total of 1 898 829 patients underwent colectomy, coronary artery bypass graft, cholecystectomy, appendectomy, or incisional hernia repair at 3240 hospitals with a CMS hospital star rating. Mean (SD) age was 74.8 (7.0) years, 50.6% of the patients were male, and 86.5% identified as White. Risk- and reliability-adjusted 30-day mortality rate decreased in a stepwise fashion from 6.80% (95% CI, 6.79%-6.81%) in 1-star hospitals to 4.93% (95% CI, 4.93%-4.94%) in 5-star hospitals (adjusted odds ratio, 1.86; 95% CI, 1.73-2.00). There was wide variation in the rates of hospital mortality (variation, 1.89%; range, 2.4%-16.2%), serious complications (variation, 1.97%; range, 5.5%-45.1%), and readmission (variation, 1.27%; range, 9.1%-22.5%) across all hospitals. After stratifying hospitals by their star rating, similar patterns of variation were observed within star rating groups for 30-day mortality: 1 star (variation, 1.91%; range, 3.6%-12.0%), 2 star (variation, 1.86%; range, 2.8%-16.2%), 3 star (variation, 1.84%; range, 2.9%-12.3%), 4 star (variation, 1.76%; range, 2.9%-11.5%), and 5 star (variation, 1.79%; range, 2.4%-9.1%). Similar patterns were observed for serious complications and readmissions.

Conclusion and Relevance

Although CMS hospital star rating was associated with postoperative mortality, serious complications, and readmissions, there was wide variation in surgical outcomes within each star rating group. These findings highlight the limitations of the CMS hospital star rating system as a measure of surgical quality and should be a call for continued improvement of publicly reported hospital grade measures.

Comparison of Simulated Outcomes of Colorectal Cancer Surgery at the Highest-Performing vs Chosen Local Hospitals

IMPORTANCE

Variation in outcomes across hospitals adversely affects surgical patients. The use of high-quality hospitals varies by population, which may contribute to surgical disparities.

OBJECTIVE

To simulate the implications of data-driven hospital selection for social welfare among patients who underwent colorectal cancer surgery.

DESIGN, SETTING, AND PARTICIPANTS

This economic evaluation used the hospital inpatient file from the Florida Agency for Health Care Administration. Surgical outcomes of patients who were treated between January 1, 2016, and December 31, 2018 (training cohort), were used to estimate hospital performance. Costs and benefits of care at alternative hospitals were assessed in patients who were treated between January 1, 2019, and December 31, 2019 (testing cohort). The cohorts comprised patients 18 years or older who underwent elective colorectal resection for benign or malignant neoplasms. Data were analyzed from March to October 2022.

EXPOSURES

Using hierarchical logistic regression, we estimated the implications of hospital selection for in-hospital mortality risk in patients in the training cohort. These estimates were applied to patients in the testing cohort using bayesian simulations to compare outcomes at each patient's highest-performing and chosen local hospitals. Analyses were stratified by race and ethnicity to evaluate the potential implications for equity.

MAIN OUTCOMES AND MEASURES

The primary outcome was the mean patient-level change in social welfare, a composite measure balancing the value of reduced mortality with associated costs of care at higher-performing hospitals.

RESULTS

A total of 21 098 patients (mean [SD] age, 67.3 [12.0] years; 10 782 males [51.1%]; 2232 Black [10.6%] and 18 866 White [89.4%] individuals) who were treated at 178 hospitals were included. A higher-quality local hospital was identified for 3057 of 5000 patients (61.1%) in the testing cohort. Selecting the highest-performing hospital was associated with a 26.5% (95% CI, 24.5%-29.0%) relative reduction and 0.24% (95% CI, 0.23%-0.25%) absolute reduction in mortality risk. A mean amount of $1953 (95% CI, $1744-$2162) was gained in social welfare per patient treated. Simulated reassignment to a higher-quality local hospital was associated with a 23.5% (95% CI, 19.3%-32.9%) relative reduction and 0.26% (95% CI, 0.21%-0.30%) absolute reduction in mortality risk for Black patients, with $2427 (95% CI, $1697-$3158) gained in social welfare.

CONCLUSIONS AND RELEVANCE

In this economic evaluation, using procedure-specific hospital performance as the primary factor in the selection of a local hospital for colorectal cancer surgery was associated with improved outcomes for both patients and society. Surgical outcomes data can be used to transform care and guide policy in colorectal cancer.

Concordance of Hospital Ranks and Category Ratings Using the Current Technical Specification of US Hospital Star Ratings and Reasonable Alternative Specifications

Importance

The Centers for Medicare & Medicaid Services (CMS) Hospital Compare star ratings are widely used summaries of hospital quality that can influence patient choice and organizational reputation.

Objective

To identify the changes in hospital ratings and rankings associated with alternative methodological choices in the calculation of the 2021 CMS Hospital Compare star ratings.

Design Setting and Participants

This cross-sectional study used publicly available 2021 Hospital Compare data for 3339 US hospitals from the October 2020 data release. Change in apparent hospital performance was assessed, arising when plausible alternatives to current methods are used for calculating star ratings in relation to individual measure standardization, domain derivation, and domain weighting. Three example changes were examined in detail, with more comprehensive changes considered using Monte Carlo simulation. Changes in centile of hospital ranks and in star rating overall were examined, as well as separately in CMS peer groupings defined by the number of reported quality domains.

Main Outcomes and Measures

Proportion of hospitals receiving a different star rating under the alternative technical specifications than under the current (2021) CMS approach and mean absolute change in centile rank of hospitals under alternative technical specifications compared with the current (2021) CMS approach.

Results

A total of 3339 US hospitals were included in the analysis. Of the specific changes to current specifications considered, the alternative method of standardization was associated with the most substantial changes, with 55.4% (95% CI, 53.7%-57.1%) of hospitals having their star rating reclassified. The change in domain weights was associated with the smallest differences, but even that resulted in reclassification of the star rating in approximately 1 in 4 (24.5%; 95% CI, 23.0%-26.0%) hospitals. In the simulation study, which covered a range of possible changes, on average half of hospitals (51.8%; IQR, 44.2%-59.7%) were assigned a different star rating from that produced by the current specification, usually into an adjacent star category.

Conclusions and Relevance

In this cross-sectional study of data on US hospitals, CMS Hospital Compare star ratings were found to be highly sensitive to how performance ratings are calculated, demonstrating the need for transparent justification of the technical approaches used in calculating composite performance ratings.