Comparative Effectiveness of New Approaches to Improve Mortality Risk Models From Medicare Claims Data

An Update to the Kaiser Permanente Inpatient Risk Adjustment Methodology Accurately Predicts In-Hospital Mortality: a Retrospective Cohort Study

BACKGROUND

Methods to accurately predict the risk of in-hospital mortality are important for applications including quality assessment of healthcare institutions and research.

OBJECTIVE

To update and validate the Kaiser Permanente inpatient risk adjustment methodology (KP method) to predict in-hospital mortality, using open-source tools to measure comorbidity and diagnosis groups, and removing troponin which is difficult to standardize across modern clinical assays.

DESIGN

Retrospective cohort study using electronic health record data from GEMINI. GEMINI is a research collaborative that collects administrative and clinical data from hospital information systems.

PARTICIPANTS

Adult general medicine inpatients at 28 hospitals in Ontario, Canada, between April 2010 and December 2022.

MAIN MEASURES

The outcome was in-hospital mortality, modeled by diagnosis group using 56 logistic regressions. We compared models with and without troponin as an input to the laboratory-based acute physiology score. We fit and validated the updated method using internal-external cross-validation at 28 hospitals from April 2015 to December 2022.

KEY RESULTS

In 938,103 hospitalizations with 7.2% in-hospital mortality, the updated KP method accurately predicted the risk of mortality. The c-statistic at the median hospital was 0.866 (see Fig. 3) (25th-75th 0.848-0.876, range 0.816-0.927) and calibration was strong for nearly all patients at all hospitals. The 95th percentile absolute difference between predicted and observed probabilities was 0.038 at the median hospital (25th-75th 0.024-0.057, range 0.006-0.118). Model performance was very similar with and without troponin in a subset of 7 hospitals, and performance was similar with and without troponin for patients hospitalized for heart failure and acute myocardial infarction.

CONCLUSIONS

An update to the KP method accurately predicted in-hospital mortality for general medicine inpatients in 28 hospitals in Ontario, Canada. This updated method can be implemented in a wider range of settings using common open-source tools.

Risk-standardized mortality metric to monitor hospital performance for chest pain presentations

AIMS

Risk-standardized mortality rates (RSMR) have been used to monitor hospital performance in procedural and disease-based registries, but limitations include the potential to promote risk-averse clinician decisions and a lack of assessment of the whole patient journey. We aimed to determine whether it is feasible to use RSMR at the symptom-level to monitor hospital performance using routinely collected, linked, clinical and administrative data of chest pain presentations.

METHODS AND RESULTS

We included 192 978 consecutive adult patients (mean age 62 years; 51% female) with acute chest pain without ST-elevation brought via emergency medical services (EMS) to 53 emergency departments in Victoria, Australia (1/1/2015-30/6/2019). From 32 candidate variables, a risk-adjusted logistic regression model for 30-day mortality (C-statistic 0.899) was developed, with excellent calibration in the full cohort and with optimism-adjusted bootstrap internal validation. Annual 30-day RSMR was calculated by dividing each hospital's observed mortality by the expected mortality rate and multiplying it by the annual mean 30-day mortality rate. Hospital performance according to annual 30-day RSMR was lower for outer regional or remote locations and at hospitals without revascularisation capabilities. Hospital rates of angiography or transfer for patients diagnosed with non-ST elevation myocardial infarction (NSTEMI) correlated with annual 30-day RSMR, but no correlations were observed with other existing key performance indicators.

CONCLUSION

Annual hospital 30-day RSMR can be feasibly calculated at the symptom-level using routinely collected, linked clinical, and administrative data. This outcome-based metric appears to provide additional information for monitoring hospital performance in comparison with existing process of care key performance measures.

Guiding Risk Adjustment Models Toward Machine Learning Methods

This Viewpoint reviews the history of administrative risk adjustment models used in health care and provides recommendations for modernizing these models to promote their safe, transparent, equitable, and efficient use.

Risk-Adjusting Mortality in the Nationwide Veterans Affairs Healthcare System

BACKGROUND

The US Veterans Affairs (VA) healthcare system began reporting risk-adjusted mortality for intensive care (ICU) admissions in 2005. However, while the VA's mortality model has been updated and adapted for risk-adjustment of all inpatient hospitalizations, recent model performance has not been published. We sought to assess the current performance of VA's 4 standardized mortality models: acute care 30-day mortality (acute care SMR-30); ICU 30-day mortality (ICU SMR-30); acute care in-hospital mortality (acute care SMR); and ICU in-hospital mortality (ICU SMR).

METHODS

Retrospective cohort study with split derivation and validation samples. Standardized mortality models were fit using derivation data, with coefficients applied to the validation sample. Nationwide VA hospitalizations that met model inclusion criteria during fiscal years 2017-2018(derivation) and 2019 (validation) were included. Model performance was evaluated using c-statistics to assess discrimination and comparison of observed versus predicted deaths to assess calibration.

RESULTS

Among 1,143,351 hospitalizations eligible for the acute care SMR-30 during 2017-2019, in-hospital mortality was 1.8%, and 30-day mortality was 4.3%. C-statistics for the SMR models in validation data were 0.870 (acute care SMR-30); 0.864 (ICU SMR-30); 0.914 (acute care SMR); and 0.887 (ICU SMR). There were 16,036 deaths (4.29% mortality) in the SMR-30 validation cohort versus 17,458 predicted deaths (4.67%), reflecting 0.38% over-prediction. Across deciles of predicted risk, the absolute difference in observed versus predicted percent mortality was a mean of 0.38%, with a maximum error of 1.81% seen in the highest-risk decile.

CONCLUSIONS AND RELEVANCE

The VA's SMR models, which incorporate patient physiology on presentation, are highly predictive and demonstrate good calibration both overall and across risk deciles. The current SMR models perform similarly to the initial ICU SMR model, indicating appropriate adaption and re-calibration.

Predicting mortality in the very old: a machine learning analysis on claims data

Machine learning (ML) may be used to predict mortality. We used claims data from one large German insurer to develop and test differently complex ML prediction models, comparing them for their (balanced) accuracy, but also the importance of different predictors, the relevance of the follow-up period before death (i.e. the amount of accumulated data) and the time distance of the data used for prediction and death. A sample of 373,077 insured very old, aged 75 years or above, living in the Northeast of Germany in 2012 was drawn and followed over 6 years. Our outcome was whether an individual died in one of the years of interest (2013-2017) or not; the primary metric was (balanced) accuracy in a hold-out test dataset. From the 86,326 potential variables, we used the 30 most important ones for modeling. We trained a total of 45 model combinations: (1) Three different ML models were used; logistic regression (LR), random forest (RF), extreme gradient boosting (XGB); (2) Different periods of follow-up were employed for training; 1-5 years; (3) Different time distances between data used for prediction and the time of the event (death/survival) were set; 0-4 years. The mortality rate was 9.15% in mean per year. The models showed (balanced) accuracy between 65 and 93%. A longer follow-up period showed limited to no advantage, but models with short time distance from the event were more accurate than models trained on more distant data. RF and XGB were more accurate than LR. For RF and XGB sensitivity and specificity were similar, while for LR sensitivity was significantly lower than specificity. For all three models, the positive-predictive-value was below 62% (and even dropped to below 20% for longer time distances from death), while the negative-predictive-value significantly exceeded 90% for all analyses. The utilization of and costs for emergency transport as well as emergency and any hospital visits as well as the utilization of conventional outpatient care and laboratory services were consistently found most relevant for predicting mortality. All models showed useful accuracies, and more complex models showed advantages. The variables employed for prediction were consistent across models and with medical reasoning. Identifying individuals at risk could assist tailored decision-making and interventions.

Is the Centers for Medicare and Medicaid Services Hierarchical Condition Category Risk Adjustment Model Satisfactory for Quantifying Risk After Spine Surgery?

BACKGROUND

The Centers for Medicare and Medicaid Services (CMS) hierarchical condition category (HCC) coding is a risk adjustment model that allows for the estimation of risk-and cost-associated with health care provision. Current models may not include key factors that fully delineate the risk associated with spine surgery.

OBJECTIVE

To augment CMS HCC risk adjustment methodology with socioeconomic data to improve its predictive capabilities for spine surgery.

METHODS

The National Inpatient Sample was queried for spinal fusion, and the data was merged with county-level coverage and socioeconomic status variables obtained from the Brookings Institute. We predicted outcomes (death, nonroutine discharge, length of stay [LOS], total charges, and perioperative complication) with pairs of hierarchical, mixed effects logistic regression models-one using CMS HCC score alone and another augmenting CMS HCC scores with demographic and socioeconomic status variables. Models were compared using receiver operating characteristic curves. Variable importance was assessed in conjunction with Wald testing for model optimization.

RESULTS

We analyzed 653 815 patients. Expanded models outperformed models using CMS HCC score alone for mortality, nonroutine discharge, LOS, total charges, and complications. For expanded models, variable importance analyses demonstrated that CMS HCC score was of chief importance for models of mortality, LOS, total charges, and complications. For the model of nonroutine discharge, age was the most important variable. For the model of total charges, unemployment rate was nearly as important as CMS HCC score.

CONCLUSION

The addition of key demographic and socioeconomic characteristics substantially improves the CMS HCC risk-adjustment models when modeling spinal fusion outcomes. This finding may have important implications for payers, hospitals, and policymakers.

Association of Time Elapsed Since Ischemic Stroke With Risk of Recurrent Stroke in Older Patients Undergoing Elective Nonneurologic, Noncardiac Surgery

Importance

Perioperative strokes are a major cause of death and disability. There is limited information on which to base decisions for how long to delay elective nonneurologic, noncardiac surgery in patients with a history of stroke.

Objective

To examine whether an association exists between the time elapsed since an ischemic stroke and the risk of recurrent stroke in older patients undergoing elective nonneurologic, noncardiac surgery.

Design, Setting, and Participants

This cohort study used data from the 100% Medicare Provider Analysis and Review files, including the Master Beneficiary Summary File, between 2011 and 2018 and included elective nonneurologic, noncardiac surgeries in patients 66 years or older. Patients were excluded if they had more than 1 procedure during a 30-day period, were transferred from another hospital or facility, were missing information on race and ethnicity, were admitted in December 2018, or had tracheostomies or gastrostomies. Data were analyzed May 7 to October 23, 2021.

Exposures

Time interval between a previous hospital admission for acute ischemic stroke and surgery.

Main Outcomes and Measures

Acute ischemic stroke during the index surgical admission or rehospitalization for stroke within 30 days of surgery, 30-day all-cause mortality, composite of stroke and mortality, and discharge to a nursing home or skilled nursing facility. Multivariable logistic regression models were used to estimate adjusted odds ratios (AORs) to quantify the association between outcome and time since ischemic stroke.

Results

The final cohort included 5 841 539 patients who underwent elective nonneurologic, noncardiac surgeries (mean [SD] age, 74.1 [6.1] years; 3 371 329 [57.7%] women), of which 54 033 (0.9%) had a previous stroke. Patients with a stroke within 30 days before surgery had higher adjusted odds of perioperative stroke (AOR, 8.02; 95% CI, 6.37-10.10; P < .001) compared with patients without a previous stroke. The adjusted odds of stroke were not significantly different at an interval of 61 to 90 days between previous stroke and surgery (AOR, 5.01; 95% CI, 4.00-6.29; P < .001) compared with 181 to 360 days (AOR, 4.76; 95% CI, 4.26-5.32; P < .001). The adjusted odds of 30-day all-cause mortality were higher in patients who underwent surgery within 30 days of a previous stroke (AOR, 2.51; 95% CI, 1.99-3.16; P < .001) compared with those without a history of stroke, and the AOR decreased to 1.49 (95% CI, 1.15-1.92; P < .001) at 61 to 90 days from previous stroke to surgery but did not decline significantly, even after an interval of 360 or more days.

Conclusions and Relevance

The findings of this cohort study suggest that, among patients undergoing nonneurologic, noncardiac surgery, the risk of stroke and death leveled off when more than 90 days elapsed between a previous stroke and elective surgery. These findings suggest that the recent scientific statement by the American Heart Association to delay elective nonneurologic, noncardiac surgery for at least 6 months after a recent stroke may be too conservative.

Application of the Risk Stratification Index to Multilevel Models of All-condition 30-Day Mortality in Hospitalized Populations Over the Age of 65

BACKGROUND

The Risk Stratification Index (RSI) is superior to Hierarchical Conditions Categories (HCC) in patient-level regressions but has not been applied to assess hospital effects.

OBJECTIVE

The objective of this study was to measure the accuracy of RSI in modeling 30-day hospital mortality across all conditions using multilevel logistic regression.

SUBJECTS AND DATA SOURCES

A 100% sample of Medicare inpatient stays from 2009 to 2014, restricted to patients greater than 65 years of age in general hospitals, resulting in 64 million stays at 3504 hospitals.

RESEARCH DESIGN

We calculated RSI and HCC scores for patient stays using multilevel logistic regression in 3 populations: all inpatients, surgical, and nonsurgical. Correlations of risk-standardized mortality rates with rates of specific case types assessed case-mix balance. Patient stay volume was included to assess smaller hospitals.

RESULTS

We found a negligible correlation of all-conditions risk-standardized mortality rates with hospitals' proportions of orthopedic, cardiac, or pneumonia cases. RSI outperformed HCC in multilevel regressions containing both patient and hospital-level effects. C-statistics using RSI were 0.87 for the all-inpatients group, 0.87 for surgical, and 0.86 for nonsurgical stays. With HCC they were 0.82, 0.82, and 0.81. Akaike Information Criteria and Bayesian Information Criteria values were higher with HCC. RSI shifted 41% of hospitals' rankings by >1 decile. Hospitals with smaller volumes had higher 30-day observed and standardized mortality: 11.2% in the lowest volume quintile versus 8.5% in the highest volume quintile.

CONCLUSION

RSI has superior accuracy and results in a significant shift in rankings compared with HCC in multilevel models of 30-day hospital mortality across all conditions.

Application of machine learning approaches to administrative claims data to predict clinical outcomes in medical and surgical patient populations

Objective

This study aimed to develop and validate a claims-based, machine learning algorithm to predict clinical outcomes across both medical and surgical patient populations.

Methods

This retrospective, observational cohort study, used a random 5% sample of 770,777 fee-for-service Medicare beneficiaries with an inpatient hospitalization between 2009–2011. The machine learning algorithms tested included: support vector machine, random forest, multilayer perceptron, extreme gradient boosted tree, and logistic regression. The extreme gradient boosted tree algorithm outperformed the alternatives and was the machine learning method used for the final risk model. Primary outcome was 30-day mortality. Secondary outcomes were: rehospitalization, and any of 23 adverse clinical events occurring within 30 days of the index admission date.

Results

The machine learning algorithm performance was evaluated by both the area under the receiver operating curve (AUROC) and Brier Score. The risk model demonstrated high performance for prediction of: 30-day mortality (AUROC = 0.88; Brier Score = 0.06), and 17 of the 23 adverse events (AUROC range: 0.80–0.86; Brier Score range: 0.01–0.05). The risk model demonstrated moderate performance for prediction of: rehospitalization within 30 days (AUROC = 0.73; Brier Score: = 0.07) and six of the 23 adverse events (AUROC range: 0.74–0.79; Brier Score range: 0.01–0.02). The machine learning risk model performed comparably on a second, independent validation dataset, confirming that the risk model was not overfit.

Conclusions and relevance

We have developed and validated a robust, claims-based, machine learning risk model that is applicable to both medical and surgical patient populations and demonstrates comparable predictive accuracy to existing risk models.

Incorporating Present-on-Admission Indicators in Medicare Claims to Inform Hospital Quality Measure Risk Adjustment Models

Importance

Present-on-admission (POA) indicators in administrative claims data allow researchers to distinguish between preexisting conditions and those acquired during a hospital stay. The impact of adding POA information to claims-based measures of hospital quality has not yet been investigated to better understand patient underlying risk factors in the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision setting.

Objective

To assess POA indicator use on Medicare claims and to assess the hospital- and patient-level outcomes associated with incorporating POA indicators in identifying risk factors for publicly reported outcome measures used by the Centers for Medicare & Medicaid Services (CMS).

Design, Setting, and Participants

This comparative effectiveness study used national CMS claims data between July 1, 2015, and June 30, 2018. Six hospital quality measures assessing readmission and mortality outcomes were modified to include POA indicators in risk adjustment models. The models using POA were then compared with models using the existing complications-of-care algorithm to evaluate changes in risk model performance. Patient claims data were included for all Medicare fee-for-service and Veterans Administration beneficiaries aged 65 years or older with inpatient hospitalizations for acute myocardial infarction, heart failure, or pneumonia within the measurement period. Data were analyzed between September 2019 and March 2020.

Main Outcomes and Measures

Changes in patient-level (C statistics) and hospital-level (quintile shifts in risk-standardized outcome rates) model performance after including POA indicators in risk adjustment.

Results

Data from a total of 6 027 988 index admissions were included for analysis, ranging from 491 366 admissions (269 209 [54.8%] men; mean [SD] age, 78.2 [8.3] years) for the acute myocardial infarction mortality outcome measure to 1 395 870 admissions (677 158 [48.5%] men; mean [SD] age, 80.3 [8.7] years) for the pneumonia readmission measure. Use of POA indicators was associated with improvements in risk adjustment model performance, particularly for mortality measures (eg, the C statistic increased from 0.728 [95% CI, 0.726-0.730] to 0.774 [95% CI, 0.773-0.776] when incorporating POA indicators into the acute myocardial infarction mortality measure).

Conclusions and Relevance

The findings of this quality improvement study suggest that leveraging POA indicators in the risk adjustment methodology for hospital quality outcome measures may help to more fully capture patients' risk factors and improve overall model performance. Incorporating POA indicators does not require extra effort on the part of hospitals and would be easy to implement in publicly reported quality outcome measures.

Predicting mortality and hospitalization in heart failure using machine learning: A systematic literature review

Objective

The partnership between humans and machines can enhance clinical decisions accuracy, leading to improved patient outcomes. Despite this, the application of machine learning techniques in the healthcare sector, particularly in guiding heart failure patient management, remains unpopular. This systematic review aims to identify factors restricting the integration of machine learning derived risk scores into clinical practice when treating adults with acute and chronic heart failure.

Methods

Four academic research databases and Google Scholar were searched to identify original research studies where heart failure patient data was used to build models predicting all-cause mortality, cardiac death, all-cause and heart failure-related hospitalization.

Results

Thirty studies met the inclusion criteria. The selected studies' sample size ranged between 71 and 716 790 patients, and the median age was 72.1 (interquartile range: 61.1–76.8) years. The minimum and maximum area under the receiver operating characteristic curve (AUC) for models predicting mortality were 0.48 and 0.92, respectively. Models predicting hospitalization had an AUC of 0.47 to 0.84. Nineteen studies (63%) used logistic regression, 53% random forests, and 37% of studies used decision trees to build predictive models. None of the models were built or externally validated using data originating from Africa or the Middle-East.

Conclusions

The variation in the aetiologies of heart failure, limited access to structured health data, distrust in machine learning techniques among clinicians and the modest accuracy of existing predictive models are some of the factors precluding the widespread use of machine learning derived risk calculators.

Logistic regression and machine learning predicted patient mortality from large sets of diagnosis codes comparably

OBJECTIVE

The objective of the study was to compare the performance of logistic regression and boosted trees for predicting patient mortality from large sets of diagnosis codes in electronic healthcare records.

STUDY DESIGN AND SETTING

We analyzed national hospital records and official death records for patients with myocardial infarction (n = 200,119), hip fracture (n = 169,646), or colorectal cancer surgery (n = 56,515) in England in 2015-2017. One-year mortality was predicted from patient age, sex, and socioeconomic status, and 202 to 257 International Classification of Diseases 10th Revision codes recorded in the preceding year or not (binary predictors). Performance measures included the c-statistic, scaled Brier score, and several measures of calibration.

RESULTS

One-year mortality was 17.2% (34,520) after myocardial infarction, 27.2% (46,115) after hip fracture, and 9.3% (5,273) after colorectal surgery. Optimism-adjusted c-statistics for the logistic regression models were 0.884 (95% confidence interval [CI]: 0.882, 0.886), 0.798 (0.796, 0.800), and 0.811 (0.805, 0.817). The equivalent c-statistics for the boosted tree models were 0.891 (95% CI: 0.889, 0.892), 0.804 (0.802, 0.806), and 0.803 (0.797, 0.809). Model performance was also similar when measured using scaled Brier scores. All models were well calibrated overall.

CONCLUSION

In large datasets of electronic healthcare records, logistic regression and boosted tree models of numerous diagnosis codes predicted patient mortality comparably.

Machine Learning-Based Risk Assessment for Cancer Therapy-Related Cardiac Dysfunction in 4300 Longitudinal Oncology Patients

Background The growing awareness of cardiovascular toxicity from cancer therapies has led to the emerging field of cardio-oncology, which centers on preventing, detecting, and treating patients with cardiac dysfunction before, during, or after cancer treatment. Early detection and prevention of cancer therapy-related cardiac dysfunction (CTRCD) play important roles in precision cardio-oncology. Methods and Results This retrospective study included 4309 cancer patients between 1997 and 2018 whose laboratory tests and cardiovascular echocardiographic variables were collected from the Cleveland Clinic institutional electronic medical record database (Epic Systems). Among these patients, 1560 (36%) were diagnosed with at least 1 type of CTRCD, and 838 (19%) developed CTRCD after cancer therapy (de novo). We posited that machine learning algorithms can be implemented to predict CTRCDs in cancer patients according to clinically relevant variables. Classification models were trained and evaluated for 6 types of cardiovascular outcomes, including coronary artery disease (area under the receiver operating characteristic curve [AUROC], 0.821; 95% CI, 0.815-0.826), atrial fibrillation (AUROC, 0.787; 95% CI, 0.782-0.792), heart failure (AUROC, 0.882; 95% CI, 0.878-0.887), stroke (AUROC, 0.660; 95% CI, 0.650-0.670), myocardial infarction (AUROC, 0.807; 95% CI, 0.799-0.816), and de novo CTRCD (AUROC, 0.802; 95% CI, 0.797-0.807). Model generalizability was further confirmed using time-split data. Model inspection revealed several clinically relevant variables significantly associated with CTRCDs, including age, hypertension, glucose levels, left ventricular ejection fraction, creatinine, and aspartate aminotransferase levels. Conclusions This study suggests that machine learning approaches offer powerful tools for cardiac risk stratification in oncology patients by utilizing large-scale, longitudinal patient data from healthcare systems.

Paying for Performance Improvement in Quality and Outcomes of Cardiovascular Care: Challenges and Prospects

Over the past two decades, Medicare and other payers have been looking at ways to base payment for cardiovascular care on the quality and outcomes of care delivered. Public reporting of hospital performance on a series of quality measures began in 2004 with basic processes of care such as aspirin use and influenza vaccination, and it expanded in later years to include outcomes such as mortality and readmission rates. Following the passage of the Affordable Care Act in March 2010, Medicare and other payers moved forward with pay-for-performance programs, more commonly referred to as value-based purchasing (VBP) programs. These programs are largely based on an underlying fee-for-service payment infrastructure and give hospitals and clinicians bonuses or penalties based on their performance. Another new payment mechanism, called alternative payment models (APMs), aims to move towards episode-based or global payments to improve quality and efficiency. The two most relevant APMs for cardiovascular care include Accountable Care Organizations and bundled payments. Both VBP programs and APMs have challenges related to program efficacy, accuracy, and equity. In fact, despite over a decade of progress in measuring and incentivizing high-quality care delivery within cardiology, major limitations remain. Many of the programs have had little benefit in terms of clinical outcomes yet have led to marked administrative burden for participants. However, there are several encouraging prospects to aid the successful implementation of value-based high-quality cardiovascular care, such as more sophisticated data science to improve risk adjustment and flexible electronic health records to decrease administrative burden. Furthermore, payment models designed specifically for cardiovascular care could incentivize innovative care delivery models that could improve quality and outcomes for patients. This review provides an overview of current efforts, largely at the federal level, to pay for high-quality cardiovascular care and discusses the challenges and prospects related to doing so.

Diagnostic Category Prevalence in 3 Classification Systems Across the Transition to the International Classification of Diseases, Tenth Revision, Clinical Modification

Importance

On October 1, 2015, the US transitioned to the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) for recording diagnoses, symptoms, and procedures. It is unknown whether this transition was associated with changes in diagnostic category prevalence based on diagnosis classification systems commonly used for payment and quality reporting.

Objective

To assess changes in diagnostic category prevalence associated with the ICD-10-CM transition.

Design, Setting, and Participants

This interrupted time series analysis and cross-sectional study examined level and trend changes in diagnostic category prevalence associated with the ICD-10-CM transition and clinically reviewed a subset of diagnostic categories with changes of 20% or more. Data included insurance claim diagnoses from the IBM MarketScan Commercial Database from January 1, 2010, to December 31, 2017, for more than 18 million people aged 0 to 64 years with private insurance. Diagnoses were mapped using 3 common diagnostic classification systems: World Health Organization (WHO) disease chapters, Department of Health and Human Services Hierarchical Condition Categories (HHS-HCCs), and Agency for Healthcare Research and Quality Clinical Classification System (AHRQ-CCS). Data were analyzed from December 1, 2018, to January 21, 2020.

Exposures

US implementation of ICD-10-CM.

Main Outcomes and Measures

Monthly rates of individuals with at least 1 diagnosis in a diagnostic classification category per 10 000 eligible members.

Results

The analytic sample contained information on 2.1 billion enrollee person-months with 3.4 billion clinically assigned diagnoses; the mean (range) monthly sample size was 22.1 (18.4 to 27.1 ) million individuals. While diagnostic category prevalence changed minimally for WHO disease chapters, the ICD-10-CM transition was associated with level changes of 20% or more among 20 of 127 HHS-HCCs (15.7%) and 46 of 282 AHRQ-CCS categories (16.3%) and with trend changes of 20% or more among 12 of 127 of HHS-HCCs (9.4%) and 27 of 282 of AHRQ-CCS categories (9.6%). For HHS-HCCs, monthly rates of individuals with any acute myocardial infarction diagnosis increased 131.5% (95% CI, 124.1% to 138.8%), primarily because HHS added non-ST-segment-elevation myocardial infarction diagnoses to this category. The HHS-HCC for diabetes with chronic complications increased by 92.4% (95% CI, 84.2% to 100.5%), primarily from including new diabetes-related hypoglycemia and hyperglycemia codes, and the rate for completed pregnancy with complications decreased by 54.5% (95% CI, -58.7% to -50.2%) partly due to removing vaginal birth after cesarean delivery as a complication.

Conclusions and Relevance

These findings suggest that the ICD-10-CM transition was associated with large prevalence changes for many diagnostic categories. Diagnostic classification systems developed using ICD-9-CM may need to be refined using ICD-10-CM data to avoid unintended consequences for disease surveillance, performance assessment, and risk-adjusted payments.

Development and Testing of Improved Models to Predict Payment Using Centers for Medicare & Medicaid Services Claims Data

Importance

Predicting payments for particular conditions or populations is essential for research, benchmarking, public reporting, and calculations for population-based programs. Centers for Medicare & Medicaid Services (CMS) models often group codes into disease categories, but using single, rather than grouped, diagnostic codes and leveraging present on admission (POA) codes may enhance these models.

Objective

To determine whether changes to the candidate variables in CMS models would improve risk models predicting patient total payment within 30 days of hospitalization for acute myocardial infarction (AMI), heart failure (HF), and pneumonia.

Design, Setting, and Participants

This comparative effectiveness research study used data from Medicare fee-for-service hospitalizations for AMI, HF, and pneumonia at acute care hospitals from July 1, 2013, through September 30, 2015. Payments across multiple care settings, services, and supplies were included and adjusted for geographic and policy variations, corrected for inflation, and winsorized. The same data source was used but varied for the candidate variables and their selection, and the method used by CMS for public reporting that used grouped codes was compared with variations that used POA codes and single diagnostic codes. Combinations of use of POA codes, separation of index admission diagnoses from those in the previous 12 months, and use of individual International Classification of Diseases, Ninth Revision, Clinical Modification codes instead of grouped diagnostic categories were tested. Data analysis was performed from December 4, 2017, to June 10, 2019.

Main Outcomes and Measures

The models' goodness of fit was compared using root mean square error (RMSE) and the McFadden pseudo R2.

Results

Among the 1 943 049 total hospitalizations of the study participants, 343 116 admissions were for AMI (52.5% male; 37.4% aged ≤74 years), 677 044 for HF (45.5% male; 25.9% aged ≤74 years), and 922 889 for pneumonia (46.4% male; 28.2% aged ≤74 years). The mean (SD) 30-day payment was $23 103 ($18 221) for AMI, $16 365 ($12 527) for HF, and $17 097 ($12 087) for pneumonia. Each incremental model change improved the pseudo R2 and RMSE. Incorporating all 3 changes improved the pseudo R2 of the patient-level models from 0.077 to 0.129 for AMI, from 0.042 to 0.129 for HF, and from 0.114 to 0.237 for pneumonia. Parallel improvements in RMSE were found for all 3 conditions.

Conclusions and Relevance

Leveraging POA codes, separating index from previous diagnoses, and using single diagnostic codes improved payment models. Better models can potentially improve research, benchmarking, public reporting, and calculations for population-based programs.