1
|
Roberts SB, Colacci M, Razak F, Verma AA. An Update to the Kaiser Permanente Inpatient Risk Adjustment Methodology Accurately Predicts In-Hospital Mortality: a Retrospective Cohort Study. J Gen Intern Med 2023; 38:3303-3312. [PMID: 37296357 PMCID: PMC10682304 DOI: 10.1007/s11606-023-08245-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/10/2023] [Accepted: 05/16/2023] [Indexed: 06/12/2023]
Abstract
BACKGROUND Methods to accurately predict the risk of in-hospital mortality are important for applications including quality assessment of healthcare institutions and research. OBJECTIVE To update and validate the Kaiser Permanente inpatient risk adjustment methodology (KP method) to predict in-hospital mortality, using open-source tools to measure comorbidity and diagnosis groups, and removing troponin which is difficult to standardize across modern clinical assays. DESIGN Retrospective cohort study using electronic health record data from GEMINI. GEMINI is a research collaborative that collects administrative and clinical data from hospital information systems. PARTICIPANTS Adult general medicine inpatients at 28 hospitals in Ontario, Canada, between April 2010 and December 2022. MAIN MEASURES The outcome was in-hospital mortality, modeled by diagnosis group using 56 logistic regressions. We compared models with and without troponin as an input to the laboratory-based acute physiology score. We fit and validated the updated method using internal-external cross-validation at 28 hospitals from April 2015 to December 2022. KEY RESULTS In 938,103 hospitalizations with 7.2% in-hospital mortality, the updated KP method accurately predicted the risk of mortality. The c-statistic at the median hospital was 0.866 (see Fig. 3) (25th-75th 0.848-0.876, range 0.816-0.927) and calibration was strong for nearly all patients at all hospitals. The 95th percentile absolute difference between predicted and observed probabilities was 0.038 at the median hospital (25th-75th 0.024-0.057, range 0.006-0.118). Model performance was very similar with and without troponin in a subset of 7 hospitals, and performance was similar with and without troponin for patients hospitalized for heart failure and acute myocardial infarction. CONCLUSIONS An update to the KP method accurately predicted in-hospital mortality for general medicine inpatients in 28 hospitals in Ontario, Canada. This updated method can be implemented in a wider range of settings using common open-source tools.
Collapse
Affiliation(s)
- Surain B Roberts
- Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, ON, Canada.
| | - Michael Colacci
- Department of Medicine, University of Toronto, Toronto, ON, Canada
| | - Fahad Razak
- Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, ON, Canada
- Department of Medicine, University of Toronto, Toronto, ON, Canada
- Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, ON, Canada
| | - Amol A Verma
- Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, ON, Canada
- Department of Medicine, University of Toronto, Toronto, ON, Canada
- Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
2
|
Dawson LP, Andrew E, Nehme Z, Bloom J, Okyere D, Cox S, Anderson D, Stephenson M, Lefkovits J, Taylor AJ, Kaye D, Smith K, Stub D. Risk-standardized mortality metric to monitor hospital performance for chest pain presentations. EUROPEAN HEART JOURNAL. QUALITY OF CARE & CLINICAL OUTCOMES 2023; 9:583-591. [PMID: 36195327 DOI: 10.1093/ehjqcco/qcac062] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/08/2022] [Revised: 07/10/2022] [Accepted: 09/29/2022] [Indexed: 09/13/2023]
Abstract
AIMS Risk-standardized mortality rates (RSMR) have been used to monitor hospital performance in procedural and disease-based registries, but limitations include the potential to promote risk-averse clinician decisions and a lack of assessment of the whole patient journey. We aimed to determine whether it is feasible to use RSMR at the symptom-level to monitor hospital performance using routinely collected, linked, clinical and administrative data of chest pain presentations. METHODS AND RESULTS We included 192 978 consecutive adult patients (mean age 62 years; 51% female) with acute chest pain without ST-elevation brought via emergency medical services (EMS) to 53 emergency departments in Victoria, Australia (1/1/2015-30/6/2019). From 32 candidate variables, a risk-adjusted logistic regression model for 30-day mortality (C-statistic 0.899) was developed, with excellent calibration in the full cohort and with optimism-adjusted bootstrap internal validation. Annual 30-day RSMR was calculated by dividing each hospital's observed mortality by the expected mortality rate and multiplying it by the annual mean 30-day mortality rate. Hospital performance according to annual 30-day RSMR was lower for outer regional or remote locations and at hospitals without revascularisation capabilities. Hospital rates of angiography or transfer for patients diagnosed with non-ST elevation myocardial infarction (NSTEMI) correlated with annual 30-day RSMR, but no correlations were observed with other existing key performance indicators. CONCLUSION Annual hospital 30-day RSMR can be feasibly calculated at the symptom-level using routinely collected, linked clinical, and administrative data. This outcome-based metric appears to provide additional information for monitoring hospital performance in comparison with existing process of care key performance measures.
Collapse
Affiliation(s)
- Luke P Dawson
- Department of Cardiology, The Alfred Hospital, Melbourne, VIC 3004, Australia
- Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia
- Department of Cardiology, The Royal Melbourne Hospital, Melbourne, VIC 3050, Australia
| | - Emily Andrew
- Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia
- Centre for Research & Evaluation, Ambulance Victoria, Melbourne, VIC 3130, Australia
| | - Ziad Nehme
- Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia
- Centre for Research & Evaluation, Ambulance Victoria, Melbourne, VIC 3130, Australia
- Department of Paramedicine, Monash University, Melbourne, VIC 3199, Australia
| | - Jason Bloom
- Department of Cardiology, The Alfred Hospital, Melbourne, VIC 3004, Australia
- Heart Failure Research Group, The Baker Institute, Melbourne, VIC 3004, Australia
| | - Daniel Okyere
- Centre for Research & Evaluation, Ambulance Victoria, Melbourne, VIC 3130, Australia
| | - Shelley Cox
- Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia
- Centre for Research & Evaluation, Ambulance Victoria, Melbourne, VIC 3130, Australia
| | - David Anderson
- Centre for Research & Evaluation, Ambulance Victoria, Melbourne, VIC 3130, Australia
- Department of Intensive Care Medicine, The Alfred Hospital, Melbourne, VIC 3004, Australia
| | - Michael Stephenson
- Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia
- Centre for Research & Evaluation, Ambulance Victoria, Melbourne, VIC 3130, Australia
- Department of Paramedicine, Monash University, Melbourne, VIC 3199, Australia
| | - Jeffrey Lefkovits
- Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia
- Department of Cardiology, The Royal Melbourne Hospital, Melbourne, VIC 3050, Australia
| | - Andrew J Taylor
- Department of Cardiology, The Alfred Hospital, Melbourne, VIC 3004, Australia
- Department of Medicine, Monash University, Melbourne, VIC 3800, Australia
| | - David Kaye
- Department of Cardiology, The Alfred Hospital, Melbourne, VIC 3004, Australia
- Heart Failure Research Group, The Baker Institute, Melbourne, VIC 3004, Australia
| | - Karen Smith
- Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia
- Centre for Research & Evaluation, Ambulance Victoria, Melbourne, VIC 3130, Australia
- Department of Paramedicine, Monash University, Melbourne, VIC 3199, Australia
| | - Dion Stub
- Department of Cardiology, The Alfred Hospital, Melbourne, VIC 3004, Australia
- Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia
- Heart Failure Research Group, The Baker Institute, Melbourne, VIC 3004, Australia
| |
Collapse
|
3
|
Weissman GE, Joynt Maddox KE. Guiding Risk Adjustment Models Toward Machine Learning Methods. JAMA 2023; 330:807-808. [PMID: 37566405 DOI: 10.1001/jama.2023.12920] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 08/12/2023]
Abstract
This Viewpoint reviews the history of administrative risk adjustment models used in health care and provides recommendations for modernizing these models to promote their safe, transparent, equitable, and efficient use.
Collapse
Affiliation(s)
- Gary E Weissman
- Palliative and Advanced Illness Research (PAIR) Center, Department of Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia
- Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia
| | - Karen E Joynt Maddox
- Cardiovascular Division, John T. Milliken Department of Internal Medicine, and Center for Advancing Health Services, Policy & Economics Research (CAHSPER), Institute for Public Health, Washington University in St Louis, St Louis, Missouri
- Associate Editor, JAMA
| |
Collapse
|
4
|
Prescott HC, Kadel RP, Eyman JR, Freyberg R, Quarrick M, Brewer D, Hasselbeck R. Risk-Adjusting Mortality in the Nationwide Veterans Affairs Healthcare System. J Gen Intern Med 2022; 37:3877-3884. [PMID: 35028862 PMCID: PMC9640507 DOI: 10.1007/s11606-021-07377-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/12/2021] [Accepted: 12/17/2021] [Indexed: 12/03/2022]
Abstract
BACKGROUND The US Veterans Affairs (VA) healthcare system began reporting risk-adjusted mortality for intensive care (ICU) admissions in 2005. However, while the VA's mortality model has been updated and adapted for risk-adjustment of all inpatient hospitalizations, recent model performance has not been published. We sought to assess the current performance of VA's 4 standardized mortality models: acute care 30-day mortality (acute care SMR-30); ICU 30-day mortality (ICU SMR-30); acute care in-hospital mortality (acute care SMR); and ICU in-hospital mortality (ICU SMR). METHODS Retrospective cohort study with split derivation and validation samples. Standardized mortality models were fit using derivation data, with coefficients applied to the validation sample. Nationwide VA hospitalizations that met model inclusion criteria during fiscal years 2017-2018(derivation) and 2019 (validation) were included. Model performance was evaluated using c-statistics to assess discrimination and comparison of observed versus predicted deaths to assess calibration. RESULTS Among 1,143,351 hospitalizations eligible for the acute care SMR-30 during 2017-2019, in-hospital mortality was 1.8%, and 30-day mortality was 4.3%. C-statistics for the SMR models in validation data were 0.870 (acute care SMR-30); 0.864 (ICU SMR-30); 0.914 (acute care SMR); and 0.887 (ICU SMR). There were 16,036 deaths (4.29% mortality) in the SMR-30 validation cohort versus 17,458 predicted deaths (4.67%), reflecting 0.38% over-prediction. Across deciles of predicted risk, the absolute difference in observed versus predicted percent mortality was a mean of 0.38%, with a maximum error of 1.81% seen in the highest-risk decile. CONCLUSIONS AND RELEVANCE The VA's SMR models, which incorporate patient physiology on presentation, are highly predictive and demonstrate good calibration both overall and across risk deciles. The current SMR models perform similarly to the initial ICU SMR model, indicating appropriate adaption and re-calibration.
Collapse
Affiliation(s)
- Hallie C Prescott
- VA Center for Clinical Management Research, Ann Arbor, MI, USA. .,University of Michigan, Department of Medicine, Ann Arbor, MI, USA.
| | - Rajendra P Kadel
- VA Center for Strategic Analytics and Reporting, Department of Veterans Affairs, Veterans Health Administration, 810 Vermont Ave. NW Room 668, Washington, DC, 20420, USA
| | - Julie R Eyman
- VA Center for Strategic Analytics and Reporting, Department of Veterans Affairs, Veterans Health Administration, 810 Vermont Ave. NW Room 668, Washington, DC, 20420, USA
| | - Ron Freyberg
- VA Center for Strategic Analytics and Reporting, Department of Veterans Affairs, Veterans Health Administration, 810 Vermont Ave. NW Room 668, Washington, DC, 20420, USA
| | - Matthew Quarrick
- VA Center for Strategic Analytics and Reporting, Department of Veterans Affairs, Veterans Health Administration, 810 Vermont Ave. NW Room 668, Washington, DC, 20420, USA
| | - David Brewer
- VA Center for Strategic Analytics and Reporting, Department of Veterans Affairs, Veterans Health Administration, 810 Vermont Ave. NW Room 668, Washington, DC, 20420, USA
| | - Rachael Hasselbeck
- VA Inpatient Evaluation Center, Department of Veterans Affairs, Veterans Health Administration, 810 Vermont Ave. NW Room 668, Washington, DC, 20420, USA
| |
Collapse
|
5
|
Predicting mortality in the very old: a machine learning analysis on claims data. Sci Rep 2022; 12:17464. [PMID: 36261581 PMCID: PMC9581892 DOI: 10.1038/s41598-022-21373-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2021] [Accepted: 09/27/2022] [Indexed: 01/12/2023] Open
Abstract
Machine learning (ML) may be used to predict mortality. We used claims data from one large German insurer to develop and test differently complex ML prediction models, comparing them for their (balanced) accuracy, but also the importance of different predictors, the relevance of the follow-up period before death (i.e. the amount of accumulated data) and the time distance of the data used for prediction and death. A sample of 373,077 insured very old, aged 75 years or above, living in the Northeast of Germany in 2012 was drawn and followed over 6 years. Our outcome was whether an individual died in one of the years of interest (2013-2017) or not; the primary metric was (balanced) accuracy in a hold-out test dataset. From the 86,326 potential variables, we used the 30 most important ones for modeling. We trained a total of 45 model combinations: (1) Three different ML models were used; logistic regression (LR), random forest (RF), extreme gradient boosting (XGB); (2) Different periods of follow-up were employed for training; 1-5 years; (3) Different time distances between data used for prediction and the time of the event (death/survival) were set; 0-4 years. The mortality rate was 9.15% in mean per year. The models showed (balanced) accuracy between 65 and 93%. A longer follow-up period showed limited to no advantage, but models with short time distance from the event were more accurate than models trained on more distant data. RF and XGB were more accurate than LR. For RF and XGB sensitivity and specificity were similar, while for LR sensitivity was significantly lower than specificity. For all three models, the positive-predictive-value was below 62% (and even dropped to below 20% for longer time distances from death), while the negative-predictive-value significantly exceeded 90% for all analyses. The utilization of and costs for emergency transport as well as emergency and any hospital visits as well as the utilization of conventional outpatient care and laboratory services were consistently found most relevant for predicting mortality. All models showed useful accuracies, and more complex models showed advantages. The variables employed for prediction were consistent across models and with medical reasoning. Identifying individuals at risk could assist tailored decision-making and interventions.
Collapse
|
6
|
Chan AK, Shahrestani S, Ballatori AM, Orrico KO, Manley GT, Tarapore PE, Huang M, Dhall SS, Chou D, Mummaneni PV, DiGiorgio AM. Is the Centers for Medicare and Medicaid Services Hierarchical Condition Category Risk Adjustment Model Satisfactory for Quantifying Risk After Spine Surgery? Neurosurgery 2022; 91:123-131. [PMID: 35550453 PMCID: PMC9514755 DOI: 10.1227/neu.0000000000001980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2021] [Accepted: 01/12/2022] [Indexed: 11/19/2022] Open
Abstract
BACKGROUND The Centers for Medicare and Medicaid Services (CMS) hierarchical condition category (HCC) coding is a risk adjustment model that allows for the estimation of risk-and cost-associated with health care provision. Current models may not include key factors that fully delineate the risk associated with spine surgery. OBJECTIVE To augment CMS HCC risk adjustment methodology with socioeconomic data to improve its predictive capabilities for spine surgery. METHODS The National Inpatient Sample was queried for spinal fusion, and the data was merged with county-level coverage and socioeconomic status variables obtained from the Brookings Institute. We predicted outcomes (death, nonroutine discharge, length of stay [LOS], total charges, and perioperative complication) with pairs of hierarchical, mixed effects logistic regression models-one using CMS HCC score alone and another augmenting CMS HCC scores with demographic and socioeconomic status variables. Models were compared using receiver operating characteristic curves. Variable importance was assessed in conjunction with Wald testing for model optimization. RESULTS We analyzed 653 815 patients. Expanded models outperformed models using CMS HCC score alone for mortality, nonroutine discharge, LOS, total charges, and complications. For expanded models, variable importance analyses demonstrated that CMS HCC score was of chief importance for models of mortality, LOS, total charges, and complications. For the model of nonroutine discharge, age was the most important variable. For the model of total charges, unemployment rate was nearly as important as CMS HCC score. CONCLUSION The addition of key demographic and socioeconomic characteristics substantially improves the CMS HCC risk-adjustment models when modeling spinal fusion outcomes. This finding may have important implications for payers, hospitals, and policymakers.
Collapse
Affiliation(s)
- Andrew K. Chan
- Department of Neurological Surgery, University of California, San Francisco, California, USA
- Department of Neurosurgery, Duke University, Durham, North Carolina, USA
| | - Shane Shahrestani
- Department of Medical Engineering, California Institute of Technology, Pasadena, California, USA
- Keck School of Medicine, University of Southern California, Los Angeles, California, USA
| | - Alexander M. Ballatori
- Keck School of Medicine, University of Southern California, Los Angeles, California, USA
| | - Katie O. Orrico
- American Association of Neurological Surgeons/Congress of Neurological Surgeons Washington Office, Washington, District of Columbia, USA
| | - Geoffrey T. Manley
- Department of Neurological Surgery, University of California, San Francisco, California, USA
| | - Phiroz E. Tarapore
- Department of Neurological Surgery, University of California, San Francisco, California, USA
| | - Michael Huang
- Department of Neurological Surgery, University of California, San Francisco, California, USA
| | - Sanjay S. Dhall
- Department of Neurological Surgery, University of California, San Francisco, California, USA
| | - Dean Chou
- Department of Neurological Surgery, University of California, San Francisco, California, USA
| | - Praveen V. Mummaneni
- Department of Neurological Surgery, University of California, San Francisco, California, USA
| | - Anthony M. DiGiorgio
- Department of Neurological Surgery, University of California, San Francisco, California, USA
| |
Collapse
|
7
|
Glance LG, Benesch CG, Holloway RG, Thirukumaran CP, Nadler JW, Eaton MP, Fleming FJ, Dick AW. Association of Time Elapsed Since Ischemic Stroke With Risk of Recurrent Stroke in Older Patients Undergoing Elective Nonneurologic, Noncardiac Surgery. JAMA Surg 2022; 157:e222236. [PMID: 35767247 DOI: 10.1001/jamasurg.2022.2236] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/07/2023]
Abstract
Importance Perioperative strokes are a major cause of death and disability. There is limited information on which to base decisions for how long to delay elective nonneurologic, noncardiac surgery in patients with a history of stroke. Objective To examine whether an association exists between the time elapsed since an ischemic stroke and the risk of recurrent stroke in older patients undergoing elective nonneurologic, noncardiac surgery. Design, Setting, and Participants This cohort study used data from the 100% Medicare Provider Analysis and Review files, including the Master Beneficiary Summary File, between 2011 and 2018 and included elective nonneurologic, noncardiac surgeries in patients 66 years or older. Patients were excluded if they had more than 1 procedure during a 30-day period, were transferred from another hospital or facility, were missing information on race and ethnicity, were admitted in December 2018, or had tracheostomies or gastrostomies. Data were analyzed May 7 to October 23, 2021. Exposures Time interval between a previous hospital admission for acute ischemic stroke and surgery. Main Outcomes and Measures Acute ischemic stroke during the index surgical admission or rehospitalization for stroke within 30 days of surgery, 30-day all-cause mortality, composite of stroke and mortality, and discharge to a nursing home or skilled nursing facility. Multivariable logistic regression models were used to estimate adjusted odds ratios (AORs) to quantify the association between outcome and time since ischemic stroke. Results The final cohort included 5 841 539 patients who underwent elective nonneurologic, noncardiac surgeries (mean [SD] age, 74.1 [6.1] years; 3 371 329 [57.7%] women), of which 54 033 (0.9%) had a previous stroke. Patients with a stroke within 30 days before surgery had higher adjusted odds of perioperative stroke (AOR, 8.02; 95% CI, 6.37-10.10; P < .001) compared with patients without a previous stroke. The adjusted odds of stroke were not significantly different at an interval of 61 to 90 days between previous stroke and surgery (AOR, 5.01; 95% CI, 4.00-6.29; P < .001) compared with 181 to 360 days (AOR, 4.76; 95% CI, 4.26-5.32; P < .001). The adjusted odds of 30-day all-cause mortality were higher in patients who underwent surgery within 30 days of a previous stroke (AOR, 2.51; 95% CI, 1.99-3.16; P < .001) compared with those without a history of stroke, and the AOR decreased to 1.49 (95% CI, 1.15-1.92; P < .001) at 61 to 90 days from previous stroke to surgery but did not decline significantly, even after an interval of 360 or more days. Conclusions and Relevance The findings of this cohort study suggest that, among patients undergoing nonneurologic, noncardiac surgery, the risk of stroke and death leveled off when more than 90 days elapsed between a previous stroke and elective surgery. These findings suggest that the recent scientific statement by the American Heart Association to delay elective nonneurologic, noncardiac surgery for at least 6 months after a recent stroke may be too conservative.
Collapse
Affiliation(s)
- Laurent G Glance
- Department of Anesthesiology and Perioperative Medicine, University of Rochester School of Medicine, Rochester, New York.,Department of Public Health Sciences, University of Rochester School of Medicine, Rochester, New York.,RAND Health, RAND, Boston, Massachusetts.,Department of Anesthesiology and Perioperative Medicine, University of Rochester Medical Center, Rochester, New York
| | - Curtis G Benesch
- Department of Neurology, University of Rochester School of Medicine, Rochester, New York
| | - Robert G Holloway
- Department of Neurology, University of Rochester School of Medicine, Rochester, New York
| | - Caroline P Thirukumaran
- Department of Public Health Sciences, University of Rochester School of Medicine, Rochester, New York.,Department of Orthopedics, University of Rochester School of Medicine, Rochester, New York
| | - Jacob W Nadler
- Department of Anesthesiology and Perioperative Medicine, University of Rochester School of Medicine, Rochester, New York
| | - Michael P Eaton
- Department of Anesthesiology and Perioperative Medicine, University of Rochester School of Medicine, Rochester, New York
| | - Fergal J Fleming
- Department of Surgery, University of Rochester School of Medicine, Rochester, New York
| | | |
Collapse
|
8
|
Saini V, Gopinath V. Application of the Risk Stratification Index to Multilevel Models of All-condition 30-Day Mortality in Hospitalized Populations Over the Age of 65. Med Care 2021; 59:836-842. [PMID: 33989249 PMCID: PMC8360662 DOI: 10.1097/mlr.0000000000001570] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
BACKGROUND The Risk Stratification Index (RSI) is superior to Hierarchical Conditions Categories (HCC) in patient-level regressions but has not been applied to assess hospital effects. OBJECTIVE The objective of this study was to measure the accuracy of RSI in modeling 30-day hospital mortality across all conditions using multilevel logistic regression. SUBJECTS AND DATA SOURCES A 100% sample of Medicare inpatient stays from 2009 to 2014, restricted to patients greater than 65 years of age in general hospitals, resulting in 64 million stays at 3504 hospitals. RESEARCH DESIGN We calculated RSI and HCC scores for patient stays using multilevel logistic regression in 3 populations: all inpatients, surgical, and nonsurgical. Correlations of risk-standardized mortality rates with rates of specific case types assessed case-mix balance. Patient stay volume was included to assess smaller hospitals. RESULTS We found a negligible correlation of all-conditions risk-standardized mortality rates with hospitals' proportions of orthopedic, cardiac, or pneumonia cases. RSI outperformed HCC in multilevel regressions containing both patient and hospital-level effects. C-statistics using RSI were 0.87 for the all-inpatients group, 0.87 for surgical, and 0.86 for nonsurgical stays. With HCC they were 0.82, 0.82, and 0.81. Akaike Information Criteria and Bayesian Information Criteria values were higher with HCC. RSI shifted 41% of hospitals' rankings by >1 decile. Hospitals with smaller volumes had higher 30-day observed and standardized mortality: 11.2% in the lowest volume quintile versus 8.5% in the highest volume quintile. CONCLUSION RSI has superior accuracy and results in a significant shift in rankings compared with HCC in multilevel models of 30-day hospital mortality across all conditions.
Collapse
|
9
|
Application of machine learning approaches to administrative claims data to predict clinical outcomes in medical and surgical patient populations. PLoS One 2021; 16:e0252585. [PMID: 34081720 PMCID: PMC8174683 DOI: 10.1371/journal.pone.0252585] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2021] [Accepted: 05/18/2021] [Indexed: 12/16/2022] Open
Abstract
Objective This study aimed to develop and validate a claims-based, machine learning algorithm to predict clinical outcomes across both medical and surgical patient populations. Methods This retrospective, observational cohort study, used a random 5% sample of 770,777 fee-for-service Medicare beneficiaries with an inpatient hospitalization between 2009–2011. The machine learning algorithms tested included: support vector machine, random forest, multilayer perceptron, extreme gradient boosted tree, and logistic regression. The extreme gradient boosted tree algorithm outperformed the alternatives and was the machine learning method used for the final risk model. Primary outcome was 30-day mortality. Secondary outcomes were: rehospitalization, and any of 23 adverse clinical events occurring within 30 days of the index admission date. Results The machine learning algorithm performance was evaluated by both the area under the receiver operating curve (AUROC) and Brier Score. The risk model demonstrated high performance for prediction of: 30-day mortality (AUROC = 0.88; Brier Score = 0.06), and 17 of the 23 adverse events (AUROC range: 0.80–0.86; Brier Score range: 0.01–0.05). The risk model demonstrated moderate performance for prediction of: rehospitalization within 30 days (AUROC = 0.73; Brier Score: = 0.07) and six of the 23 adverse events (AUROC range: 0.74–0.79; Brier Score range: 0.01–0.02). The machine learning risk model performed comparably on a second, independent validation dataset, confirming that the risk model was not overfit. Conclusions and relevance We have developed and validated a robust, claims-based, machine learning risk model that is applicable to both medical and surgical patient populations and demonstrates comparable predictive accuracy to existing risk models.
Collapse
|
10
|
Triche EW, Xin X, Stackland S, Purvis D, Harris A, Yu H, Grady JN, Li SX, Bernheim SM, Krumholz HM, Poyer J, Dorsey K. Incorporating Present-on-Admission Indicators in Medicare Claims to Inform Hospital Quality Measure Risk Adjustment Models. JAMA Netw Open 2021; 4:e218512. [PMID: 33978722 PMCID: PMC8116982 DOI: 10.1001/jamanetworkopen.2021.8512] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/03/2020] [Accepted: 03/11/2021] [Indexed: 11/14/2022] Open
Abstract
Importance Present-on-admission (POA) indicators in administrative claims data allow researchers to distinguish between preexisting conditions and those acquired during a hospital stay. The impact of adding POA information to claims-based measures of hospital quality has not yet been investigated to better understand patient underlying risk factors in the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision setting. Objective To assess POA indicator use on Medicare claims and to assess the hospital- and patient-level outcomes associated with incorporating POA indicators in identifying risk factors for publicly reported outcome measures used by the Centers for Medicare & Medicaid Services (CMS). Design, Setting, and Participants This comparative effectiveness study used national CMS claims data between July 1, 2015, and June 30, 2018. Six hospital quality measures assessing readmission and mortality outcomes were modified to include POA indicators in risk adjustment models. The models using POA were then compared with models using the existing complications-of-care algorithm to evaluate changes in risk model performance. Patient claims data were included for all Medicare fee-for-service and Veterans Administration beneficiaries aged 65 years or older with inpatient hospitalizations for acute myocardial infarction, heart failure, or pneumonia within the measurement period. Data were analyzed between September 2019 and March 2020. Main Outcomes and Measures Changes in patient-level (C statistics) and hospital-level (quintile shifts in risk-standardized outcome rates) model performance after including POA indicators in risk adjustment. Results Data from a total of 6 027 988 index admissions were included for analysis, ranging from 491 366 admissions (269 209 [54.8%] men; mean [SD] age, 78.2 [8.3] years) for the acute myocardial infarction mortality outcome measure to 1 395 870 admissions (677 158 [48.5%] men; mean [SD] age, 80.3 [8.7] years) for the pneumonia readmission measure. Use of POA indicators was associated with improvements in risk adjustment model performance, particularly for mortality measures (eg, the C statistic increased from 0.728 [95% CI, 0.726-0.730] to 0.774 [95% CI, 0.773-0.776] when incorporating POA indicators into the acute myocardial infarction mortality measure). Conclusions and Relevance The findings of this quality improvement study suggest that leveraging POA indicators in the risk adjustment methodology for hospital quality outcome measures may help to more fully capture patients' risk factors and improve overall model performance. Incorporating POA indicators does not require extra effort on the part of hospitals and would be easy to implement in publicly reported quality outcome measures.
Collapse
Affiliation(s)
- Elizabeth W. Triche
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
| | - Xin Xin
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Sydnie Stackland
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
| | - Danielle Purvis
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
| | - Alexandra Harris
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
| | - Huihui Yu
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Jacqueline N. Grady
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
| | - Shu-Xia Li
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
| | - Susannah M. Bernheim
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
- Section of General Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Harlan M. Krumholz
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
- Department of Health Policy and Administration, Yale School of Public Health, New Haven, Connecticut
| | - James Poyer
- Centers for Medicare & Medicaid Services (CMS), Woodlawn, Maryland
| | - Karen Dorsey
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
- Section of General Pediatrics, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
| |
Collapse
|
11
|
Predicting mortality and hospitalization in heart failure using machine learning: A systematic literature review. IJC HEART & VASCULATURE 2021; 34:100773. [PMID: 33912652 PMCID: PMC8065274 DOI: 10.1016/j.ijcha.2021.100773] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2021] [Revised: 03/11/2021] [Accepted: 03/23/2021] [Indexed: 12/13/2022]
Abstract
Objective The partnership between humans and machines can enhance clinical decisions accuracy, leading to improved patient outcomes. Despite this, the application of machine learning techniques in the healthcare sector, particularly in guiding heart failure patient management, remains unpopular. This systematic review aims to identify factors restricting the integration of machine learning derived risk scores into clinical practice when treating adults with acute and chronic heart failure. Methods Four academic research databases and Google Scholar were searched to identify original research studies where heart failure patient data was used to build models predicting all-cause mortality, cardiac death, all-cause and heart failure-related hospitalization. Results Thirty studies met the inclusion criteria. The selected studies' sample size ranged between 71 and 716 790 patients, and the median age was 72.1 (interquartile range: 61.1–76.8) years. The minimum and maximum area under the receiver operating characteristic curve (AUC) for models predicting mortality were 0.48 and 0.92, respectively. Models predicting hospitalization had an AUC of 0.47 to 0.84. Nineteen studies (63%) used logistic regression, 53% random forests, and 37% of studies used decision trees to build predictive models. None of the models were built or externally validated using data originating from Africa or the Middle-East. Conclusions The variation in the aetiologies of heart failure, limited access to structured health data, distrust in machine learning techniques among clinicians and the modest accuracy of existing predictive models are some of the factors precluding the widespread use of machine learning derived risk calculators.
Collapse
|
12
|
Logistic regression and machine learning predicted patient mortality from large sets of diagnosis codes comparably. J Clin Epidemiol 2021; 133:43-52. [PMID: 33359319 DOI: 10.1016/j.jclinepi.2020.12.018] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2020] [Revised: 11/18/2020] [Accepted: 12/15/2020] [Indexed: 12/20/2022]
Abstract
OBJECTIVE The objective of the study was to compare the performance of logistic regression and boosted trees for predicting patient mortality from large sets of diagnosis codes in electronic healthcare records. STUDY DESIGN AND SETTING We analyzed national hospital records and official death records for patients with myocardial infarction (n = 200,119), hip fracture (n = 169,646), or colorectal cancer surgery (n = 56,515) in England in 2015-2017. One-year mortality was predicted from patient age, sex, and socioeconomic status, and 202 to 257 International Classification of Diseases 10th Revision codes recorded in the preceding year or not (binary predictors). Performance measures included the c-statistic, scaled Brier score, and several measures of calibration. RESULTS One-year mortality was 17.2% (34,520) after myocardial infarction, 27.2% (46,115) after hip fracture, and 9.3% (5,273) after colorectal surgery. Optimism-adjusted c-statistics for the logistic regression models were 0.884 (95% confidence interval [CI]: 0.882, 0.886), 0.798 (0.796, 0.800), and 0.811 (0.805, 0.817). The equivalent c-statistics for the boosted tree models were 0.891 (95% CI: 0.889, 0.892), 0.804 (0.802, 0.806), and 0.803 (0.797, 0.809). Model performance was also similar when measured using scaled Brier scores. All models were well calibrated overall. CONCLUSION In large datasets of electronic healthcare records, logistic regression and boosted tree models of numerous diagnosis codes predicted patient mortality comparably.
Collapse
|
13
|
Cowling TE, Cromwell DA, Sharples LD, van der Meulen J. A novel approach selected small sets of diagnosis codes with high prediction performance in large healthcare datasets. J Clin Epidemiol 2020; 128:20-28. [DOI: 10.1016/j.jclinepi.2020.08.001] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2020] [Revised: 07/15/2020] [Accepted: 08/05/2020] [Indexed: 12/23/2022]
|
14
|
Zhou Y, Hou Y, Hussain M, Brown SA, Budd T, Tang WHW, Abraham J, Xu B, Shah C, Moudgil R, Popovic Z, Cho L, Kanj M, Watson C, Griffin B, Chung MK, Kapadia S, Svensson L, Collier P, Cheng F. Machine Learning-Based Risk Assessment for Cancer Therapy-Related Cardiac Dysfunction in 4300 Longitudinal Oncology Patients. J Am Heart Assoc 2020; 9:e019628. [PMID: 33241727 PMCID: PMC7763760 DOI: 10.1161/jaha.120.019628] [Citation(s) in RCA: 25] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
Abstract
Background The growing awareness of cardiovascular toxicity from cancer therapies has led to the emerging field of cardio-oncology, which centers on preventing, detecting, and treating patients with cardiac dysfunction before, during, or after cancer treatment. Early detection and prevention of cancer therapy-related cardiac dysfunction (CTRCD) play important roles in precision cardio-oncology. Methods and Results This retrospective study included 4309 cancer patients between 1997 and 2018 whose laboratory tests and cardiovascular echocardiographic variables were collected from the Cleveland Clinic institutional electronic medical record database (Epic Systems). Among these patients, 1560 (36%) were diagnosed with at least 1 type of CTRCD, and 838 (19%) developed CTRCD after cancer therapy (de novo). We posited that machine learning algorithms can be implemented to predict CTRCDs in cancer patients according to clinically relevant variables. Classification models were trained and evaluated for 6 types of cardiovascular outcomes, including coronary artery disease (area under the receiver operating characteristic curve [AUROC], 0.821; 95% CI, 0.815-0.826), atrial fibrillation (AUROC, 0.787; 95% CI, 0.782-0.792), heart failure (AUROC, 0.882; 95% CI, 0.878-0.887), stroke (AUROC, 0.660; 95% CI, 0.650-0.670), myocardial infarction (AUROC, 0.807; 95% CI, 0.799-0.816), and de novo CTRCD (AUROC, 0.802; 95% CI, 0.797-0.807). Model generalizability was further confirmed using time-split data. Model inspection revealed several clinically relevant variables significantly associated with CTRCDs, including age, hypertension, glucose levels, left ventricular ejection fraction, creatinine, and aspartate aminotransferase levels. Conclusions This study suggests that machine learning approaches offer powerful tools for cardiac risk stratification in oncology patients by utilizing large-scale, longitudinal patient data from healthcare systems.
Collapse
Affiliation(s)
- Yadi Zhou
- Genomic Medicine Institute Lerner Research InstituteCleveland Clinic Cleveland OH
| | - Yuan Hou
- Genomic Medicine Institute Lerner Research InstituteCleveland Clinic Cleveland OH
| | - Muzna Hussain
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH.,School of Medicine Dentistry and Biomedical Sciences Wellcome-Wolfson Institute of Experimental MedicineQueen's University Belfast United Kingdom
| | - Sherry-Ann Brown
- Cardio-Oncology Program Division of Cardiovascular Medicine Medical College of Wisconsin Milwaukee WI
| | - Thomas Budd
- Department of Hematology/Medical Oncology Taussig Cancer InstituteCleveland Clinic Cleveland OH
| | - W H Wilson Tang
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH.,Department of Molecular Medicine Cleveland Clinic Lerner College of MedicineCase Western Reserve University Cleveland OH
| | - Jame Abraham
- Department of Hematology/Medical Oncology Taussig Cancer InstituteCleveland Clinic Cleveland OH
| | - Bo Xu
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH
| | - Chirag Shah
- Department of Radiation Oncology Taussig Cancer InstituteCleveland Clinic Cleveland OH
| | - Rohit Moudgil
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH
| | - Zoran Popovic
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH
| | - Leslie Cho
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH
| | - Mohamed Kanj
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH
| | - Chris Watson
- School of Medicine Dentistry and Biomedical Sciences Wellcome-Wolfson Institute of Experimental MedicineQueen's University Belfast United Kingdom
| | - Brian Griffin
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH
| | - Mina K Chung
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH.,Department of Molecular Medicine Cleveland Clinic Lerner College of MedicineCase Western Reserve University Cleveland OH
| | - Samir Kapadia
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH
| | - Lars Svensson
- Department of Cardiovascular Surgery Cleveland Clinic Cleveland OH
| | - Patrick Collier
- Robert and Suzanne Tomsich Department of Cardiovascular Medicine Sydell and Arnold Miller Family Heart and Vascular Institute Cleveland Clinic Cleveland OH.,Department of Molecular Medicine Cleveland Clinic Lerner College of MedicineCase Western Reserve University Cleveland OH
| | - Feixiong Cheng
- Genomic Medicine Institute Lerner Research InstituteCleveland Clinic Cleveland OH.,Department of Hematology/Medical Oncology Taussig Cancer InstituteCleveland Clinic Cleveland OH.,Case Comprehensive Cancer Center Case Western Reserve University School of Medicine Cleveland OH
| |
Collapse
|
15
|
Husaini M, Joynt Maddox KE. Paying for Performance Improvement in Quality and Outcomes of Cardiovascular Care: Challenges and Prospects. Methodist Debakey Cardiovasc J 2020; 16:225-231. [PMID: 33133359 DOI: 10.14797/mdcj-16-3-225] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022] Open
Abstract
Over the past two decades, Medicare and other payers have been looking at ways to base payment for cardiovascular care on the quality and outcomes of care delivered. Public reporting of hospital performance on a series of quality measures began in 2004 with basic processes of care such as aspirin use and influenza vaccination, and it expanded in later years to include outcomes such as mortality and readmission rates. Following the passage of the Affordable Care Act in March 2010, Medicare and other payers moved forward with pay-for-performance programs, more commonly referred to as value-based purchasing (VBP) programs. These programs are largely based on an underlying fee-for-service payment infrastructure and give hospitals and clinicians bonuses or penalties based on their performance. Another new payment mechanism, called alternative payment models (APMs), aims to move towards episode-based or global payments to improve quality and efficiency. The two most relevant APMs for cardiovascular care include Accountable Care Organizations and bundled payments. Both VBP programs and APMs have challenges related to program efficacy, accuracy, and equity. In fact, despite over a decade of progress in measuring and incentivizing high-quality care delivery within cardiology, major limitations remain. Many of the programs have had little benefit in terms of clinical outcomes yet have led to marked administrative burden for participants. However, there are several encouraging prospects to aid the successful implementation of value-based high-quality cardiovascular care, such as more sophisticated data science to improve risk adjustment and flexible electronic health records to decrease administrative burden. Furthermore, payment models designed specifically for cardiovascular care could incentivize innovative care delivery models that could improve quality and outcomes for patients. This review provides an overview of current efforts, largely at the federal level, to pay for high-quality cardiovascular care and discusses the challenges and prospects related to doing so.
Collapse
Affiliation(s)
- Mustafa Husaini
- WASHINGTON UNIVERSITY SCHOOL OF MEDICINE, ST. LOUIS, MISSOURI
| | - Karen E Joynt Maddox
- WASHINGTON UNIVERSITY SCHOOL OF MEDICINE, ST. LOUIS, MISSOURI.,INSTITUTE FOR PUBLIC HEALTH AT WASHINGTON UNIVERSITY, ST. LOUIS, MISSOURI
| |
Collapse
|
16
|
Ellis RP, Hsu HE, Song C, Kuo TC, Martins B, Siracuse JJ, Liu Y, Ash AS. Diagnostic Category Prevalence in 3 Classification Systems Across the Transition to the International Classification of Diseases, Tenth Revision, Clinical Modification. JAMA Netw Open 2020; 3:e202280. [PMID: 32267514 PMCID: PMC7142382 DOI: 10.1001/jamanetworkopen.2020.2280] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/15/2022] Open
Abstract
Importance On October 1, 2015, the US transitioned to the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) for recording diagnoses, symptoms, and procedures. It is unknown whether this transition was associated with changes in diagnostic category prevalence based on diagnosis classification systems commonly used for payment and quality reporting. Objective To assess changes in diagnostic category prevalence associated with the ICD-10-CM transition. Design, Setting, and Participants This interrupted time series analysis and cross-sectional study examined level and trend changes in diagnostic category prevalence associated with the ICD-10-CM transition and clinically reviewed a subset of diagnostic categories with changes of 20% or more. Data included insurance claim diagnoses from the IBM MarketScan Commercial Database from January 1, 2010, to December 31, 2017, for more than 18 million people aged 0 to 64 years with private insurance. Diagnoses were mapped using 3 common diagnostic classification systems: World Health Organization (WHO) disease chapters, Department of Health and Human Services Hierarchical Condition Categories (HHS-HCCs), and Agency for Healthcare Research and Quality Clinical Classification System (AHRQ-CCS). Data were analyzed from December 1, 2018, to January 21, 2020. Exposures US implementation of ICD-10-CM. Main Outcomes and Measures Monthly rates of individuals with at least 1 diagnosis in a diagnostic classification category per 10 000 eligible members. Results The analytic sample contained information on 2.1 billion enrollee person-months with 3.4 billion clinically assigned diagnoses; the mean (range) monthly sample size was 22.1 (18.4 to 27.1 ) million individuals. While diagnostic category prevalence changed minimally for WHO disease chapters, the ICD-10-CM transition was associated with level changes of 20% or more among 20 of 127 HHS-HCCs (15.7%) and 46 of 282 AHRQ-CCS categories (16.3%) and with trend changes of 20% or more among 12 of 127 of HHS-HCCs (9.4%) and 27 of 282 of AHRQ-CCS categories (9.6%). For HHS-HCCs, monthly rates of individuals with any acute myocardial infarction diagnosis increased 131.5% (95% CI, 124.1% to 138.8%), primarily because HHS added non-ST-segment-elevation myocardial infarction diagnoses to this category. The HHS-HCC for diabetes with chronic complications increased by 92.4% (95% CI, 84.2% to 100.5%), primarily from including new diabetes-related hypoglycemia and hyperglycemia codes, and the rate for completed pregnancy with complications decreased by 54.5% (95% CI, -58.7% to -50.2%) partly due to removing vaginal birth after cesarean delivery as a complication. Conclusions and Relevance These findings suggest that the ICD-10-CM transition was associated with large prevalence changes for many diagnostic categories. Diagnostic classification systems developed using ICD-9-CM may need to be refined using ICD-10-CM data to avoid unintended consequences for disease surveillance, performance assessment, and risk-adjusted payments.
Collapse
Affiliation(s)
- Randall P Ellis
- Department of Economics, Boston University, Boston, Massachusetts
| | - Heather E Hsu
- Department of Pediatrics, Boston University School of Medicine, Boston, Massachusetts
| | - Chenlu Song
- Department of Economics, Boston University, Boston, Massachusetts
| | | | | | - Jeffrey J Siracuse
- Division of Vascular and Endovascular Surgery, Boston Medical Center, Boston University School of Medicine, Boston, Massachusetts
| | - Ying Liu
- Department of Economics, Boston University, Boston, Massachusetts
| | - Arlene S Ash
- Department of Population and Quantitative Health Sciences, University of Massachusetts Medical School, Worcester
| |
Collapse
|
17
|
Krumholz HM, Warner F, Coppi A, Triche EW, Li SX, Mahajan S, Li Y, Bernheim SM, Grady J, Dorsey K, Desai NR, Lin Z, Normand SLT. Development and Testing of Improved Models to Predict Payment Using Centers for Medicare & Medicaid Services Claims Data. JAMA Netw Open 2019; 2:e198406. [PMID: 31411709 PMCID: PMC6694388 DOI: 10.1001/jamanetworkopen.2019.8406] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/11/2019] [Accepted: 06/11/2019] [Indexed: 11/14/2022] Open
Abstract
Importance Predicting payments for particular conditions or populations is essential for research, benchmarking, public reporting, and calculations for population-based programs. Centers for Medicare & Medicaid Services (CMS) models often group codes into disease categories, but using single, rather than grouped, diagnostic codes and leveraging present on admission (POA) codes may enhance these models. Objective To determine whether changes to the candidate variables in CMS models would improve risk models predicting patient total payment within 30 days of hospitalization for acute myocardial infarction (AMI), heart failure (HF), and pneumonia. Design, Setting, and Participants This comparative effectiveness research study used data from Medicare fee-for-service hospitalizations for AMI, HF, and pneumonia at acute care hospitals from July 1, 2013, through September 30, 2015. Payments across multiple care settings, services, and supplies were included and adjusted for geographic and policy variations, corrected for inflation, and winsorized. The same data source was used but varied for the candidate variables and their selection, and the method used by CMS for public reporting that used grouped codes was compared with variations that used POA codes and single diagnostic codes. Combinations of use of POA codes, separation of index admission diagnoses from those in the previous 12 months, and use of individual International Classification of Diseases, Ninth Revision, Clinical Modification codes instead of grouped diagnostic categories were tested. Data analysis was performed from December 4, 2017, to June 10, 2019. Main Outcomes and Measures The models' goodness of fit was compared using root mean square error (RMSE) and the McFadden pseudo R2. Results Among the 1 943 049 total hospitalizations of the study participants, 343 116 admissions were for AMI (52.5% male; 37.4% aged ≤74 years), 677 044 for HF (45.5% male; 25.9% aged ≤74 years), and 922 889 for pneumonia (46.4% male; 28.2% aged ≤74 years). The mean (SD) 30-day payment was $23 103 ($18 221) for AMI, $16 365 ($12 527) for HF, and $17 097 ($12 087) for pneumonia. Each incremental model change improved the pseudo R2 and RMSE. Incorporating all 3 changes improved the pseudo R2 of the patient-level models from 0.077 to 0.129 for AMI, from 0.042 to 0.129 for HF, and from 0.114 to 0.237 for pneumonia. Parallel improvements in RMSE were found for all 3 conditions. Conclusions and Relevance Leveraging POA codes, separating index from previous diagnoses, and using single diagnostic codes improved payment models. Better models can potentially improve research, benchmarking, public reporting, and calculations for population-based programs.
Collapse
Affiliation(s)
- Harlan M. Krumholz
- Department of Health Policy and Management, Yale School of Public Health, New Haven, Connecticut
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Frederick Warner
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
| | - Andreas Coppi
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
| | - Elizabeth W. Triche
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
| | - Shu-Xia Li
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
| | - Shiwani Mahajan
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Yixin Li
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
| | - Susannah M. Bernheim
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
- Section of General Internal Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Jacqueline Grady
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
| | - Karen Dorsey
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
- Section of General Pediatrics, Department of Pediatrics, Yale School of Medicine, New Haven, Connecticut
| | - Nihar R. Desai
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Zhenqiu Lin
- Center for Outcomes Research and Evaluation, Yale–New Haven Hospital, New Haven, Connecticut
| | - Sharon-Lise T. Normand
- Department of Biostatistics, Harvard T.H. Chan School of Public Health, Harvard University, Boston, Massachusetts
- Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts
| |
Collapse
|
18
|
Vaughan Sarrazin MS, Girotra S. Exact Science and the Art of Approximating Quality in Hospital Performance Metrics. JAMA Netw Open 2019; 2:e197321. [PMID: 31314112 DOI: 10.1001/jamanetworkopen.2019.7321] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/05/2023] Open
Affiliation(s)
| | - Saket Girotra
- Department of Internal Medicine, University of Iowa, Iowa City
| |
Collapse
|