Development and validation of an interpretable prehospital return of spontaneous circulation (P-ROSC) score for patients with out-of-hospital cardiac arrest using machine learning: A retrospective study

Post-Cardiac arrest outcome prediction using machine learning: A systematic review and meta-analysis

BACKGROUND

Early and reliable prognostication in post-cardiac arrest patients remains challenging, with various factors linked to return of spontaneous circulation (ROSC), survival, and neurological results. Machine learning and deep learning models show promise in improving these predictions. This systematic review and meta-analysis evaluates how effective these approaches are in predicting clinical outcomes at different time points using structured data.

METHODS

This study followed PRISMA guidelines, involving a comprehensive search across PubMed, Scopus, and Web of Science databases until March 2024. Studies aimed at predicting ROSC, survival (or mortality), and neurological outcomes after cardiac arrest through the application of machine learning or deep learning techniques with structured data were included. Data extraction followed the guidelines of the CHARMS checklist, and the bias risk was evaluated using PROBAST tool. Models reporting the AUC metric with 95 % confidence intervals were incorporated into the quantitative synthesis and meta-analysis.

RESULTS

After extracting 2,753 initial records, 41 studies met the inclusion criteria, yielding 97 machine learning and 16 deep learning models. The pooled AUC for predicting favorable neurological outcomes (CPC 1 or 2) at hospital discharge was 0.871 (95 % CI: 0.813 - 0.928) for machine learning models and 0.877 (95 % CI: 0.831-0.924) across deep learning algorithms. For survival prediction, this value was found to be 0.837 (95 % CI: 0.757-0.916). Considerable heterogeneity and high risk of bias were observed, mainly attributable to inadequate management of missing data and the absence of calibration plots. Most studies focused on pre-hospital factors, with age, sex, and initial arrest rhythm being the most frequent features.

CONCLUSION

Predictive models utilizing AI-based approaches, including machine and deep learning models exhibit enhanced effectiveness compared to previous regression algorithms, but significant heterogeneity and high risk of bias limit their dependability. Evaluating state-of-the-art deep learning models tailored for tabular data and their clinical generalizability can enhance outcome prediction after cardiac arrest.

External Validation and Comparison of Statistical and Machine Learning-Based Models in Predicting Outcomes Following Out-of-Hospital Cardiac Arrest: A Multicenter Retrospective Analysis

BACKGROUND

The aim of this study was to validate and compare the performance of statistical (Utstein-Based Return of Spontaneous Circulation and Shockable Rhythm-Witness-Age-pH) and machine learning-based (Prehospital Return of Spontaneous Circulation and Swedish Cardiac Arrest Risk Score) models in predicting the outcomes following out-of-hospital cardiac arrest and to assess the impact of the COVID-19 pandemic on the models' performance.

METHODS AND RESULTS

This retrospective analysis included adult patients with out-of-hospital cardiac arrest treated at 3 academic hospitals between 2015 and 2023. The primary outcome was neurological outcomes at hospital discharge. Patients were divided into pre- (2015-2019) and post-2020 (2020-2023) subgroups to examine the effect of the COVID-19 pandemic on out-of-hospital cardiac arrest outcome prediction. The models' performance was evaluated using the area under the receiver operating characteristic curve and compared by the DeLong test. The analysis included 2161 patients, 1241 (57.4%) of whom were resuscitated after 2020. The cohort had a median age of 69.2 years, and 1399 patients (64.7%) were men. Overall, 69 patients (3.2%) had neurologically intact survival. The area under the receiver operating characteristic curves for predicting neurological outcomes were 0.85 (95% CI, 0.83-0.87) for the Utstein-Based Return of Spontaneous Circulation score, 0.82 (95% CI, 0.81-0.84) for the Shockable Rhythm-Witness-Age-pH score, 0.79 (95% CI, 0.78-0.81) for the Prehospital Return of Spontaneous Circulation score, and 0.79 (95% CI, 0.77-0.81) for the Swedish Cardiac Arrest Risk Score model. The Utstein-Based Return of Spontaneous Circulation score significantly outperformed both the Prehospital Return of Spontaneous Circulation score (P<0.001) and the Swedish Cardiac Arrest Risk Score model (P=0.007). Subgroup analysis indicated no significant difference in predictive performance for patients resuscitated before versus after 2020.

CONCLUSIONS

In this external validation, both statistical and machine learning-based models demonstrated excellent and fair performance, respectively, in predicting neurological outcomes despite different model architectures. The predictive performance of all evaluated clinical scoring systems was not significantly influenced by the COVID-19 pandemic.

The short- and mid-term mortality trends in out-of-hospital cardiac arrest survivors: insights from a 5-year multicenter retrospective study in Taiwan

Background

The survival trend and factors influencing short- and mid-term mortality in Asian out-of-hospital cardiac arrest (OHCA) survivors should be elucidated. We performed survival analyses on days 3 and 30, hypothesizing decreased survival rates within the initial 3 days post-resuscitation. Additionally, variables linked to mortality at these two timepoints were examined.

Methods

We performed a retrospective analysis on adult nontraumatic OHCA survivors admitted to the National Taiwan University Hospital and its branches between 2017 and 2021. We collected the following variables from the NTUH-Integrative Medical Database: basic characteristics, cardiopulmonary resuscitation events, inotrope administration, and post-resuscitation management. The outcomes included 3- and 30-day mortality. Subgroup analyses with the Kaplan-Meier method explored the survival probability of the OHCA survivors and assessed differences in cumulative survival among subgroups. Cox proportional hazards model was used to estimate adjusted hazard ratios with 95% confidence interval.

Results

Of the 967 survivors, 273 (28.2%) and 604 (62.5%) died within 3 and 30 days, respectively. The 30-day survival curve after OHCA showed an uneven decline, with the most significant decrease within the first 3 days of admission. Various risk factors influence mortality at 3- and 30-day intervals. Although increased age, noncardiac etiology, and prolonged low-flow time increased mortality risks, bystander CPR, targeted temperature management, and continuous renal replacement therapy were associated with reduced mortality at 3- and 30-day timeframes.

Conclusion

Survival declined in most OHCA survivors within 3 days post-resuscitation. The risk factors associated with mortality at 3- and 30-day intervals varied in this population.

Validation of Utstein-Based score to predict return of spontaneous circulation (UB-ROSC) in patients with out-of-hospital cardiac arrest

BACKGROUND AND AIMS

The Utstein Based-ROSC (UB-ROSC) score has been developed to predict ROSC in OHCA victims. Aim of the study was to validate the UB-ROSC score using two Utstein-based OHCA registries: the SWiss REgistry of Cardiac Arrest (SWISSRECA) and the Lombardia Cardiac Arrest Registry (Lombardia CARe), northern Italy.

METHODS

Consecutive patients with OHCA of any etiology occurring between January 1st, 2019 and December 31st 2021 were included in this retrospective validation study. UB-ROSC score was computed for each patient and categorized in one of three subgroups: low, medium or high likelihood of ROSC according to the UB-ROSC cut-offs (≤-19; -18 to 12; ≥13). To assess the performance of the UB-ROSC score in this new cohort, we assessed both discrimination and calibration. The score was plotted against the survival to hospital admission.

RESULTS

A total of 12.577 patients were included in the study. A sustained ROSC was obtained in 2.719 patients (22%). The UB-ROSC model resulted well calibrated and showed a good discrimination (AUC 0.71, 95% CI 0.70-0.72). In the low likelihood subgroup of UB-ROSC, only 10% of patients achieved ROSC, whereas the proportion raised to 36% for a score between -18 and 12 (OR 5.0, 95% CI 2.9-8.6, p < 0.001) and to 85% for a score ≥13 (OR 49.4, 95% CI 14.3-170.6, p < 0.001).

CONCLUSIONS

UB-ROSC score represents a reliable tool to predict ROSC probability in OHCA patients. Its application may help the medical decision-making process, providing a realistic stratification of the probability for ROSC.

The Current Discussion Regarding End-of-Life Care for Patients with Out-of-Hospital Cardiac Arrest with Initial Non-Shockable Rhythm: A Narrative Review

Despite recent advances in resuscitation science, outcomes in patients with out-of-hospital cardiac arrest (OHCA) with initial non-shockable rhythm remains poor. Those with initial non-shockable rhythm have some epidemiological features, including the proportion of patients with a witnessed arrest, bystander cardiopulmonary resuscitation (CPR), age, and presumed etiology of cardiac arrest have been reported, which differ from those with initial shockable rhythm. The discussion regarding better end-of-life care for patients with OHCA is a major concern among citizens. As one of the efforts to avoid unwanted resuscitation, advance directive is recognized as a key intervention, safeguarding patient autonomy. However, several difficulties remain in enhancing the effective use of advance directives for patients with OHCA, including local regulation of their use, insufficient utilization of advance directives by emergency medical services at the scene, and a lack of established tools for discussing futility of resuscitation in advance care planning. In addition, prehospital termination of resuscitation is a common practice in many emergency medical service systems to assist clinicians in deciding whether to discontinue resuscitation. However, there are also several unresolved problems, including the feasibility of implementing the rules for several regions and potential missed survivors among candidates for prehospital termination of resuscitation. Further investigation to address these difficulties is warranted for better end-of-life care of patients with OHCA.

Artificial intelligence to support out-of-hospital cardiac arrest care: A scoping review

Background

Artificial intelligence (AI) has demonstrated significant potential in supporting emergency medical services personnel during out-of-hospital cardiac arrest (OHCA) care; however, the extent of research evaluating this topic is unknown. This scoping review examines the breadth of literature on the application of AI in early OHCA care.

Methods

We conducted a search of PubMed®, Embase, and Web of Science in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews guidelines. Articles focused on non-traumatic OHCA and published prior to January 18th, 2023 were included. Studies were excluded if they did not use an AI intervention (including machine learning, deep learning, or natural language processing), or did not utilize data from the prehospital phase of care.

Results

Of 173 unique articles identified, 54 (31%) were included after screening. Of these studies, 15 (28%) were from the year 2022 and with an increasing trend annually starting in 2019. The majority were carried out by multinational collaborations (20/54, 38%) with additional studies from the United States (10/54, 19%), Korea (5/54, 10%), and Spain (3/54, 6%). Studies were classified into three major categories including ECG waveform classification and outcome prediction (24/54, 44%), early dispatch-level detection and outcome prediction (7/54, 13%), return of spontaneous circulation and survival outcome prediction (15/54, 20%), and other (9/54, 16%). All but one study had a retrospective design.

Conclusions

A small but growing body of literature exists describing the use of AI to augment early OHCA care.

Prediction Models for Return of Spontaneous Circulation in Patients with Cardiac Arrest: A Systematic Review and Critical Appraisal

Objectives

Prediction models for the return of spontaneous circulation (ROSC) in patients with cardiac arrest play an important role in helping physicians evaluate the survival probability and providing medical decision-making reference. Although relevant models have been developed, their methodological rigor and model applicability are still unclear. Therefore, this study aims to summarize the evidence for ROSC prediction models and provide a reference for the development, validation, and application of ROSC prediction models.

Methods

PubMed, Cochrane Library, Embase, Elsevier, Web of Science, SpringerLink, Ovid, CNKI, Wanfang, and SinoMed were systematically searched for studies on ROSC prediction models. The search time limit was from the establishment of the database to August 30, 2022. Two reviewers independently screened the literature and extracted the data. The PROBAST was used to evaluate the quality of the included literature.

Results

A total of 8 relevant prediction models were included, and 6 models reported the AUC of 0.662-0.830 in the modeling population, which showed good overall applicability but high risk of bias. The main reasons were improper handling of missing values and variable screening, lack of external validation of the model, and insufficient information of overfitting. Age, gender, etiology, initial heart rhythm, EMS arrival time/BLS intervention time, location, bystander CPR, witnessed during sudden arrest, and ACLS duration/compression duration were the most commonly included predictors. Obvious chest injury, body temperature below 33°C, and possible etiologies were predictive factors for ROSC failure in patients with TOHCA. Age, gender, initial heart rhythm, reason for the hospital visit, length of hospital stay, and the location of occurrence in hospital were the predictors of ROSC in IHCA patients.

Conclusion

The performance of current ROSC prediction models varies greatly and has a high risk of bias, which should be selected with caution. Future studies can further optimize and externally validate the existing models.

Machine learning pre-hospital real-time cardiac arrest outcome prediction (PReCAP) using time-adaptive cohort model based on the Pan-Asian Resuscitation Outcome Study

To save time during transport, where resuscitation quality can degrade in a moving ambulance, it would be prudent to continue the resuscitation on scene if there is a high likelihood of ROSC occurring at the scene. We developed the pre-hospital real-time cardiac arrest outcome prediction (PReCAP) model to predict ROSC at the scene using prehospital input variables with time-adaptive cohort. The patient survival at discharge from the emergency department (ED), the 30-day survival rate, and the final Cerebral Performance Category (CPC) were secondary prediction outcomes in this study. The Pan-Asian Resuscitation Outcome Study (PAROS) database, which includes out-of-hospital cardiac arrest (OHCA) patients transferred by emergency medical service in Asia between 2009 and 2018, was utilized for this study. From the variables available in the PAROS database, we selected relevant variables to predict OHCA outcomes. Light gradient-boosting machine (LightGBM) was used to build the PReCAP model. Between 2009 and 2018, 157,654 patients in the PAROS database were enrolled in our study. In terms of prediction of ROSC on scene, the PReCAP had an AUROC score between 0.85 and 0.87. The PReCAP had an AUROC score between 0.91 and 0.93 for predicting survived to discharge from ED, and an AUROC score between 0.80 and 0.86 for predicting the 30-day survival. The PReCAP predicted CPC with an AUROC score ranging from 0.84 to 0.91. The feature importance differed with time in the PReCAP model prediction of ROSC on scene. Using the PAROS database, PReCAP predicted ROSC on scene, survival to discharge from ED, 30-day survival, and CPC for each minute with an AUROC score ranging from 0.8 to 0.93. As this model used a multi-national database, it might be applicable for a variety of environments and populations.

Establishment of a prediction model for prehospital return of spontaneous circulation in out-of-hospital patients with cardiac arrest

BACKGROUND

Out-of-hospital cardiac arrest (OHCA) is a leading cause of death worldwide.

AIM

To explore factors influencing prehospital return of spontaneous circulation (P-ROSC) in patients with OHCA and develop a nomogram prediction model.

METHODS

Clinical data of patients with OHCA in Shenzhen, China, from January 2012 to December 2019 were retrospectively analyzed. Least absolute shrinkage and selection operator (LASSO) regression and multivariate logistic regression were applied to select the optimal factors predicting P-ROSC in patients with OHCA. A nomogram prediction model was established based on these influencing factors. Discrimination and calibration were assessed using receiver operating characteristic (ROC) and calibration curves. Decision curve analysis (DCA) was used to evaluate the model's clinical utility.

RESULTS

Among the included 2685 patients with OHCA, the P-ROSC incidence was 5.8%. LASSO and multivariate logistic regression analyses showed that age, bystander cardiopulmonary resuscitation (CPR), initial rhythm, CPR duration, ventilation mode, and pathogenesis were independent factors influencing P-ROSC in these patients. The area under the ROC was 0.963. The calibration plot demonstrated that the predicted P-ROSC model was concordant with the actual P-ROSC. The good clinical usability of the prediction model was confirmed using DCA.

CONCLUSION

The nomogram prediction model could effectively predict the probability of P-ROSC in patients with OHCA.

FedScore: A privacy-preserving framework for federated scoring system development

OBJECTIVE

We propose FedScore, a privacy-preserving federated learning framework for scoring system generation across multiple sites to facilitate cross-institutional collaborations.

MATERIALS AND METHODS

The FedScore framework includes five modules: federated variable ranking, federated variable transformation, federated score derivation, federated model selection and federated model evaluation. To illustrate usage and assess FedScore's performance, we built a hypothetical global scoring system for mortality prediction within 30 days after a visit to an emergency department using 10 simulated sites divided from a tertiary hospital in Singapore. We employed a pre-existing score generator to construct 10 local scoring systems independently at each site and we also developed a scoring system using centralized data for comparison.

RESULTS

We compared the acquired FedScore model's performance with that of other scoring models using the receiver operating characteristic (ROC) analysis. The FedScore model achieved an average area under the curve (AUC) value of 0.763 across all sites, with a standard deviation (SD) of 0.020. We also calculated the average AUC values and SDs for each local model, and the FedScore model showed promising accuracy and stability with a high average AUC value which was closest to the one of the pooled model and SD which was lower than that of most local models.

CONCLUSION

This study demonstrates that FedScore is a privacy-preserving scoring system generator with potentially good generalizability.

Prehospital predicting factors using a decision tree model for patients with witnessed out-of-hospital cardiac arrest and an initial shockable rhythm

The effect of prehospital factors on favorable neurological outcomes remains unclear in patients with witnessed out-of-hospital cardiac arrest (OHCA) and a shockable rhythm. We developed a decision tree model for these patients by using prehospital factors. Using a nationwide OHCA registry database between 2005 and 2020, we retrospectively analyzed a cohort of 1,930,273 patients, of whom 86,495 with witnessed OHCA and an initial shockable rhythm were included. The primary endpoint was defined as favorable neurological survival (cerebral performance category score of 1 or 2 at 1 month). A decision tree model was developed from randomly selected 77,845 patients (development cohort) and validated in 8650 patients (validation cohort). In the development cohort, the presence of prehospital return of spontaneous circulation was the best predictor of favorable neurological survival, followed by the absence of adrenaline administration and age. The patients were categorized into 9 groups with probabilities of favorable neurological survival ranging from 5.7 to 70.8% (areas under the receiver operating characteristic curve of 0.851 and 0.844 in the development and validation cohorts, respectively). Our model is potentially helpful in stratifying the probability of favorable neurological survival in patients with witnessed OHCA and an initial shockable rhythm.

AI and machine learning in resuscitation: Ongoing research, new concepts, and key challenges

Aim

Artificial intelligence (AI) and machine learning (ML) are important areas of computer science that have recently attracted attention for their application to medicine. However, as techniques continue to advance and become more complex, it is increasingly challenging for clinicians to stay abreast of the latest research. This overview aims to translate research concepts and potential concerns to healthcare professionals interested in applying AI and ML to resuscitation research but who are not experts in the field.

Main text

We present various research including prediction models using structured and unstructured data, exploring treatment heterogeneity, reinforcement learning, language processing, and large-scale language models. These studies potentially offer valuable insights for optimizing treatment strategies and clinical workflows. However, implementing AI and ML in clinical settings presents its own set of challenges. The availability of high-quality and reliable data is crucial for developing accurate ML models. A rigorous validation process and the integration of ML into clinical practice is essential for practical implementation. We furthermore highlight the potential risks associated with self-fulfilling prophecies and feedback loops, emphasizing the importance of transparency, interpretability, and trustworthiness in AI and ML models. These issues need to be addressed in order to establish reliable and trustworthy AI and ML models.

Conclusion

In this article, we overview concepts and examples of AI and ML research in the resuscitation field. Moving forward, appropriate understanding of ML and collaboration with relevant experts will be essential for researchers and clinicians to overcome the challenges and harness the full potential of AI and ML in resuscitation.

A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes

The AutoScore framework can automatically generate data-driven clinical scores in various clinical applications. Here, we present a protocol for developing clinical scoring systems for binary, survival, and ordinal outcomes using the open-source AutoScore package. We describe steps for package installation, detailed data processing and checking, and variable ranking. We then explain how to iterate through steps for variable selection, score generation, fine-tuning, and evaluation to generate understandable and explainable scoring systems using data-driven evidence and clinical knowledge. For complete details on the use and execution of this protocol, please refer to Xie et al. (2020),1 Xie et al. (2022)2, Saffari et al. (2022)3 and the online tutorial https://nliulab.github.io/AutoScore/.

Development and Asian-wide validation of the Grade for Interpretable Field Triage (GIFT) for predicting mortality in pre-hospital patients using the Pan-Asian Trauma Outcomes Study (PATOS)

Background

Field triage is critical in injury patients as the appropriate transport of patients to trauma centers is directly associated with clinical outcomes. Several prehospital triage scores have been developed in Western and European cohorts; however, their validity and applicability in Asia remains unclear. Therefore, we aimed to develop and validate an interpretable field triage scoring systems based on a multinational trauma registry in Asia.

Methods

This retrospective and multinational cohort study included all adult transferred injury patients from Korea, Malaysia, Vietnam, and Taiwan between 2016 and 2018. The outcome of interest was a death in the emergency department (ED) after the patients' ED visit. Using these results, we developed the interpretable field triage score with the Korea registry using an interpretable machine learning framework and validated the score externally. The performance of each country's score was assessed using the area under the receiver operating characteristic curve (AUROC). Furthermore, a website for real-world application was developed using R Shiny.

Findings

The study population included 26,294, 9404, 673 and 826 transferred injury patients between 2016 and 2018 from Korea, Malaysia, Vietnam, and Taiwan, respectively. The corresponding rates of a death in the ED were 0.30%, 0.60%, 4.0%, and 4.6% respectively. Age and vital sign were found to be the significant variables for predicting mortality. External validation showed the accuracy of the model with an AUROC of 0.756-0.850.

Interpretation

The Grade for Interpretable Field Triage (GIFT) score is an interpretable and practical tool to predict mortality in field triage for trauma.

Funding

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (Grant Number: HI19C1328).

Sustained Return of Spontaneous Circulation Following Out-of-Hospital Cardiac Arrest; Developing a Predictive Model Based on Multivariate Analysis

Introduction

Identifying the predictive factors of sustained return of spontaneous circulation (ROSC) following out-of-hospital cardiac arrest (OHCA) will be helpful in management of these patients. This study aimed to develop a predictive model in this regard.

Methods

In a retrospective observational study, data of adult patients with OHCA, were collected from Vajira emergency medical services patient care report. Multiple logistic regression analysis with a regression coefficient was used to develop a predictive score for a sustained ROSC at the scene. Area under the receiver operating characteristic (ROC) curve (AUC) was used to validate the accuracy of the predictive score for a sustained ROSC.

Results

Independent factors associated with a sustained ROSC included cardiopulmonary resuscitation (CPR) duration < 30 min (adjusted odds ratio (AOR)= 5.05, 95% confidence interval (CI): 3.34-7.65; p < 0.001); advanced airway management with an endotracheal tube (AOR= 3.06, 95% CI: 1.77-5.31; p < 0.001); advanced airway management with laryngeal mask airway (AOR= 3.42, 95% CI: 1.02-11.46; p = 0.046); defibrillation (AOR = 2.05, 95% CI: 1.31-3.2; p = 0.002); Capillary blood glucose (CBG) level < 150 mg% (AOR= 1.95, 95% CI: 1.05-3.65; p = 0.035); CBG at least 150 mg% (AOR= 2.87, 95% CI: 1.56-5.29; p = 0.001); pupil reflex (AOR = 2.96, 95% CI: 1.1-7.96; p = 0.032); and response time at most 8 min (AOR= 1.66, 95% CI: 1.07-2.57; p = 0.023). These were developed into the pupil reflex, response time, advanced airway management, defibrillation, CBG, and CPR duration (PRAD-CCPR) score. The most accurate cutoff point of score using Youden's index was ≥ 6 with AUC of 0.759 (95% CI: 0.715-0.802; p < 0.001), sensitivity of 62.0% (95% CI: 51.2-71.9%), specificity of 75.7% (95% CI: 69.4-81.2%), positive predictive value of 51.8% (95% CI: 40.9-62.3%), and negative predictive value of 79.5% (95% CI: 73.5-84.6%).

Conclusion

An optimal PRAD-CCPR score of ≥ 6 provides an acceptable accuracy of 0.759 with sensitivity of 62.0% and specificity of 75.7% in prediction of sustained ROSC following OHCA. This predictive score might help CPR commanders to prognosticate the outcome of patients with OHCA at the scene.

AutoScore-Ordinal: an interpretable machine learning framework for generating scoring models for ordinal outcomes

Background

Risk prediction models are useful tools in clinical decision-making which help with risk stratification and resource allocations and may lead to a better health care for patients. AutoScore is a machine learning–based automatic clinical score generator for binary outcomes. This study aims to expand the AutoScore framework to provide a tool for interpretable risk prediction for ordinal outcomes.

Methods

The AutoScore-Ordinal framework is generated using the same 6 modules of the original AutoScore algorithm including variable ranking, variable transformation, score derivation (from proportional odds models), model selection, score fine-tuning, and model evaluation. To illustrate the AutoScore-Ordinal performance, the method was conducted on electronic health records data from the emergency department at Singapore General Hospital over 2008 to 2017. The model was trained on 70% of the data, validated on 10% and tested on the remaining 20%.

Results

This study included 445,989 inpatient cases, where the distribution of the ordinal outcome was 80.7% alive without 30-day readmission, 12.5% alive with 30-day readmission, and 6.8% died inpatient or by day 30 post discharge. Two point-based risk prediction models were developed using two sets of 8 predictor variables identified by the flexible variable selection procedure. The two models indicated reasonably good performance measured by mean area under the receiver operating characteristic curve (0.758 and 0.793) and generalized c-index (0.737 and 0.760), which were comparable to alternative models.

Conclusion

AutoScore-Ordinal provides an automated and easy-to-use framework for development and validation of risk prediction models for ordinal outcomes, which can systematically identify potential predictors from high-dimensional data.