Validation the performance of New York Sepsis Severity Score compared with Sepsis Severity Score in predicting hospital mortality among sepsis patients

Comparative Predictive Accuracies of the Simplified Mortality Score for the Intensive Care Unit, Sepsis Severity Score, and Standard Severity Scores for 90-day Mortality in Sepsis Patients

Background

The standard severity scores were used for predicting hospital mortality of intensive care unit (ICU) patients. Recently, the new predictive score, Simplified Mortality Score for the ICU (SMS-ICU), was developed for predicting 90-day mortality.

Objective

To validate the ability of the SMS-ICU and compare with sepsis severity score (SSS) and original severity scores for predicting 90-day mortality in sepsis patients.

Method

An analysis of retrospective data was conducted in the ICU of a university teaching hospital. Also, 90-day mortality was used for the primary outcome.

Results

A total of 1,161 patients with sepsis were included. The 90-day mortality was 42.4%. The SMS-ICU presented the area under the receiver operating characteristic curve (AUROC) of 0.71, whereas the SSS had significantly higher AUROC than that of the SMS-ICU (AUROC 0.876, p < 0.001). The acute physiology and chronic health evaluation (APACHE) II and IV, and the simplified acute physiology scores (SAPS) II demonstrated good discrimination, with an AUROC above 0.90. The SMS-ICU provides poor calibration for 90-day mortality prediction, similar to the SSS and other standard severity scores. Furthermore, 90-day mortality was underestimated by the SMS-ICU, which had a standardized mortality ratio (SMR) of 1.36. The overall performance by Brier score demonstrated that the SMS-ICU was inferior to the SSS (0.222 and 0.169, respectively). Also, SAPS II presented the best overall performance with a Brier score of 0.092.

Conclusion

The SMS-ICU indicated lower performance compared to the SSS, standard severity scores. Consequently, modifications are required to enhance the performance of the SMS-ICU.

How to cite this article

Sathaporn N, Khwannimit B. Comparative Predictive Accuracies of the Simplified Mortality Score for the Intensive Care Unit, Sepsis Severity Score, and Standard Severity Scores for 90-day Mortality in Sepsis Patients. Indian J Crit Care Med 2024;28(4):343-348.

A critical appraisal of the prognostic predictive models for patients with sepsis: Which model can be applied in clinical practice?

BACKGROUND

Sepsis is associated with high mortality and predictive models can help in clinical decision-making. The objective of this study was to carry out a systematic review of these models.

METHODS

In 2019, we conducted a systematic review in MEDLINE and EMBASE (CDR42018111121:PROSPERO) of articles that developed predictive models for mortality in septic patients (inclusion criteria). We followed the CHARMS recommendations (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies), extracting the information from its 11 domains (Source of data, Participants, etc). We determined the risk of bias and applicability (participants, outcome, predictors and analysis) through PROBAST (Prediction model Risk Of Bias ASsessment Tool).

RESULTS

A total of 14 studies were included. In the CHARMS extraction, the models found showed great variability in its 11 domains. Regarding the PROBAST checklist, only one article had an unclear risk of bias as it did not indicate how missing data were handled while the others all had a high risk of bias. This was mainly due to the statistical analysis (inadequate sample size, handling of continuous predictors, missing data and selection of predictors), since 13 studies had a high risk of bias. Applicability was satisfactory in six articles. Most of the models integrate predictors from routine clinical practice. Discrimination and calibration were assessed for almost all the models, with the area under the ROC curve ranging from 0.59 to 0.955 and no lack of calibration. Only three models were externally validated and their maximum discrimination values in the derivation were from 0.712 and 0.84. One of them (Osborn) had undergone multiple validation studies.

DISCUSSION

Despite most of the studies showing a high risk of bias, we very cautiously recommend applying the Osborn model, as this has been externally validated various times.

Association Between Red Cell Distribution Width and Hospital Mortality in Patients with Sepsis

Objective

Sepsis is the leading cause of death in patients admitted to adult intensive care units (ICUs). We aimed to determine the predictive value of red blood cell distribution width (RDW) in patients with sepsis in a large cohort.

Methods

This retrospective observational study used data from the eICU Collaborative Research Database. The prognostic value of RDW was investigated using the receiver operating characteristic (ROC) curve, multiple logistic regression model, integrated discriminatory index (IDI), and net reclassification index (NRI).

Results

In total, 9743 patients were included. The area under the ROC curve of the RDW for predicting hospital mortality was 0.631 (95% confidence interval [CI]: 0.616–0.645). Based on the multiple logistic regression model, an RDW of ≥14.5% was correlated with hospital mortality, regardless of Sequential Organ Failure Assessment (SOFA) and Acute Physiology and Chronic Health Evaluation IV (APACHE IV) scores (odds ratio [OR]: 1.838, 95% CI: 1.598–2.119). Using SOFA and APACHE IV scores as reference, the IDI and continuous NRI of RDW for hospital mortality was about 0.3 and 0.014, respectively.

Conclusions

The RDW may be useful in predicting hospital mortality in patients with sepsis, offering extra prognostic value beyond SOFA and APACHE IV scores.

Determining the Electronic Signature of Infection in Electronic Health Record Data

OBJECTIVES

Recent sepsis studies have defined patients as "infected" using a combination of culture and antibiotic orders rather than billing data. However, the accuracy of these definitions is unclear. We aimed to compare the accuracy of different established criteria for identifying infected patients using detailed chart review.

DESIGN

Retrospective observational study.

SETTING

Six hospitals from three health systems in Illinois.

PATIENTS

Adult admissions with blood culture or antibiotic orders, or Angus International Classification of Diseases infection codes and death were eligible for study inclusion as potentially infected patients. Nine-hundred to 1,000 of these admissions were randomly selected from each health system for chart review, and a proportional number of patients who did not meet chart review eligibility criteria were also included and deemed not infected.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The accuracy of published billing code criteria by Angus et al and electronic health record criteria by Rhee et al and Seymour et al (Sepsis-3) was determined using the manual chart review results as the gold standard. A total of 5,215 patients were included, with 2,874 encounters analyzed via chart review and a proportional 2,341 added who did not meet chart review eligibility criteria. In the study cohort, 27.5% of admissions had at least one infection. This was most similar to the percentage of admissions with blood culture orders (26.8%), Angus infection criteria (28.7%), and the Sepsis-3 criteria (30.4%). Sepsis-3 criteria was the most sensitive (81%), followed by Angus (77%) and Rhee (52%), while Rhee (97%) and Angus (90%) were more specific than the Sepsis-3 criteria (89%). Results were similar for patients with organ dysfunction during their admission.

CONCLUSIONS

Published criteria have a wide range of accuracy for identifying infected patients, with the Sepsis-3 criteria being the most sensitive and Rhee criteria being the most specific. These findings have important implications for studies investigating the burden of sepsis on a local and national level.

External validation of the sepsis severity score

INTRODUCTION

Sepsis is defined as a life-threatening organ dysfunction caused by a dysregulated host response to infection. Mortality rates are high, exceeding 50% in patients with septic shock. The sepsis severity score (SSS) was developed to determine the severity of sepsis and as a prognostic model. The aim of this study was to externally validate the SSS model.

METHODS

Calibration and discrimination of the SSS were retrospectively evaluated using data from a single-center sepsis registry.

RESULTS

Data from 156 septic patients were recorded; 56% of them had septic shock, 94% of patients required mechanical ventilation. The observed hospital mortality was 60.3%. The mean SSS value was 94.4 (95% CI 90.5-98.3). The SSS presented excellent discrimination with an area under the receiver operating characteristic curve (AUC) of 0.806 (95% CI 0.734-0.866). The pairwise comparison of APACHE II (AUC = 0.789; 95% CI 0.715-0.851) with SSS and 1st day SOFA (AUC = 0.75; 95% CI 0.673-0.817) with SSS revealed no significant differences in discrimination between the models. The calibration of the SSS was good with the Hosmer-Lemeshow goodness-of-fit H test 9.59, P > 0.05. Analyses of calibration curve show absence of accurate predictions in lower deciles of lower risk (2nd and 4th).

CONCLUSION

The SSS demonstrated excellent discrimination. The calibration evaluation gave conflicting results; the H-L test result indicated a good calibration, while the visual analysis of the calibration curve suggested the opposite. The SSS requires further evaluation before it can be safely recommended as an outcome prediction model.