Validation of the Sepsis Severity Score Compared with Updated Severity Scores in Predicting Hospital Mortality in Sepsis Patients

Development and validation of an interpretable model for predicting sepsis mortality across care settings

There are numerous prognostic predictive models for evaluating mortality risk, but current scoring models might not fully cater to sepsis patients' needs. This study developed and validated a new model for sepsis patients that is suitable for any care setting and accurately forecasts 28-day mortality. The derivation dataset, gathered from 20 hospitals between September 2019 and December 2021, contrasted with the validation dataset, collected from 15 hospitals from January 2022 to December 2022. In this study, 7436 patients were classified as members of the derivation dataset, and 2284 patients were classified as members of the validation dataset. The point system model emerged as the optimal model among the tested predictive models for foreseeing sepsis mortality. For community-acquired sepsis, the model's performance was satisfactory (derivation dataset AUC: 0.779, 95% CI 0.765-0.792; validation dataset AUC: 0.787, 95% CI 0.765-0.810). Similarly, for hospital-acquired sepsis, it performed well (derivation dataset AUC: 0.768, 95% CI 0.748-0.788; validation dataset AUC: 0.729, 95% CI 0.687-0.770). The calculator, accessible at https://avonlea76.shinyapps.io/shiny_app_up/ , is user-friendly and compatible. The new predictive model of sepsis mortality is user-friendly and satisfactorily forecasts 28-day mortality. Its versatility lies in its applicability to all patients, encompassing both community-acquired and hospital-acquired sepsis.

Is it Time to Develop an Indian Sepsis-related Mortality Prediction Score?

How to cite this article: Dedeepiya VD. Is it Time to Develop an Indian Sepsis-related Mortality Prediction Score? Indian J Crit Care Med 2024;28(4):320-322.

Machine learning for the prediction of sepsis-related death: a systematic review and meta-analysis

BACKGROUND AND OBJECTIVES

Sepsis is accompanied by a considerably high risk of mortality in the short term, despite the availability of recommended mortality risk assessment tools. However, these risk assessment tools seem to have limited predictive value. With the gradual integration of machine learning into clinical practice, some researchers have attempted to employ machine learning for early mortality risk prediction in sepsis patients. Nevertheless, there is a lack of comprehensive understanding regarding the construction of predictive variables using machine learning and the value of various machine learning methods. Thus, we carried out this systematic review and meta-analysis to explore the predictive value of machine learning for sepsis-related death at different time points.

METHODS

PubMed, Embase, Cochrane, and Web of Science databases were searched until August 9th, 2022. The risk of bias in predictive models was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST). We also performed subgroup analysis according to time of death and type of model and summarized current predictive variables used to construct models for sepsis death prediction.

RESULTS

Fifty original studies were included, covering 104 models. The combined Concordance index (C-index), sensitivity, and specificity of machine learning models were 0.799, 0.81, and 0.80 in the training set, and 0.774, 0.71, and 0.68 in the validation set, respectively. Machine learning outperformed conventional clinical scoring tools and showed excellent C-index, sensitivity, and specificity in different subgroups. Random Forest (RF) and eXtreme Gradient Boosting (XGBoost) are the preferred machine learning models because they showed more favorable accuracy with similar modeling variables. This study found that lactate was the most frequent predictor but was seriously ignored by current clinical scoring tools.

CONCLUSION

Machine learning methods demonstrate relatively favorable accuracy in predicting the mortality risk in sepsis patients. Given the limitations in accuracy and applicability of existing prediction scoring systems, there is an opportunity to explore updates based on existing machine learning approaches. Specifically, it is essential to develop or update more suitable mortality risk assessment tools based on the specific contexts of use, such as emergency departments, general wards, and intensive care units.

Validating the APACHE IV score in predicting length of stay in the intensive care unit among patients with sepsis

The Acute Physiology and Chronic Health Evaluation (APACHE) IV model can predict the intensive care unit (ICU) length of stay (LOS) in critically ill patients. Thus, this study aimed to validate the performance of the APACHE IV score in predicting ICU LOS among patients with sepsis. This retrospective study was conducted in the medical ICU of a tertiary university between 2017 and 2020. A total of 1,039 sepsis patients were enrolled. Patients with an ICU stay of 1 and > 3 days accounted for 20.1% and 43.9%. The overall observed and APACHE IV predicted ICU LOS were 6.3 ± 6.5 and 6.8 ± 6.5, respectively. The APACHE IV slightly over-predicted ICU LOS with standardized length of stay ratio 0.95 (95% CI 0.89-1.02). The predicted ICU LOS based on the APACHE IV score was statistically longer than the observed ICU LOS (p < 0.001) and were poorly correlated (R² = 0.02, p < 0.001), especially in patients with a lower severity of illness. In conclusions the APACHE IV model poorly predicted ICU LOS in patients with sepsis. The APACHE IV score needs to be modified or we need to make a new specific model to predict ICU stays in patients with sepsis.

Better prognostic ability of NEWS2, SOFA and SAPS-II in septic patients

BACKGROUND AND OBJECTIVES

To compare the ability of qSOFA, NEWS2, SOFA, LODS, SIRS, APACHE-II and SAPS-II scores.

MATERIAL AND METHODS

Analysis of in-hospital mortality of 203 patients admitted to the ICU because of sepsis. The scores were compared according to their application. Discrimination was evaluated with AUC-ROC curve and performance with the Akaike's (AIC) and Bayesian information criterion (BIC).

RESULTS

In-hospital mortality was 31.53%. NEWS2 showed better mortality discrimination ability and better performance considering the AIC/BIC criterion for mortality tan qSOFA (AUC-ROC=.615 and .536; P=.039). SOFA presented higher performance and AUC-ROC tan LODS (.776 vs .693; P=.01) and both showed higher discrimination ability than SIRS (AUC-ROC=.521; P<.003). Finally, SAPS-II was able to predict mortality with better performance than APACHE-II and presented higher discrimination capacity but without statistical significance compared (AUROC=.738 for SAPS-II and AUROC=.673 for APACHE-II; P=.08).

CONCLUSION

NEWS2 is a better predictor of mortality than qSOFA and its implementation for the early recognition of the septic patient or the patient with higher risk in the emergency and hospitalization wards should be addressed. In addition, given that SOFA and SAPS-II showed better performance and are simpler than LODS and APACHE-II, respectively, both should be considered the scores of choice in this setting.

Association between the Predicted Value of APACHE IV Scores and Intensive Care Unit Mortality: A Secondary Analysis Based on EICU Dataset

Objective

The evidence regarding the relationship between Acute Physiological and Chronic Health Assessment (APACHE) IV scores and emergency intensive care unit (EICU) mortality in patients following organ transplantation remains controversial. The purpose of this study was to investigate the relationship between APACHE IV score and EICU mortality.

Methods

Data from 391 American men and women admitted to the EICU after undergoing organ transplants including heart, bone marrow, liver, kidney, lung, and pancreas in the United States. We used this data to analyze the relationship between APACHE IV scores and in-hospital mortality in the postoperative EICU. The primary endpoint was ICU hospitalization mortality after organ transplantation. The entire study data was extracted from the EICU database and uploaded to the DataDryad website.

Results

Interaction tests indicate age, respiratory failure, and hormone use can modify the association between APACHE IV and EICU mortality. A stronger association of APACHE and mortality can be observed at <60 years old, no respiratory failure, and no use of hormones. In contrast, there was no association between respiratory failure, hormone use, APACHE, and ICU mortality in patients over 60 years of age.

Conclusion

When using the APACHE score for risk stratification of critically ill patients after transplantation, the patient's age, respiratory failure, and use of hormones should be taken into account.

Establishment and validation of the predictive model for the in-hospital death in patients with sepsis

BACKGROUND

Identifying sepsis patients with risk of in-hospital death early can improve the prognosis of patients. This study aimed to develop a model to predict in-hospital death of sepsis patients based on the Medical Information Mart for Intensive Care-Ⅲ (MIMIC-Ⅲ) database, and use clinical data to externally validate the model.

METHODS

A total of 1,839 sepsis patients were used for model development, and 125 clinical cases were used for external validation. The discriminatory ability of the model was determined by calculating the area under the curve (AUC) with 95% confidence intervals (CI).

RESULTS

The AUC of the random forest model and logistic regression model was 0.754 (95%CI, 0.732-0.776) and 0.703 (95%CI, 0.680-0.727), respectively, and the random forest model had higher AUC (Z = 3.070, P = .002). External validation showed that the AUC of the random forest model was 0.539 (95%CI, 0.440-0.628). Further validation was carried out according to gender and SOFA score. The AUC of the model in males and females was 0.648 and 0.412, respectively. In addition, the AUC of the model in the population with SOFA scores of 3-8, 9-12, and 13-15 were 0.705, 0.495, and 0.769, respectively.

CONCLUSIONS

The random forest model had a better predictive ability and a good applicability to external populations with SOFA score of 13-15.

Performance of the Simple Clinical Score (SCS) and the Rapid Emergency Medicine Score (REMS) to predict severity level and mortality rate among patients with sepsis in the emergency department

Nurses play a key role as the first line of service for patients with medical conditions and injuries in the emergency department (ED), which includes assessing patients for sepsis. The researchers evaluated tools to examine the performance of the Simple Clinical Score (SCS) and the Rapid Emergency Medicine Score (REMS) to predict sepsis severity and mortality among sepsis patients in the ED. A retrospective survey was performed, selecting participants by using a purposive sampling method, and including the medical records of all patients diagnosed with sepsis admitted to the ED at Singburi Hospital, Thailand. Data were analysed using the ROC curve and the Area Under Curve (AUC) to calculate the accuracy of each patient's mortality prediction. A total of 225 patients diagnosed with sepsis was identified, with a mortality rate of 59.11% after admission to the medical service and intensive care unit. The AUC analysis showed that the accuracy of the model generated from the REMS (88.6%) was higher than that of the SCS (76.7%). The authors also recommend that key variables identified in this research should be used to develop screening and assessment tools for sepsis in the context of the ED.

Algorithmic prognostication in critical care: a promising but unproven technology for supporting difficult decisions

PURPOSE OF REVIEW

Patients, surrogate decision makers, and clinicians face weighty and urgent decisions under uncertainty in the ICU, which could be aided by risk prediction. Although emerging artificial intelligence/machine learning (AI/ML) algorithms could reduce uncertainty surrounding these life and death decisions, certain criteria must be met to ensure their bedside value.

RECENT FINDINGS

Although ICU severity of illness scores have existed for decades, these tools have not been shown to predict well or to improve outcomes for individual patients. Novel AI/ML tools offer the promise of personalized ICU care but remain untested in clinical trials. Ensuring that these predictive models account for heterogeneity in patient characteristics and treatments, are not only specific to a clinical action but also consider the longitudinal course of critical illness, and address patient-centered outcomes related to equity, transparency, and shared decision-making will increase the likelihood that these tools improve outcomes. Improved clarity around standards and contributions from institutions and critical care departments will be essential.

SUMMARY

Improved ICU prognostication, enabled by advanced ML/AI methods, offer a promising approach to inform difficult and urgent decisions under uncertainty. However, critical knowledge gaps around performance, equity, safety, and effectiveness must be filled and prospective, randomized testing of predictive interventions are still needed.

A critical appraisal of the prognostic predictive models for patients with sepsis: Which model can be applied in clinical practice?

BACKGROUND

Sepsis is associated with high mortality and predictive models can help in clinical decision-making. The objective of this study was to carry out a systematic review of these models.

METHODS

In 2019, we conducted a systematic review in MEDLINE and EMBASE (CDR42018111121:PROSPERO) of articles that developed predictive models for mortality in septic patients (inclusion criteria). We followed the CHARMS recommendations (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies), extracting the information from its 11 domains (Source of data, Participants, etc). We determined the risk of bias and applicability (participants, outcome, predictors and analysis) through PROBAST (Prediction model Risk Of Bias ASsessment Tool).

RESULTS

A total of 14 studies were included. In the CHARMS extraction, the models found showed great variability in its 11 domains. Regarding the PROBAST checklist, only one article had an unclear risk of bias as it did not indicate how missing data were handled while the others all had a high risk of bias. This was mainly due to the statistical analysis (inadequate sample size, handling of continuous predictors, missing data and selection of predictors), since 13 studies had a high risk of bias. Applicability was satisfactory in six articles. Most of the models integrate predictors from routine clinical practice. Discrimination and calibration were assessed for almost all the models, with the area under the ROC curve ranging from 0.59 to 0.955 and no lack of calibration. Only three models were externally validated and their maximum discrimination values in the derivation were from 0.712 and 0.84. One of them (Osborn) had undergone multiple validation studies.

DISCUSSION

Despite most of the studies showing a high risk of bias, we very cautiously recommend applying the Osborn model, as this has been externally validated various times.

Severity indexes of blunt trauma victims in intensive therapy: prediction capacity for mortality

OBJECTIVE

To identify the predictive capacity for mortality of the indexes Revised Trauma Score, Rapid Emergency Medicine Score, modified Rapid Emergency Medicine Score, and Simplified Acute Physiology Score III in blunt trauma victims hospitalized in an intensive care unit and compare their performance.

METHOD

Retrospective cohort of patients with blunt trauma in an intensive care unit from medical records. Receiver Operating Characteristic and a 95% confidence interval of the area under the curve were analyzed to compare results.

RESULTS

Out of 165 analyzed patients, 66.7% have received surgical treatment. The mortality in the intensive care unit and in the hospital was 17.6% and 20.6%, respectively. For the mortality in the intensive care unit, the area under the curve varied from 0.672 to 0.738; however, better results have been observed in surgical patients (0.747 to 0.811). Similar results have been observed for in-hospital mortality. In all analyses, the areas under the curve of the indexes presented no significant difference.

CONCLUSION

The accuracy of the severity indexes was moderate, with an improved performance when applied to surgical patients. The four indexes presented a similar prediction for the analyzed outcomes.

The 4-Hour Cairns Sepsis Model: A novel approach to predicting sepsis mortality at intensive care unit admission

BACKGROUND

Sepsis commonly causes intensive care unit (ICU) mortality, yet early identification of adults with sepsis at risk of dying in the ICU remains a challenge.

OBJECTIVE

The aim of the study was to derive a mortality prediction model (MPM) to assist ICU clinicians and researchers as a clinical decision support tool for adults with sepsis within 4 h of ICU admission.

METHODS

A cohort study was performed using 500 consecutive admissions between 2014 and 2018 to an Australian tertiary ICU, who were aged ≥18 years and had sepsis. A total of 106 independent variables were assessed against ICU episode-of-care mortality. Multivariable backward stepwise logistic regression derived an MPM, which was assessed on discrimination, calibration, fit, sensitivity, specificity, and predictive values and bootstrapped.

RESULTS

The average cohort age was 58 years, the Acute Physiology and Chronic Health Evaluation III-j severity score was 72, and the case fatality rate was 12%. The 4-Hour Cairns Sepsis Model (CSM-4) consists of age, history of renal disease, number of vasopressors, Glasgow Coma Scale, lactate, bicarbonate, aspartate aminotransferase, lactate dehydrogenase, albumin, and magnesium with an area under the receiver operating characteristic curve of 0.90 (95% confidence interval = 0.84-0.95, p < 0.00001), a Nagelkerke R² of 0.51, specificity of 0.94, a negative predictive value of 0.98, and almost identical odds ratios during bootstrapping. The CSM-4 outperformed existing MPMs tested on our data set. The CSM-4 also performed similar to existing MPMs in their derivation papers whilst using fewer, routinely collected, and inexpensive variables.

CONCLUSIONS

The CSM-4 is a newly derived MPM for adults with sepsis at ICU admission. It displays excellent discrimination, calibration, fit, specificity, negative predictive value, and bootstrapping values whilst being easy to use and inexpensive. External validation is required.

Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU

Background

Early and accurate identification of sepsis patients with high risk of in-hospital death can help physicians in intensive care units (ICUs) make optimal clinical decisions. This study aimed to develop machine learning-based tools to predict the risk of hospital death of patients with sepsis in ICUs.

Methods

The source database used for model development and validation is the medical information mart for intensive care (MIMIC) III. We identified adult sepsis patients using the new sepsis definition Sepsis-3. A total of 86 predictor variables consisting of demographics, laboratory tests and comorbidities were used. We employed the least absolute shrinkage and selection operator (LASSO), random forest (RF), gradient boosting machine (GBM) and the traditional logistic regression (LR) method to develop prediction models. In addition, the prediction performance of the four developed models was evaluated and compared with that of an existent scoring tool – simplified acute physiology score (SAPS) II – using five different performance measures: the area under the receiver operating characteristic curve (AUROC), Brier score, sensitivity, specificity and calibration plot.

Results

The records of 16,688 sepsis patients in MIMIC III were used for model training and test. Amongst them, 2949 (17.7%) patients had in-hospital death. The average AUROCs of the LASSO, RF, GBM, LR and SAPS II models were 0.829, 0.829, 0.845, 0.833 and 0.77, respectively. The Brier scores of the LASSO, RF, GBM, LR and SAPS II models were 0.108, 0.109, 0.104, 0.107 and 0.146, respectively. The calibration plots showed that the GBM, LASSO and LR models had good calibration; the RF model underestimated high-risk patients; and SAPS II had the poorest calibration.

Conclusion

The machine learning-based models developed in this study had good prediction performance. Amongst them, the GBM model showed the best performance in predicting the risk of in-hospital death. It has the potential to assist physicians in the ICU to perform appropriate clinical interventions for critically ill sepsis patients and thus may help improve the prognoses of sepsis patients in the ICU.

External validation of the sepsis severity score

INTRODUCTION

Sepsis is defined as a life-threatening organ dysfunction caused by a dysregulated host response to infection. Mortality rates are high, exceeding 50% in patients with septic shock. The sepsis severity score (SSS) was developed to determine the severity of sepsis and as a prognostic model. The aim of this study was to externally validate the SSS model.

METHODS

Calibration and discrimination of the SSS were retrospectively evaluated using data from a single-center sepsis registry.

RESULTS

Data from 156 septic patients were recorded; 56% of them had septic shock, 94% of patients required mechanical ventilation. The observed hospital mortality was 60.3%. The mean SSS value was 94.4 (95% CI 90.5-98.3). The SSS presented excellent discrimination with an area under the receiver operating characteristic curve (AUC) of 0.806 (95% CI 0.734-0.866). The pairwise comparison of APACHE II (AUC = 0.789; 95% CI 0.715-0.851) with SSS and 1st day SOFA (AUC = 0.75; 95% CI 0.673-0.817) with SSS revealed no significant differences in discrimination between the models. The calibration of the SSS was good with the Hosmer-Lemeshow goodness-of-fit H test 9.59, P > 0.05. Analyses of calibration curve show absence of accurate predictions in lower deciles of lower risk (2nd and 4th).

CONCLUSION

The SSS demonstrated excellent discrimination. The calibration evaluation gave conflicting results; the H-L test result indicated a good calibration, while the visual analysis of the calibration curve suggested the opposite. The SSS requires further evaluation before it can be safely recommended as an outcome prediction model.

Sepsis as the primary admitting diagnosis of transferred patients who died within 48 hours of arrival at a Central Texas hospital

Interhospital transfers are independently associated with inpatient mortality, and transferred patients have worse outcomes. The aim of this study was to retrospectively assess the 48-hour mortality rate in interhospital transfer cohorts of all transfers to a Central Texas teaching hospital and to identify a primary admitting diagnosis for potential intervention. A total of 15,435 patients with 19,161 transfers over the course of the study were retrospectively reviewed and placed in 18 different categories based upon the primary admitting diagnosis. There were about 5000 transfer patients yearly with ∼1.4% deaths within 48 hours of arrival. The three leading categories for transferred patients were cardiovascular, neurologic, and psychiatric. In this group, 268 of 19,161 transfers died within 48 hours of arrival. Despite being the 10th leading category for transfer, sepsis was the leading primary admitting diagnosis of patients who died within 48 hours of arrival, accounting for nearly 22% of those patients. Given the significant association found between sepsis and 48-hour mortality after transfer, we devised a novel interhospital transfer checklist based upon the Surviving Sepsis guidelines in an attempt to decrease mortality associated with these transfers.

Validation the performance of New York Sepsis Severity Score compared with Sepsis Severity Score in predicting hospital mortality among sepsis patients

PURPOSE

The aim of this study was to compare the performance of the New York Sepsis Severity Score (NYSSS) with the Sepsis Severity Score (SSS) and Acute Physiology and Chronic Health Evaluation and Simplified Acute Physiology Scores for predicting mortality in sepsis patients.

METHOD

A retrospective analysis was conducted in the intensive care unit. The primary outcome was in-hospital mortality.

RESULTS

Overall 1680 sepsis patients were enrolled. The hospital mortality rate was 44.4%. The NYSSS underestimated actual mortality with standard mortality ratio (SMR) of 1.28 (95%CI 1.19-1.38). However, the SSS slightly overestimated the actual mortality with an SMR of 0.94 (0.88-1.01). The NYSSS had moderate discrimination with an AUC of 0.772 (0.750-0.794), in contrast to the SSS which had good discrimination with an AUC of 0.889 (0.873-0.904). The AUC of the SSS was statistically higher than that of the NYSSS. The AUCs of both the NYSSS and SSS were significantly lower than other standard severity scores. The calibrations for all severity scores were poor. The SSS had better overall performance than the NYSSS (Brier score 0.149 and 0.201, respectively).

CONCLUSION

The SSS had better discrimination and overall performance than the NYSSS. However, both sepsis severity scores were poorly calibrated.

Comparison of the performance of SOFA, qSOFA and SIRS for predicting mortality and organ failure among sepsis patients admitted to the intensive care unit in a middle-income country

INTRODUCTION

The Sepsis-3 definition provides a change of two or more scores from zero or a known baseline of the Sequential Organ Failure Assessment (SOFA) as criteria of sepsis. The aim of this study was to compare the SOFA score and the quick SOFA (qSOFA) to Systemic Inflammatory Response Syndrome (SIRS) criteria in predictive ability of mortality and organ failure.

METHODS

A-10year retrospective cohort study was conducted in a teaching hospital in Thailand.

RESULTS

A total of 2350 of mixed sepsis patients by Sepsis-2 definition were included. The all-cause hospital mortality rate was 44.5%. Of the total sample, 95.6% (n=2247) of patients met criteria for sepsis under the Sepsis-3 definition. The SOFA score presented the best discrimination with an area under the receiver operating characteristic curve (AUC) of 0.839. The AUC of SOFA score for hospital mortality was significantly higher than qSOFA (AUC 0.814, P=0.003) and SIRS (AUC 0.587, P<0.0001). Also, the SOFA score had superior performance than other scores for predicting intensive care unit (ICU) mortality and organ failure.

CONCLUSIONS

The SOFA is a superior prognostic tool for predicting mortality and organ failure than qSOFA and SIRS criteria among sepsis patients admitted to the ICU.