External validation of the sepsis severity score

Head-to-head comparison of 19 prediction models for short-term outcome in medical patients in the emergency department: a retrospective study

INTRODUCTION

Prediction models for identifying emergency department (ED) patients at high risk of poor outcome are often not externally validated. We aimed to perform a head-to-head comparison of the discriminatory performance of several prediction models in a large cohort of ED patients.

METHODS

In this retrospective study, we selected prediction models that aim to predict poor outcome and we included adult medical ED patients. Primary outcome was 31-day mortality, secondary outcomes were 1-day mortality, 7-day mortality, and a composite endpoint of 31-day mortality and admission to intensive care unit (ICU).The discriminatory performance of the prediction models was assessed using an area under the receiver operating characteristic curve (AUC). Finally, the prediction models with the highest performance to predict 31-day mortality were selected to further examine calibration and appropriate clinical cut-off points.

RESULTS

We included 19 prediction models and applied these to 2185 ED patients. Thirty-one-day mortality was 10.6% (231 patients), 1-day mortality was 1.4%, 7-day mortality was 4.4%, and 331 patients (15.1%) met the composite endpoint. The RISE UP and COPE score showed similar and very good discriminatory performance for 31-day mortality (AUC 0.86), 1-day mortality (AUC 0.87), 7-day mortality (AUC 0.86) and for the composite endpoint (AUC 0.81). Both scores were well calibrated. Almost no patients with RISE UP and COPE scores below 5% had an adverse outcome, while those with scores above 20% were at high risk of adverse outcome. Some of the other prediction models (i.e. APACHE II, NEWS, WPSS, MEWS, EWS and SOFA) showed significantly higher discriminatory performance for 1-day and 7-day mortality than for 31-day mortality.

CONCLUSIONS

Head-to-head validation of 19 prediction models in medical ED patients showed that the RISE UP and COPE score outperformed other models regarding 31-day mortality.

Explainable Artificial Intelligence Helps in Understanding the Effect of Fibronectin on Survival of Sepsis

Fibronectin (FN) plays an essential role in the host’s response to infection. In previous studies, a significant decrease in the FN level was observed in sepsis; however, it has not been clearly elucidated how this parameter affects the patient’s survival. To better understand the relationship between FN and survival, we utilized innovative approaches from the field of explainable machine learning, including local explanations (Break Down, Shapley Additive Values, Ceteris Paribus), to understand the contribution of FN to predicting individual patient survival. The methodology provides new opportunities to personalize informative predictions for patients. The results showed that the most important indicators for predicting survival in sepsis were INR, FN, age, and the APACHE II score. ROC curve analysis showed that the model’s successful classification rate was 0.92, its sensitivity was 0.92, its positive predictive value was 0.76, and its accuracy was 0.79. To illustrate these possibilities, we have developed and shared a web-based risk calculator for exploring individual patient risk. The web application can be continuously updated with new data in order to further improve the model.

Positive Role of Delta Neutrophil Index (DNI) as a Prodiagnostic Marker in Cecal Ligation and Puncture (CLP)-Induced Sepsis Murine Model

Sepsis is an emergent infectious disease and a leading cause of death despite immediate intervention. While Delta neutrophil index (DNI) and myeloperoxidase (MPO) are known as a prodiagnostic marker of sepsis, the preclinical evidence of the best marker of sepsis is unclear. For this, using a well-designed cecal ligation and puncture (CLP)-induced sepsis mouse model, we comparatively measured the level and cost-effectiveness of sepsis biomarkers such as DNI, myeloperoxidase (MPO), procalcitonin (PCT), and tumor necrosis factor-alpha (TNF-α). First, we found that the optimal time point for early detection is at 6 h, 24 h post-CLP. Strikingly, the peak level and fold change of DNI was revealed at 24 h, further showing the best fold change as compared with other biomarker levels. Given the fold change at 6, 24 h, PCT was next to DNI. Third, a cost-effectiveness survey showed that DNI was the best, with PCT next. Further, DNI level was moderate positively associated with PCT (ρ = 0.697, p = 0.012) and TNF-α (ρ = 0.599, p = 0.040). Collectively, these data indicate that DNI in CLP-induced sepsis mice is as effective as the existent inflammatory biomarkers such as MPO, PCT and TNF-α to predict the prognosis of sepsis. This might have clinically important implications that DNI is cost effective, thus quickly and rationally applying to diverse types of imminent sepsis regardless of species. This might be the first report on the validity of DNI in preclinical CLP-induced murine sepsis.

A critical appraisal of the prognostic predictive models for patients with sepsis: Which model can be applied in clinical practice?

BACKGROUND

Sepsis is associated with high mortality and predictive models can help in clinical decision-making. The objective of this study was to carry out a systematic review of these models.

METHODS

In 2019, we conducted a systematic review in MEDLINE and EMBASE (CDR42018111121:PROSPERO) of articles that developed predictive models for mortality in septic patients (inclusion criteria). We followed the CHARMS recommendations (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies), extracting the information from its 11 domains (Source of data, Participants, etc). We determined the risk of bias and applicability (participants, outcome, predictors and analysis) through PROBAST (Prediction model Risk Of Bias ASsessment Tool).

RESULTS

A total of 14 studies were included. In the CHARMS extraction, the models found showed great variability in its 11 domains. Regarding the PROBAST checklist, only one article had an unclear risk of bias as it did not indicate how missing data were handled while the others all had a high risk of bias. This was mainly due to the statistical analysis (inadequate sample size, handling of continuous predictors, missing data and selection of predictors), since 13 studies had a high risk of bias. Applicability was satisfactory in six articles. Most of the models integrate predictors from routine clinical practice. Discrimination and calibration were assessed for almost all the models, with the area under the ROC curve ranging from 0.59 to 0.955 and no lack of calibration. Only three models were externally validated and their maximum discrimination values in the derivation were from 0.712 and 0.84. One of them (Osborn) had undergone multiple validation studies.

DISCUSSION

Despite most of the studies showing a high risk of bias, we very cautiously recommend applying the Osborn model, as this has been externally validated various times.