Internal validation and evaluation of the predictive performance of models based on the PRISM-3 (Pediatric Risk of Mortality) and PIM-3 (Pediatric Index of Mortality) scoring systems for predicting mortality in Pediatric Intensive Care Units (PICUs)

Development and validation of nomogram for predicting the risk of transferring to the ICU for children with influenza

OBJECTIVE

Development of a nomogram model for predicting the magnitude of risk of transferring hospitalized children with influenza to the ICU.

METHODS

In a single-center retrospective study, 318 children with influenza who were hospitalized in our hospital from January 2018 to August 2023 were collected as study subjects. Children with influenza were randomly assigned to the training set and validation set in a ratio of 4:1. In the training set, risk factors were identified using univariate and multivariate logistic regression analyses, and a nomogram model was created on this basis. The validation set was used to evaluate the predictive power of the model.

RESULTS

Multifactorial logistic regression analysis revealed six independent risk factors for transfer to the ICU in hospitalized children with influenza, including elevated peripheral white blood cell counts, elevated large platelet ratios, reduced mean platelet width, reduced complement C3, elevated serum globulin levels, and reduced total immunoglobulin M levels. Using these six metrics as predictors to construct a nomogram graphical model, the C-index was 0.970 (95% Cl: 0.953-0.988). The areas under the curve for the training and validation sets were 0.966 (95%Cl 0.947-0.985) and 0.919 (95%Cl 0.851-0.986), respectively.

CONCLUSION

A nomogram for predicting the risk of transferring to the ICU for children with influenza was developed and validated, which demonstrates good calibration and clinical benefits.

A comparative study of explainable ensemble learning and logistic regression for predicting in-hospital mortality in the emergency department

This study addresses the challenges associated with emergency department (ED) overcrowding and emphasizes the need for efficient risk stratification tools to identify high-risk patients for early intervention. While several scoring systems, often based on logistic regression (LR) models, have been proposed to indicate patient illness severity, this study aims to compare the predictive performance of ensemble learning (EL) models with LR for in-hospital mortality in the ED. A cross-sectional single-center study was conducted at the ED of Imam Reza Hospital in northeast Iran from March 2016 to March 2017. The study included adult patients with one to three levels of emergency severity index. EL models using Bagging, AdaBoost, random forests (RF), Stacking and extreme gradient boosting (XGB) algorithms, along with an LR model, were constructed. The training and validation visits from the ED were randomly divided into 80% and 20%, respectively. After training the proposed models using tenfold cross-validation, their predictive performance was evaluated. Model performance was compared using the Brier score (BS), The area under the receiver operating characteristics curve (AUROC), The area and precision-recall curve (AUCPR), Hosmer-Lemeshow (H-L) goodness-of-fit test, precision, sensitivity, accuracy, F1-score, and Matthews correlation coefficient (MCC). The study included 2025 unique patients admitted to the hospital's ED, with a total percentage of hospital deaths at approximately 19%. In the training group and the validation group, 274 of 1476 (18.6%) and 152 of 728 (20.8%) patients died during hospitalization, respectively. According to the evaluation of the presented framework, EL models, particularly Bagging, predicted in-hospital mortality with the highest AUROC (0.839, CI (0.802-0.875)) and AUCPR = 0.64 comparable in terms of discrimination power with LR (AUROC (0.826, CI (0.787-0.864)) and AUCPR = 0.61). XGB achieved the highest precision (0.83), sensitivity (0.831), accuracy (0.842), F1-score (0.833), and the highest MCC (0.48). Additionally, the most accurate models in the unbalanced dataset belonged to RF with the lowest BS (0.128). Although all studied models overestimate mortality risk and have insufficient calibration (P > 0.05), stacking demonstrated relatively good agreement between predicted and actual mortality. EL models are not superior to LR in predicting in-hospital mortality in the ED. Both EL and LR models can be considered as screening tools to identify patients at risk of mortality.

Predicting Clinical Deterioration and Mortality at Differing Stages During Hospitalization: A Systematic Review of Risk Prediction Models in Children in Low- and Middle-Income Countries

OBJECTIVE

To determine which risk prediction model best predicts clinical deterioration in children at different stages of hospital admission in low- and middle-income countries.

METHODS

For this systematic review, Embase and MEDLINE databases were searched, and Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines were followed. The key search terms were "development or validation study with risk-prediction model" AND "deterioration or mortality" AND "age 0-18 years" AND "hospital-setting: emergency department (ED), pediatric ward (PW), or pediatric intensive care unit (PICU)" AND "low- and middle-income countries." The Prediction Model Risk of Bias Assessment Tool was used by two independent authors. Forest plots were used to plot area under the curve according to hospital setting. Risk prediction models used in two or more studies were included in a meta-analysis.

RESULTS

We screened 9486 articles and selected 78 publications, including 67 unique predictive models comprising 1.5 million children. The best performing models individually were signs of inflammation in children that can kill (SICK) (ED), pediatric early warning signs resource limited settings (PEWS-RL) (PW), and Pediatric Index of Mortality (PIM) 3 as well as pediatric sequential organ failure assessment (pSOFA) (PICU). Best performing models after meta-analysis were SICK (ED), pSOFA and Pediatric Early Death Index for Africa (PEDIA)-immediate score (PW), and pediatric logistic organ dysfunction (PELOD) (PICU). There was a high risk of bias in all studies.

CONCLUSIONS

We identified risk prediction models that best estimate deterioration, although these risk prediction models are not routinely used in low- and middle-income countries. Future studies should focus on large scale external validation with strict methodological criteria of multiple risk prediction models as well as study the barriers in the way of implementation.

TRIAL REGISTRATION

PROSPERO International Prospective Register of Systematic Reviews: Prospero ID: CRD42021210489.

Phase angle as a marker of outcome in hospitalized pediatric patients. A systematic review of the evidence (GRADE) with meta-analysis

Phase angle (PhA) is a valuable tool for evaluating the nutritional and inflammatory status, which can accompany acute and severe disorders. PhA is a cellular health biomarker, whose value is particularly substantial due to the negative consequences of these situations in the pediatric population. Relevant literature was collected with the aim of comprehensively analysing the evidence on the association between an altered PhA can serve as a predictive-marker for mortality and poor-outcomes in at-risk-pediatric patients. Understanding this relationship could have significant implications for identifying high-risk individuals and implementing timely interventions. A systematic review with meta-analysis was conducted in the primary electronic databases from inception until January 2023. Overall, four studies with a total of 740 patients were eligible for our analysis. Evidence demonstrates that PhA is associated with nutritional status, reflecting undernutrition and changes in body composition related to illness. This review suggests that PhA can indeed be used as an indicator of nutritional status and a tool for predicting prognosis, including mortality and poor-outcomes, in hospitalized pediatric patients. A low PhA was associated with a significant mortality risk [RR:1.51;95%CI (1.22-1.88),p = 0.0002;I2 = 0%,(p = 0.99)] and an increased complications risk [OR:8.17;95%CI (2.44-27.4),p = 0.0007;I2 = 44%,(p = 0.18)]. These findings highlight the importance of taking a comprehensive approach to clinical nutrition, integrating multiple evaluation aspects to establish an accurate diagnosis and personalized therapeutic plans. While PhA emerges as a valuable tool for assessing the risk of malnutrition and as a prognostic-indicator for poor-outcomes in pediatric patients. Further future studies are needed to focus on investigating this relationship in larger and diverse population to strengthen the evidence base.

Comparison of Six Scoring Systems for Predicting In-hospital Mortality among Patients with SARS-COV2 Presenting to the Emergency Department

Background

The study aimed to compare the prognostic accuracy of six different severity-of-illness scoring systems for predicting in-hospital mortality among patients with confirmed SARS-COV2 who presented to the emergency department (ED). The scoring systems assessed were worthing physiological score (WPS), early warning score (EWS), rapid acute physiology score (RAPS), rapid emergency medicine score (REMS), national early warning score (NEWS), and quick sequential organ failure assessment (qSOFA).

Materials and methods

A cohort study was conducted using data obtained from electronic medical records of 6,429 confirmed SARS-COV2 patients presenting to the ED. Logistic regression models were fitted on the original severity-of-illness scores to assess the models' performance using the Area Under the Curve for ROC (AUC-ROC) and Precision-Recall curves (AUC-PR), Brier Score (BS), and calibration plots were used to assess the models' performance. Bootstrap samples with multiple imputations were used for internal validation.

Results

The mean age of the patients was 64 years (IQR:50-76) and 57.5% were male. The WPS, REMS, and NEWS models had AUROC of 0.714, 0.705, and 0.701, respectively. The poorest performance was observed in the RAPS model, with an AUROC of 0.601. The BS for the NEWS, qSOFA, EWS, WPS, RAPS, and REMS was 0.18, 0.09, 0.03, 0.14, 0.15, and 0.11 respectively. Excellent calibration was obtained for the NEWS, while the other models had proper calibration.

Conclusion

The WPS, REMS, and NEWS have a fair discriminatory performance and may assist in risk stratification for SARS-COV2 patients presenting to the ED. Generally, underlying diseases and most vital signs are positively associated with mortality and were different between the survivors and non-survivors.

How to cite this article

Rahmatinejad Z, Hoseini B, Reihani H, Hanna AA, Pourmand A, Tabatabaei SM, et al. Comparison of Six Scoring Systems for Predicting In-hospital Mortality among Patients with SARS-COV2 Presenting to the Emergency Department. Indian J Crit Care Med 2023;27(6):416-425.

Radiologic Assessment of Lung Edema Score as a Predictor of Clinical Outcome in Children with Acute Respiratory Distress Syndrome

PURPOSE

The radiographic assessment of lung edema (RALE) score enables objective quantification of lung edema and is a valuable prognostic marker of adult acute respiratory distress syndrome (ARDS). We aimed to evaluate the validity of RALE score in children with ARDS.

MATERIALS AND METHODS

The RALE score was measured for its reliability and correlation to other ARDS severity indices. ARDS-specific mortality was defined as death from severe pulmonary dysfunction or the need for extracorporeal membrane oxygenation therapy. The C-index of the RALE score and other ARDS severity indices were compared via survival analyses.

RESULTS

Among 296 children with ARDS, 88 did not survive, and there were 70 ARDS-specific non-survivors. The RALE score showed good reliability with an intraclass correlation coefficient of 0.809 [95% confidence interval (CI), 0.760-0.848]. In univariable analysis, the RALE score had a hazard ratio (HR) of 1.19 (95% CI, 1.18-3.11), and the significance was maintained in multivariable analysis adjusting with age, ARDS etiology, and comorbidity, with an HR of 1.77 (95% CI, 1.05-2.91). The RALE score was a good predictor of ARDS-specific mortality, with a C-index of 0.607 (95% CI, 0.519-0.695).

CONCLUSION

The RALE score is a reliable measure for ARDS severity and a useful prognostic marker of mortality in children, especially for ARDS-specific mortality. This score provides information that clinicians can use to decide the proper time of aggressive therapy targeting severe lung injury and to appropriately manage the fluid balance of children with ARDS.

Comparing In-Hospital Mortality Prediction by Senior Emergency Resident's Judgment and Prognostic Models in the Emergency Department

Background

A comparison of emergency residents' judgments and two derivatives of the Sequential Organ Failure Assessment (SOFA), namely, the mSOFA and the qSOFA, was conducted to determine the accuracy of predicting in-hospital mortality among critically ill patients in the emergency department (ED).

Methods

A prospective cohort research was performed on patients over 18 years of age presented to the ED. We used logistic regression to develop a model for predicting in-hospital mortality by using qSOFA, mSOFA, and residents' judgment scores. We compared the accuracy of prognostic models and residents' judgment in terms of the overall accuracy of the predicted probabilities (Brier score), discrimination (area under the ROC curve), and calibration (calibration graph). Analyses were carried out using R software version R-4.2.0.

Results

In the study, 2,205 patients with median age of 64 (IQR: 50-77) years were included. There were no significant differences between the qSOFA (AUC 0.70; 95% CI: 0.67-0.73) and physician's judgment (AUC 0.68; 0.65-0.71). Despite this, the discrimination of mSOFA (AUC 0.74; 0.71-0.77) was significantly higher than that of the qSOFA and residents' judgments. Additionally, the AUC-PR of mSOFA, qSOFA, and emergency resident's judgments was 0.45 (0.43-0.47), 0.38 (0.36-0.40), and 0.35 (0.33-0.37), respectively. The mSOFA appears stronger in terms of overall performance: 0.13 vs. 0.14 and 0.15. All three models showed good calibration.

Conclusion

The performance of emergency residents' judgment and the qSOFA was the same in predicting in-hospital mortality. However, the mSOFA predicted better-calibrated mortality risk. Large-scale studies should be conducted to determine the utility of these models.

Internal Validation of the Predictive Performance of Models Based on Three ED and ICU Scoring Systems to Predict Inhospital Mortality for Intensive Care Patients Referred from the Emergency Department

Background.A variety of scoring systems have been introduced for use in both the emergency department (ED) such as WPS, REMS, and MEWS and the intensive care unit (ICU) such as APACHE II, SAPS II, and SOFA for risk stratification and mortality prediction. However, the performance of these models in the ICU remains unclear and we aimed to evaluate and compare their performance in the ICU. Methods. This multicenter retrospective cohort study was conducted on severely ill patients admitted to the ICU directly from the ED in seven tertiary hospitals in Iran from August 2018 to August 2020. We evaluated all models in terms of discrimination (AUROC), the balance between positive predictive value and sensitivity (AUPRC), calibration (Hosmer-Lemeshow test and calibration plots), and overall performance using the Brier score (BS). The endpoint was considered inhospital mortality. Results. Among the 3,455 patients included in the study, 54.4% of individuals were male ( $N=\mathrm{1,879}$ ) and 26.5% deceased ( $N=916$ ). The BS for the WPS, REMS, MEWS, APACHE II, SAPS II, and SOFA were 0.178, 0.165, 0.183, 0.157, 0.170, and 0.182, respectively. The AUROC of these models were 0.728 (0.71-0.75), 0.761 (0.74-0.78), 0.682 (0.66-0.70), 0.810 (0.79-0.83), 0.767 (0.75-0.79), and 0.785 (0.77-0.80), respectively. The AUPRC was 0.517 (0.50-0.53) for WPS, 0.547 (0.53-0.56) for REMS, 0.445 (0.42-0.46) for MEWS, 0.630 (0.61-0.65) for APACHE II, 0.559 (0.54-0.58) for SAPS II, and 0.564 (0.54-0.57) for SOFA. All models except the MEWS and SOFA had good calibration. The most accurate model belonged to APACHE II with lowest BS. Conclusion. The APACHE II outperformed all the ED and ICU models and was found to be the most appropriate model in predicting inhospital mortality of patients in the ICU in terms of discrimination, calibration, and accuracy of predicted probability. Except for MEWS, the rest of the models had fair discrimination and partially good calibration. Interestingly, although the REMS is less complicated than the SAPS II, both models exhibited similar performance. Clinicians can utilize the REMS as part of a larger clinical assessment to manage patients more effectively.