A systematic review of neonatal treatment intensity scores and their potential application in low-resource setting hospitals for predicting mortality, morbidity and estimating resource use

Opportunities and Challenges of Prognostic Models for Extremely Preterm Infants

Predicting the short- and long-term outcomes of extremely preterm infants remains a challenge. Multivariable prognostic models might be valuable tools for clinicians, parents, and policymakers for providing accurate outcome estimates. In this perspective, we discuss the opportunities and challenges of using prognostic models in extremely preterm infants at population and individual levels. At a population level, these models could support the development of guidelines for decisions about treatment limits and may support policy processes such as benchmarking and resource allocation. At an individual level, these models may enhance prenatal counselling conversations by considering multiple variables and improving transparency about expected outcomes. Furthermore, they may improve consistency in projections shared with parents. For the development of prognostic models, we discuss important considerations such as predictor and outcome measure selection, clinical impact assessment, and generalizability. Lastly, future recommendations for developing and using prognostic models are suggested. Importantly, the purpose of a prognostic model should be clearly defined, and integrating these models into prenatal counselling requires thoughtful consideration.

Patterns of antibiotic use, pathogens, and prediction of mortality in hospitalized neonates and young infants with sepsis: A global neonatal sepsis observational cohort study (NeoOBS)

BACKGROUND

There is limited data on antibiotic treatment in hospitalized neonates in low- and middle-income countries (LMICs). We aimed to describe patterns of antibiotic use, pathogens, and clinical outcomes, and to develop a severity score predicting mortality in neonatal sepsis to inform future clinical trial design.

METHODS AND FINDINGS

Hospitalized infants <60 days with clinical sepsis were enrolled during 2018 to 2020 by 19 sites in 11 countries (mainly Asia and Africa). Prospective daily observational data was collected on clinical signs, supportive care, antibiotic treatment, microbiology, and 28-day mortality. Two prediction models were developed for (1) 28-day mortality from baseline variables (baseline NeoSep Severity Score); and (2) daily risk of death on IV antibiotics from daily updated assessments (NeoSep Recovery Score). Multivariable Cox regression models included a randomly selected 85% of infants, with 15% for validation. A total of 3,204 infants were enrolled, with median birth weight of 2,500 g (IQR 1,400 to 3,000) and postnatal age of 5 days (IQR 1 to 15). 206 different empiric antibiotic combinations were started in 3,141 infants, which were structured into 5 groups based on the World Health Organization (WHO) AWaRe classification. Approximately 25.9% (n = 814) of infants started WHO first line regimens (Group 1-Access) and 13.8% (n = 432) started WHO second-line cephalosporins (cefotaxime/ceftriaxone) (Group 2-"Low" Watch). The largest group (34.0%, n = 1,068) started a regimen providing partial extended-spectrum beta-lactamase (ESBL)/pseudomonal coverage (piperacillin-tazobactam, ceftazidime, or fluoroquinolone-based) (Group 3-"Medium" Watch), 18.0% (n = 566) started a carbapenem (Group 4-"High" Watch), and 1.8% (n = 57) a Reserve antibiotic (Group 5, largely colistin-based), and 728/2,880 (25.3%) of initial regimens in Groups 1 to 4 were escalated, mainly to carbapenems, usually for clinical deterioration (n = 480; 65.9%). A total of 564/3,195 infants (17.7%) were blood culture pathogen positive, of whom 62.9% (n = 355) had a gram-negative organism, predominantly Klebsiella pneumoniae (n = 132) or Acinetobacter spp. (n = 72). Both were commonly resistant to WHO-recommended regimens and to carbapenems in 43 (32.6%) and 50 (71.4%) of cases, respectively. MRSA accounted for 33 (61.1%) of 54 Staphylococcus aureus isolates. Overall, 350/3,204 infants died (11.3%; 95% CI 10.2% to 12.5%), 17.7% if blood cultures were positive for pathogens (95% CI 14.7% to 21.1%, n = 99/564). A baseline NeoSep Severity Score had a C-index of 0.76 (0.69 to 0.82) in the validation sample, with mortality of 1.6% (3/189; 95% CI: 0.5% to 4.6%), 11.0% (27/245; 7.7% to 15.6%), and 27.3% (12/44; 16.3% to 41.8%) in low (score 0 to 4), medium (5 to 8), and high (9 to 16) risk groups, respectively, with similar performance across subgroups. A related NeoSep Recovery Score had an area under the receiver operating curve for predicting death the next day between 0.8 and 0.9 over the first week. There was significant variation in outcomes between sites and external validation would strengthen score applicability.

CONCLUSION

Antibiotic regimens used in neonatal sepsis commonly diverge from WHO guidelines, and trials of novel empiric regimens are urgently needed in the context of increasing antimicrobial resistance (AMR). The baseline NeoSep Severity Score identifies high mortality risk criteria for trial entry, while the NeoSep Recovery Score can help guide decisions on regimen change. NeoOBS data informed the NeoSep1 antibiotic trial (ISRCTN48721236), which aims to identify novel first- and second-line empiric antibiotic regimens for neonatal sepsis.

TRIAL REGISTRATION

ClinicalTrials.gov, (NCT03721302).

Evaluation of Modified Extended Sick Neonate Score to Predict In-Hospital Mortality among Newborns Admitted to Resource-Poor Settings in Rural India

OBJECTIVES

To devise a score based on eight extended sick neonatal score (ESNS) variables by adding two more variables, birth weight, and gestational age to check its efficacy.

METHODS

A retrospective study was conducted on 521 neonates admitted from January 2018 to June 2020 in the neonatal intensive care unit (NICU) of a tertiary care center. The proposed score has the following components, respiratory effort, heart rate, axillary temperature, capillary refill time, random blood sugar, pulse oximeter saturation, Moro reflex, modified Downes score, gestation age, and birth weight. Each was scored as 0, 1, and 2. The total score for each neonate was calculated from the records available and outcome documented. Receiver operating characteristic (ROC) curve was generated; and a cutoff score was derived to predict mortality and compared with modified sick neonatal score (MSNS).

RESULTS

Modified ESNS has a strong correlation with the outcome. Area under the ROC curve was 0.995 (95% CI: 0.925-0.985) for modified ESNS and 0.933 (95% CI: 0.901-0.976) for MSNS. The optimum cutoff values for predicting mortality were 15.5 for modified ESNS and 12.5 for MSNS. For a cutoff score of ≤ 15, sensitivity and specificity were 86.27% and 86.60% for modified ESNS and 90.20% and 84.89%, respectively, for MSNS, in predicting mortality. Positive and negative predictive values were 41.12% and 98.31% for modified ESNS and 39.32% and 98.76%, respectively for MSNS.

CONCLUSION

The modified ESNS can predict in-hospital mortality among neonates, with good sensitivity and specificity.

External validation of inpatient neonatal mortality prediction models in high-mortality settings

BACKGROUND

Two neonatal mortality prediction models, the Neonatal Essential Treatment Score (NETS) which uses treatments prescribed at admission and the Score for Essential Neonatal Symptoms and Signs (SENSS) which uses basic clinical signs, were derived in high-mortality, low-resource settings to utilise data more likely to be available in these settings. In this study, we evaluate the predictive accuracy of two neonatal prediction models for all-cause in-hospital mortality.

METHODS

We used retrospectively collected routine clinical data recorded by duty clinicians at admission from 16 Kenyan hospitals used to externally validate and update the SENSS and NETS models that were initially developed from the data from the largest Kenyan maternity hospital to predict in-hospital mortality. Model performance was evaluated by assessing discrimination and calibration. Discrimination, the ability of the model to differentiate between those with and without the outcome, was measured using the c-statistic. Calibration, the agreement between predictions from the model and what was observed, was measured using the calibration intercept and slope (with values of 0 and 1 denoting perfect calibration).

RESULTS

At initial external validation, the estimated mortality risks from the original SENSS and NETS models were markedly overestimated with calibration intercepts of - 0.703 (95% CI - 0.738 to - 0.669) and - 1.109 (95% CI - 1.148 to - 1.069) and too extreme with calibration slopes of 0.565 (95% CI 0.552 to 0.577) and 0.466 (95% CI 0.451 to 0.480), respectively. After model updating, the calibration of the model improved. The updated SENSS and NETS models had calibration intercepts of 0.311 (95% CI 0.282 to 0.350) and 0.032 (95% CI - 0.002 to 0.066) and calibration slopes of 1.029 (95% CI 1.006 to 1.051) and 0.799 (95% CI 0.774 to 0.823), respectively, while showing good discrimination with c-statistics of 0.834 (95% CI 0.829 to 0.839) and 0.775 (95% CI 0.768 to 0.782), respectively. The overall calibration performance of the updated SENSS and NETS models was better than any existing neonatal in-hospital mortality prediction models externally validated for settings comparable to Kenya.

CONCLUSION

Few prediction models undergo rigorous external validation. We show how external validation using data from multiple locations enables model updating and improving their performance and potential value. The improved models indicate it is possible to predict in-hospital mortality using either treatments or signs and symptoms derived from routine neonatal data from low-resource hospital settings also making possible their use for case-mix adjustment when contrasting similar hospital settings.

Internal validation and evaluation of the predictive performance of models based on the PRISM-3 (Pediatric Risk of Mortality) and PIM-3 (Pediatric Index of Mortality) scoring systems for predicting mortality in Pediatric Intensive Care Units (PICUs)

PURPOSE

The study was aimed to assess the prognostic power The Pediatric Risk of Mortality-3 (PRISM-3) and the Pediatric Index of Mortality-3 (PIM-3) to predict in-hospital mortality in a sample of patients admitted to the PICUs.

DESIGN AND METHODS

The study was performed to include all children younger than 18 years of age admitted to receive critical care in two hospitals, Mashhad, northeast of Iran from December 2017 to November 2018. The predictive performance was quantified in terms of the overall performance by measuring the Brier Score (BS) and standardized mortality ratio (SMR), discrimination by assessing the AUC, and calibration by applying the Hosmer-Lemeshow test.

RESULTS

A total of 2446 patients with the median age of 4.2 months (56% male) were included in the study. The PICU and in-hospital mortality were 12.4 and 16.14%, respectively. The BS of the PRISM-3 and PIM-3 was 0.088 and 0.093 for PICU mortality and 0.108 and 0.113 for in-hospital mortality. For the entire sample, the SMR of the PRISM-3 and PIM-3 were 1.34 and 1.37 for PICU mortality and 1.73 and 1.78 for in-hospital mortality, respectively. The PRISM-3 demonstrated significantly higher discrimination power in comparison with the PIM-3 (AUC = 0.829 vs 0.745) for in-hospital mortality. (AUC = 0.779 vs 0.739) for in-hospital mortality. The HL test revealed poor calibration for both models in both outcomes.

CONCLUSIONS

The performance measures of PRISM-3 were better than PIM-3 in both PICU and in-hospital mortality. However, further recalibration and modification studies are required to improve the predictive power to a clinically acceptable level before daily clinical use.

PRACTICE IMPLICATIONS

The calibration of the PRISM-3 model is more satisfactory than PIM-3, however both models have fair discrimination power.

Machine Learning Algorithms to Predict Mortality of Neonates on Mechanical Intubation for Respiratory Failure

BACKGROUND

Early identification of critically ill neonates with poor outcomes can optimize therapeutic strategies. We aimed to examine whether machine learning (ML) methods can improve mortality prediction for neonatal intensive care unit (NICU) patients on intubation for respiratory failure.

METHODS

A total of 1734 neonates with respiratory failure were randomly divided into training (70%, n = 1214) and test (30%, n = 520) sets. The primary outcome was the probability of NICU mortality. The areas under the receiver operating characteristic curves (AUCs) of several ML algorithms were compared with those of the conventional neonatal illness severity scoring systems including the NTISS and SNAPPE-II.

RESULTS

For NICU mortality, the random forest (RF) model showed the highest AUC (0.939 (0.921-0.958)) for the prediction of neonates with respiratory failure, and the bagged classification and regression tree model demonstrated the next best results (0.915 (0.891-0.939)). The AUCs of both models were significantly better than the traditional NTISS (0.836 (0.800-0.871)) and SNAPPE-II scores (0.805 (0.766-0.843)). The superior performances were confirmed by higher accuracy and F1 score and better calibration, and the superior and net benefit was confirmed by decision curve analysis. In addition, Shapley additive explanation (SHAP) values were utilized to explain the RF prediction model.

CONCLUSIONS

Machine learning algorithms increase the accuracy and predictive ability for mortality of neonates with respiratory failure compared with conventional neonatal illness severity scores. The RF model is suitable for clinical use in the NICU, and clinicians can gain insights and have better communication with families in advance.

Machine Learning Approaches to Predict In-Hospital Mortality among Neonates with Clinically Suspected Sepsis in the Neonatal Intensive Care Unit

Background: preterm and critically ill neonates often experience clinically suspected sepsis during their prolonged hospitalization in the neonatal intensive care unit (NICU), which can be the initial sign of final adverse outcomes. Therefore, we aimed to utilize machine learning approaches to predict neonatal in-hospital mortality through data-driven learning. Methods: a total of 1095 neonates who experienced clinically suspected sepsis in a tertiary-level NICU in Taiwan between August 2017 and July 2020 were enrolled. Clinically suspected sepsis was defined based on clinical features and laboratory criteria and the administration of empiric antibiotics by clinicians. The variables used for analysis included patient demographics, clinical features, laboratory data, and medications. The machine learning methods used included deep neural network (DNN), k-nearest neighbors, support vector machine, random forest, and extreme gradient boost. The performance of these models was evaluated using the area under the receiver operating characteristic curve (AUC). Results: the final in-hospital mortality of this cohort was 8.2% (90 neonates died). A total of 765 (69.8%) and 330 (30.2%) patients were randomly assigned to the training and test sets, respectively. Regarding the efficacy of the single model that most accurately predicted the outcome, DNN exhibited the greatest AUC (0.923, 95% confidence interval [CI] 0.953–0.893) and the best accuracy (95.64%, 95% CI 96.76–94.52%), Cohen’s kappa coefficient value (0.74, 95% CI 0.79–0.69) and Matthews correlation coefficient value (0.75, 95% CI 0.80–0.70). The top three most influential variables in the DNN importance matrix plot were the requirement of ventilator support at the onset of suspected sepsis, the feeding conditions, and intravascular volume expansion. The model performance was indistinguishable between the training and test sets. Conclusions: the DNN model was successfully established to predict in-hospital mortality in neonates with clinically suspected sepsis, and the machine learning algorithm is applicable for clinicians to gain insights and have better communication with families in advance.

Prediction modelling of inpatient neonatal mortality in high-mortality settings

OBJECTIVE

Prognostic models aid clinical decision making and evaluation of hospital performance. Existing neonatal prognostic models typically use physiological measures that are often not available, such as pulse oximetry values, in routine practice in low-resource settings. We aimed to develop and validate two novel models to predict all cause in-hospital mortality following neonatal unit admission in a low-resource, high-mortality setting.

STUDY DESIGN AND SETTING

We used basic, routine clinical data recorded by duty clinicians at the time of admission to derive (n=5427) and validate (n=1627) two novel models to predict in-hospital mortality. The Neonatal Essential Treatment Score (NETS) included treatments prescribed at the time of admission while the Score for Essential Neonatal Symptoms and Signs (SENSS) used basic clinical signs. Logistic regression was used, and performance was evaluated using discrimination and calibration.

RESULTS

At derivation, c-statistic (discrimination) for NETS was 0.92 (95% CI 0.90 to 0.93) and that for SENSS was 0.91 (95% CI 0.89 to 0.93). At external (temporal) validation, NETS had a c-statistic of 0.89 (95% CI 0.86 to 0.92) and SENSS 0.89 (95% CI 0.84 to 0.93). The calibration intercept for NETS was -0.72 (95% CI -0.96 to -0.49) and that for SENSS was -0.33 (95% CI -0.56 to -0.11).

CONCLUSION

Using routine neonatal data in a low-resource setting, we found that it is possible to predict in-hospital mortality using either treatments or signs and symptoms. Further validation of these models may support their use in treatment decisions and for case-mix adjustment to help understand performance variation across hospitals.

Changepoint analysis of gestational age and birth weight: proposing a refinement of Diagnosis Related Groups

BACKGROUND

Although the complexity and length of treatment is connected to the newborn's maturity and birth weight, most case-mix grouping schemes classify newborns by birth weight alone. The objective of this study was to determine whether the definition of thresholds based on a changepoint analysis of variability of birth weight and gestational age contributes to a more homogenous classification.

METHODS

This retrospective observational study was conducted at a Tertiary Care Center with Level III Neonatal Intensive Care and included neonate cases from 2016 through 2018. The institutional database of routinely collected health data was used. The design of this cohort study was explorative. The cases were categorized according to WHO gestational age classes and SwissDRG birth weight classes. A changepoint analysis was conducted. Cut-off values were determined.

RESULTS

When grouping the cases according to the calculated changepoints, the variability within the groups with regard to case related costs could be reduced. A refined grouping was achieved especially with cases of >2500 g birth weight. An adjusted Grouping Grid for practical purposes was developed.

CONCLUSIONS

A novel method of classification of newborn cases by changepoint analysis was developed, providing the possibility to assign costs or outcome indicators to grouping mechanisms by gestational age and birth weight combined.

Modified Sick Neonatal Score (MSNS): A Novel Neonatal Disease Severity Scoring System for Resource-Limited Settings

Neonatal disease severity scoring systems are needed to make standardized comparison between performances of different units and to give prognostic information to parents of individual babies admitted. Existing scoring systems are unsuitable for resource-limited settings which lack investigations like pH, pO₂/FiO₂ ratio, and base excess. This study was planned to evaluate Modified Sick Neonatal Score (MSNS), a novel neonatal disease severity score designed for resource-constrained settings. It was a facility-based cross-sectional analytical study, conducted in the “Special Newborn Care Unit” (SNCU) of government district hospital, attached to Kasturba Medical College, Mangalore, India from November 2016 to October 2017. A convenience sample of 585 neonates was included. Disease severity was assessed immediately at admission using MSNS. MSNS had 8 parameters with 0, 1, and 2 scores for each. 41% of study population was preterm (n=240), and 84.1% had birth weight less than 2500 grams (n=492). The mean (SD) of the total MSNS scores for neonates who expired and discharged was, respectively, 8.22 (2.96) and 13.4 (2.14), a difference being statistically significant at P < 0.001. Expired newborns had statistically significant frequency of lower scores across each of the parameters. An optimum cutoff score of ≤10 with 80% sensitivity and 88.8% specificity in predicting mortality was obtained when the ROC curve was generated with the MSNS score as the test variable. Area under the curve was 0.913 (95% CI: 0.879–0.946). In conclusion, MSNS is a practicable disease severity score in resource-restricted settings like district SNCUs. It is for application in both term and preterm neonates. Total score ≤10 has a good sensitivity and specificity in predicting mortality of admitted neonates when used early during the course of hospitalization. MSNS could be used as a tool to compare performance of SNCUs and also enable early referral of individual cases to units with better facilities.