Accuracy of automated computer-aided risk scoring systems to estimate the risk of COVID-19: a retrospective cohort study

BACKGROUND

In the UK National Health Service (NHS), the patient's vital signs are monitored and summarised into a National Early Warning Score (NEWS) score. A set of computer-aided risk scoring systems (CARSS) was developed and validated for predicting in-hospital mortality and sepsis in unplanned admission to hospital using NEWS and routine blood tests results. We sought to assess the accuracy of these models to predict the risk of COVID-19 in unplanned admissions during the first phase of the pandemic.

METHODS

Adult ( > = 18 years) non-elective admissions discharged (alive/deceased) between 11-March-2020 to 13-June-2020 from two acute hospitals with an index NEWS electronically recorded within ± 24 h of admission. We identified COVID-19 admission based on ICD-10 code 'U071' which was determined by COVID-19 swab test results (hospital or community). We assessed the performance of CARSS (CARS_N, CARS_NB, CARM_N, CARM_NB) for predicting the risk of COVID-19 in terms of discrimination (c-statistic) and calibration (graphically).

RESULTS

The risk of in-hospital mortality following emergency medical admission was 8.4% (500/6444) and 9.6% (620/6444) had a diagnosis of COVID-19. For predicting COVID-19 admissions, the CARS_N model had the highest discrimination 0.73 (0.71 to 0.75) and calibration slope 0.81 (0.72 to 0.89) compared to other CARSS models: CARM_N (discrimination:0.68 (0.66 to 0.70) and calibration slope 0.47 (0.41 to 0.54)), CARM_NB (discrimination:0.68 (0.65 to 0.70) and calibration slope 0.37 (0.31 to 0.43)), and CARS_NB (discrimination:0.68 (0.66 to 0.70) and calibration slope 0.56 (0.47 to 0.64)).

CONCLUSIONS

The CARS_N model is reasonably accurate for predicting the risk of COVID-19. It may be clinically useful as an early warning system at the time of admission especially to triage large numbers of unplanned admissions because it requires no additional data collection and is readily automated.

National early warning score (NEWS) 2 predicts hospital mortality from COVID-19 patients

Background

COVID-19 has a high risk of mortality, especially in patients with comorbid diseases such as cardiac disease, type 2 diabetes mellitus, chronic kidney disease, and hypertension. The National Early Warning Score (NEWS) is a tool that helps in identifying changes in patient conditions that require intensive treatment.

Objective

Analyzing NEWS-2 to identify the risk of death in COVID-19 patients.

Methods

This research was conducted from June to July 2020 by using quota sampling. The number of participants in this study was 112 participants (case group = 56 participants and control group = 56 participants). Participants were assessed for NEWS-2 and evaluated for their treatment outcomes. The analysis used in this study was the Chi-squared test and logistic regression with p < 0.05.

Results

45 participants died of having NEWS-2 score >5, and as many as 50 participants showed an improvement in their condition by having NEWS-2 score 5 (OR = 34.091; p < 0.001). The accuracy of NEWS-2's assessment of mortality of COVID-19 patients had a sensitivity of 80.4% and a specificity of 89.3%. There were several comorbid diseases that had a significant relationship on mortality of COVID-19 patients such as cardiac disease (β = 5.907; 1.107-31.527 95% CI; p = 0.038), T2DM (β = 3.143; 1.269-7.783 95% CI; p = 0.013), CKD (β = 3.851; 1.195-12.416 95% CI; p = 0.024), and hypertension (β = 2.820; 1.075-7.399 95% CI; p = 0.035).

Conclusion

The NEWS-2 can be used to identify the risk of death of COVID-19 patients.

The predictive power of the National Early Warning Score (NEWS) 2, as compared to NEWS, among patients assessed by a Rapid response team: A prospective multi-centre trial

Aim

Early identification of patients at risk of serious adverse events (SAEs) is of vital importance, yet it remains a challenging task. We investigated the predictive power of National Early Warning Score (NEWS) 2, as compared to NEWS, among patients assessed by a Rapid response team (RRT).

Methods

Prospective, observational cohort study on 898 consecutive patients assessed by the RRTs in 26 Swedish hospitals. For each patient, NEWS and NEWS 2 scores were uniformly calculated by the study team. The associations of NEWS and NEWS 2 scores with unanticipated admissions to Intensive care unit (ICU), mortality and in-hospital cardiac arrests (IHCA) within 24 h, and the composite of these three events were investigated using logistic regression. The predictive power of NEWS and NEWS 2 was assessed using the area under the receiver operating characteristic (AUROC) curves.

Results

The prognostic accuracy of NEWS/NEWS 2 in predicting mortality was acceptable (AUROC 0.69/0.67). In discriminating the composite outcome and unanticipated ICU admission, both NEWS and NEWS 2 were relatively weak (AUROC 0.62/0.62 and AUROC 0.59/0.60 respectively); for IHCA the performance was poor. There were no differences between NEWS and NEWS 2 as to the predictive power.

Conclusion

The prognostic accuracy of NEWS 2 to predict mortality within 24 h was acceptable. However, the prognostic accuracy of NEWS 2 to predict IHCA was poor. NEWS and NEWS 2 performed similar in predicting the risk of SAEs but their performances were not sufficient for use as a risk stratification tool in patients assessed by a RRT.

Predictive accuracy of enhanced versions of the on-admission National Early Warning Score in estimating the risk of COVID-19 for unplanned admission to hospital: a retrospective development and validation study

Background

The novel coronavirus SARS-19 produces ‘COVID-19’ in patients with symptoms. COVID-19 patients admitted to the hospital require early assessment and care including isolation. The National Early Warning Score (NEWS) and its updated version NEWS2 is a simple physiological scoring system used in hospitals, which may be useful in the early identification of COVID-19 patients. We investigate the performance of multiple enhanced NEWS2 models in predicting the risk of COVID-19.

Methods

Our cohort included unplanned adult medical admissions discharged over 3 months (11 March 2020 to 13 June 2020 ) from two hospitals (YH for model development; SH for external model validation). We used logistic regression to build multiple prediction models for the risk of COVID-19 using the first electronically recorded NEWS2 within ± 24 hours of admission. Model M0’ included NEWS2; model M1’ included NEWS2 + age + sex, and model M2’ extends model M1’ with subcomponents of NEWS2 (including diastolic blood pressure + oxygen flow rate + oxygen scale). Model performance was evaluated according to discrimination (c statistic), calibration (graphically), and clinical usefulness at NEWS2 ≥ 5.

Results

The prevalence of COVID-19 was higher in SH (11.0 %=277/2520) than YH (8.7 %=343/3924) with a higher first NEWS2 scores ( SH 3.2 vs YH 2.8) but similar in-hospital mortality (SH 8.4 % vs YH 8.2 %). The c-statistics for predicting the risk of COVID-19 for models M0’,M1’,M2’ in the development dataset were: M0’: 0.71 (95 %CI 0.68–0.74); M1’: 0.67 (95 %CI 0.64–0.70) and M2’: 0.78 (95 %CI 0.75–0.80)). For the validation datasets the c-statistics were: M0’ 0.65 (95 %CI 0.61–0.68); M1’: 0.67 (95 %CI 0.64–0.70) and M2’: 0.72 (95 %CI 0.69–0.75) ). The calibration slope was similar across all models but Model M2’ had the highest sensitivity (M0’ 44 % (95 %CI 38-50 %); M1’ 53 % (95 %CI 47-59 %) and M2’: 57 % (95 %CI 51-63 %)) and specificity (M0’ 75 % (95 %CI 73-77 %); M1’ 72 % (95 %CI 70-74 %) and M2’: 76 % (95 %CI 74-78 %)) for the validation dataset at NEWS2 ≥ 5.

Conclusions

Model M2’ appears to be reasonably accurate for predicting the risk of COVID-19. It may be clinically useful as an early warning system at the time of admission especially to triage large numbers of unplanned hospital admissions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06951-x.

National Early Warning Score Does Not Accurately Predict Mortality for Patients With Infection Outside the Intensive Care Unit: A Systematic Review and Meta-Analysis

Background: The prognostic value of the national early warning score (NEWS) in patients with infections remains controversial. We aimed to evaluate the prognostic accuracy of NEWS for prediction of in-hospital mortality in patients with infections outside the intensive care unit (ICU). Methods: We searched PubMed, Embase, and Scopus for related articles from January 2012 to April 2021. Sensitivity, specificity, and likelihood ratios were pooled by using the bivariate random-effects model. Overall prognostic performance was summarized by using the area under the curve (AUC). We performed subgroup analyses to assess the prognostic accuracy of NEWS in selected populations. Results: A total of 21 studies with 107,008 participants were included. The pooled sensitivity and specificity of NEWS were 0.71 and 0.60. The pooled AUC of NEWS was 0.70, which was similar to quick sequential organ failure assessment (qSOFA, AUC: 0.70) and better than systemic inflammatory response syndrome (SIRS, AUC: 0.60). However, the sensitivity (0.55) and AUC (0.63) of NEWS were poor in elder patients. The NEWS of 5 was more sensitive, which was a better threshold for activating urgent assessment and treatment. Conclusions: The NEWS had good diagnostic accuracy for early prediction of mortality in patients with infections outside the ICU, and the sensitivity and specificity were more moderate when compared with qSOFA and SIRS. Insufficient sensitivity and poor performance in the elder population may have limitations as an early warning score for adverse outcomes. NEWS should be used for continuous monitoring rather than a single time point predictive tool.

Dynamic individual vital sign trajectory early warning score (DyniEWS) versus snapshot national early warning score (NEWS) for predicting postoperative deterioration

Aims

International early warning scores (EWS) including the additive National Early Warning Score (NEWS) and logistic EWS currently utilise physiological snapshots to predict clinical deterioration. We hypothesised that a dynamic score including vital sign trajectory would improve discriminatory power.

Methods

Multicentre retrospective analysis of electronic health record data from postoperative patients admitted to cardiac surgical wards in four UK hospitals. Least absolute shrinkage and selection operator-type regression (LASSO) was used to develop a dynamic model (DyniEWS) to predict a composite adverse event of cardiac arrest, unplanned intensive care re-admission or in-hospital death within 24 h.

Results

A total of 13,319 postoperative adult cardiac patients contributed 442,461 observations of which 4234 (0.96%) adverse events in 24 h were recorded. The new dynamic model (AUC = 0.80 [95% CI 0.78−0.83], AUPRC = 0.12 [0.10−0.14]) outperforms both an updated snapshot logistic model (AUC = 0.76 [0.73−0.79], AUPRC = 0.08 [0.60−0.10]) and the additive National Early Warning Score (AUC = 0.73 [0.70−0.76], AUPRC = 0.05 [0.02−0.08]). Controlling for the false alarm rates to be at current levels using NEWS cut-offs of 5 and 7, DyniEWS delivers a 7% improvement in balanced accuracy and increased sensitivities from 41% to 54% at NEWS 5 and 18% to –30% at NEWS 7.

Conclusions

Using an advanced statistical approach, we created a model that can detect dynamic changes in risk of unplanned readmission to intensive care, cardiac arrest or in-hospital mortality and can be used in real time to risk-prioritise clinical workload.

Evaluation of a new respiratory monitoring tool "Early Warning ScoreO2" for patients admitted at the emergency department with dyspnea

BACKGROUND

Many scores derived from Early Warning Scores have been developed to detect patients at risk of poor outcome. Few of these scores incorporate the oxygen flow rate while this is a major marker in patients with respiratory complaint. We developed and evaluated a new automatable monitoring tool (Early Warning Score O₂: EWS.O₂) that incorporates cardio-respiratory parameters (Respiratory rate, Heart rate, SpO₂, and FiO₂ derived from oxygen flow rate), aiming to achieve early detection of poor outcome among patients with dyspnea.

METHODS

All patients presenting at an emergency department for dyspnea from June 2011 to June 2018 with available initial value (nurse triage) of respiratory parameters were included. Our primary endpoint was a composite criterion including the use of non-invasive ventilation, ICU admission and death. The Area under the Receiver Operating Characteristic curve (AUROC) of the SpO₂/FiO₂ index, NEWS, NEWS2, and the EWS.O₂ were compared, including in subgroup analysis by final diagnosis or oxygen supplementation.

RESULTS

Among the 1729 patients retrieved, the composite outcome was observed in 288 (16.7%). The EWS.O₂ displayed better or comparable predictive accuracy at triage (AUROC: 0.704, 95% CI 0.672-0.736) compared to NEWS (0.662, p < 0.01), NEWS2 (0.672, p = 0.02) and SpO₂/FiO₂ (0.695, p = 0.46).

CONCLUSIONS

This new ScoreO₂ is equivalent or superior to common early warning scores and index to predict poor outcome at first medical contact. This score may be automatically and continuously recorded with new closed-loop devices to titrate oxygen flow. Further prospective studies will allow to verify its accuracy at multiple time points of the patient's journey.

National early warning score (NEWS) and the new alternative SpO2 scale during rapid response team reviews: a prospective observational study

Background

The national early warning score (NEWS) enables early detection of in-hospital patient deterioration and timely activation of hospital’s rapid response team (RRT). NEWS was updated in 2017 to include a separate SpO₂ scale for those patients with type II respiratory failure (T2RF). In this study we investigated whether NEWS with and without the new SpO₂ scale for the T2RF patients is associated with immediate and in-hospital patient outcomes among the patients actually attended by the RRT.

Methods

We conducted a two-year prospective observational study including all adult RRT patients without limitations of medical treatment (LOMT) in a large Finnish university associated tertiary level hospital. According to the first vital signs measured by the RRT, we calculated NEWSs for the RRT patients and further utilized the new SpO₂ scale for the patients with confirmed T2RF. We used multivariate logistic regression and area under the receiver operating characteristic analyses to test NEWS’s accuracy to predict two distinct outcomes: RRT patient’s I) immediate need for intensive care and/or new LOMT and 2) in-hospital death or discharge with cerebral performance category >2 and/or LOMT.

Results

The final cohort consisted of 886 RRT patients attended for the first time during their hospitalization. Most common reasons for RRT activation were respiratory (343, 39%) and circulatory (226, 26%) problems. Cohort’s median (Q₁, Q₃) NEWS at RRT arrival was 8 (5, 10) and remained unchanged if the new SpO₂ scale was applied for the 104 patients with confirmed T2RF. Higher NEWS was independently associated with both immediate (OR 1.28; 95% CI 1.22–1.35) and in-hospital (1.15; 1.10–1.21) adverse outcomes. Further, NEWS had fair discrimination for both the immediate (AUROC 0.73; 0.69–0.77) and in-hospital (0.68; 0.64–0.72) outcomes. Utilizing the new SpO₂ scale for the patients with confirmed T2RF did not improve the discrimination capability (0.73; 0.69–0.76 and 0.68; 0.64–0.71) for these outcomes, respectively.

Conclusions

We found that in patients attended by a RRT, the NEWS predicts patient’s hospital outcome with moderate accuracy. We did not find any improvement using the new SpO2 scale in T2RF patients.

Using the National Early Warning Score (NEWS/NEWS 2) in different Intensive Care Units (ICUs) to predict the discharge location of patients

BACKGROUND

The National Early Warning Score (NEWS/NEWS 2) has been adopted across the National Health Service (NHS) in the U.K. as a method of escalating care for deteriorating patients. Intensive Care Unit (ICU) resources are limited and in high demand, with patient discharge a focal point for managing resources effectively. There are currently no universally accepted methods for assessing discharge of patients from an ICU, which can cause premature discharges and put patients at risk of subsequent deterioration, readmission to ICU or death.

METHODS

We tested the ability of the NEWS to discriminate patients within 24h of admission to an ICU in a U.S. hospital during 2001-2012, by their end discharge location: home; hospital ward; nursing facility; hospice and death. The NEWS performance was compared across five different ICU specialties, using the area under the receiver operating characteristic (AUROC) curve and a large vital signs database (n=2,723,055) collected from 28,523 critical care admissions.

RESULTS

The NEWS AUROC (95% CI) at 24h following admission: all patients 0.727 (0.709-0.745); Coronary Care Unit (CCU) 0.829 (0.821-0.837); Cardiac Surgery Recovery Unit (CSRU) 0.844 (0.838-0.850); Medical Intensive Care Unit (MICU) 0.778 (0.767-0.791); Surgical Intensive Care Unit (SICU) 0.775 (0.762-0.788); Trauma Surgical Intensive Care Unit (TSICU) 0.765 (0.751-0.773).

CONCLUSIONS

The NEWS has reasonable discrimination for any ICU patient's discharge location. The NEWS has greater ability to discriminate patients in the Coronary Care Unit (CCU) and Cardiac Surgery Recovery Unit (CSRU) compared to other ICU specialties. The NEWS has the real potential to be applied within a universal discharge planning tool for ICU, improving patient safety at the point of discharge.

Comparison of the National Early Warning Score (NEWS) and the Modified Early Warning Score (MEWS) for predicting admission and in-hospital mortality in elderly patients in the pre-hospital setting and in the emergency department

The aim of this study is to evaluate the usefulness of the pre-hospital National Early Warning Score (pNEWS) and the pre-hospital Modified Early Warning Score (pMEWS) for predicting admission and in-hospital mortality in elderly patients presenting to the emergency department (ED). We also compare the value of the pNEWS with that of the ED NEWS (eNEWS) and ED MEWS (eMEWS) for predicting admission and in-hospital mortality. This retrospective, single-centre observational study was carried out in the ED of Jikei University Kashiwa Hospital, in Chiba, Japan, from 1st April 2017 to 31st March 2018. All patients aged 65 years or older were included in this study. The pNEWS/eNEWS were derived from seven common physiological vital signs: respiratory rate, peripheral oxygen saturation, the presence of inhaled oxygen parameters, body temperature, systolic blood pressure, pulse rate and Alert, responds to Voice, responds to Pain, Unresponsive (AVPU) score, whereas the pMEWS/eMEWS were derived from six common physiological vital signs: respiratory rate, peripheral oxygen saturation, body temperature, systolic blood pressure, pulse rate and AVPU score. Discrimination was assessed by plotting the receiver operating characteristic (ROC) curve and calculating the area under the ROC curve (AUC). The median pNEWS, pMEWS, eNEWS and eMEWS were significantly higher at admission than at discharge (p < 0.001). The median pNEWS, pMEWS, eNEWS and eMEWS of non-survivors were significantly higher than those of the survivors (p < 0.001). The AUC for predicting admission was 0.559 for the pNEWS and 0.547 for the pMEWS. There was no significant difference between the AUCs of the pNEWS and the pMEWS for predicting admission (p = 0.102). The AUCs for predicting in-hospital mortality were 0.678 for the pNEWS and 0.652 for the pMEWS. There was no significant difference between the AUCs of the pNEWS and the pMEWS for predicting in-hospital mortality (p = 0.081). The AUC for predicting admission was 0.628 for the eNEWS and 0.591 for the eMEWS. The AUC of the eNEWS was significantly greater than that of the eMEWS for predicting admission (p < 0.001). The AUC for predicting in-hospital mortality was 0.789 for the eNEWS and 0.720 for the eMEWS. The AUC of the eNEWS was significantly greater than that of the eMEWS for predicting in-hospital mortality (p < 0.001). For admission and in-hospital mortality, the AUC of the eNEWS was significantly greater than that of the pNEWS (p < 0.001, p < 0.001), and the AUC of the eMEWS was significantly greater than that of the pMEWS (p < 0.01, p < 0.05). Our single-centre study has demonstrated the low utility of the pNEWS and the pMEWS as predictors of admission and in-hospital mortality in elderly patients, whereas the eNEWS and the eMEWS predicted admission and in-hospital mortality more accurately. Evidence from multicentre studies is needed before introducing pre-hospital versions of risk-scoring systems.

Usage of NEWS for prediction of mortality and in-hospital cardiac arrest rates in a Turkish university hospital

Background/aim

Early warning scores (EWS), widely used around the world but not yet in Turkey, are composed of physiological parameters designed to determine potentially worsening patients to perform necessary interventions in time. The aim is to determine the national EWS (NEWS) of the patients to assess the relation between this score and length of hospital stay (LOHS), transfer to the ICU, 24-h and 28-day mortality rates, and the frequency of in-hospital cardiac arrest (IHCA).

Materials and methods

NEWS of all the patients in the internal medicine clinics were calculated via a point prevalence study. The LOHS, transfer to the ICU, rates of mortality in the 24-h and 28-day period, IHCA rate, and the period of the stay in the ICU parameters were determined.

Results

Out of 104 patients, 84 of them had low scores (<5), while 20 had mid/high scores. In mid/high-score group (score ≥ 5), transfer to the ICU, IHCA rate, and mortality rates within 24 h and 28 days were significantly higher.

Conclusion

In this, the first prospective study about EWS in Turkey, 24-h and 28-day mortality rates, transfer to the ICU, and IHCA frequency of the patients with mid/high NEWS were higher.

late admission to intensive care

OBJECTIVES

We investigated the national early warning scores (NEWSs) and related outcomes of patients in a tertiary referral center's multidisciplinary emergency department (ED). Patients were further categorized into three groups: triaged directly to intensive care unit (EDICU), triaged to general ward with later ICU admission (EDwardICU) and triaged to general ward (EDward). NEWSs and subsequent outcomes among these sub groups were compared.

METHODS

We conducted a prospective one-month cohort study in Tampere University Hospital's ED, Finland. ED-NEWSs were obtained for all adult patients without treatment limitations, and control (ward) NEWSs were further obtained for the EDwardICU and EDward patients.

RESULTS

Cohort consisted of 1,354 patients with a median ED-NEWS of 2, and higher ED-NEWS was associated with in-hospital mortality (OR 1.26, 95% CI 1.11-1.42; AUROC 0.75, 0.64‒0.86, p < 0.001) and 30-day mortality (OR 1.27, 1.17-1.39; AUROC 0.78, 0.71‒0.84, p < 0.001) irrespective of age and comorbidity. There were 64 patients in EDICU group, 12 patients in EDwardICU group and 1,278 patients in EDward group with median ED-NEWSs of 7, 3 and 2 (p < 0.001), respectively. After the first 24 h in wards, median NEWSs of the EDwardICU patients had substantially increased as compared with EDward patients (6 vs. 2, p < 0.001). There were no statistical differences in last NEWS before ICU admission between the EDICU and EDwardICU patients (7 vs. 8, p = 0.534), or in ICU severity-of-illness scores or patient outcomes.

CONCLUSIONS

ED-NEWS is independently associated with in-hospital and 30-day mortality with acceptable discrimination capability. Direct and late ICU admissions occurred with comparable NEWSs at admission.