Performance of externally validated enhanced computer-aided versions of the National Early Warning Score in predicting mortality following an emergency admission to hospital in England: a cross-sectional study

Accuracy of automated computer-aided risk scoring systems to estimate the risk of COVID-19: a retrospective cohort study

BACKGROUND

In the UK National Health Service (NHS), the patient's vital signs are monitored and summarised into a National Early Warning Score (NEWS) score. A set of computer-aided risk scoring systems (CARSS) was developed and validated for predicting in-hospital mortality and sepsis in unplanned admission to hospital using NEWS and routine blood tests results. We sought to assess the accuracy of these models to predict the risk of COVID-19 in unplanned admissions during the first phase of the pandemic.

METHODS

Adult ( > = 18 years) non-elective admissions discharged (alive/deceased) between 11-March-2020 to 13-June-2020 from two acute hospitals with an index NEWS electronically recorded within ± 24 h of admission. We identified COVID-19 admission based on ICD-10 code 'U071' which was determined by COVID-19 swab test results (hospital or community). We assessed the performance of CARSS (CARS_N, CARS_NB, CARM_N, CARM_NB) for predicting the risk of COVID-19 in terms of discrimination (c-statistic) and calibration (graphically).

RESULTS

The risk of in-hospital mortality following emergency medical admission was 8.4% (500/6444) and 9.6% (620/6444) had a diagnosis of COVID-19. For predicting COVID-19 admissions, the CARS_N model had the highest discrimination 0.73 (0.71 to 0.75) and calibration slope 0.81 (0.72 to 0.89) compared to other CARSS models: CARM_N (discrimination:0.68 (0.66 to 0.70) and calibration slope 0.47 (0.41 to 0.54)), CARM_NB (discrimination:0.68 (0.65 to 0.70) and calibration slope 0.37 (0.31 to 0.43)), and CARS_NB (discrimination:0.68 (0.66 to 0.70) and calibration slope 0.56 (0.47 to 0.64)).

CONCLUSIONS

The CARS_N model is reasonably accurate for predicting the risk of COVID-19. It may be clinically useful as an early warning system at the time of admission especially to triage large numbers of unplanned admissions because it requires no additional data collection and is readily automated.

Implementation and clinical utility of a Computer-Aided Risk Score for Mortality (CARM): a qualitative study

OBJECTIVES

The Computer-Aided Risk Score for Mortality (CARM) estimates the risk of in-hospital mortality following acute admission to the hospital by automatically amalgamating physiological measures, blood tests, gender, age and COVID-19 status. Our aims were to implement the score with a small group of practitioners and understand their first-hand experience of interacting with the score in situ.

DESIGN

Pilot implementation evaluation study involving qualitative interviews.

SETTING

This study was conducted in one of the two National Health Service hospital trusts in the North of England in which the score was developed.

PARTICIPANTS

Medical, older person and ICU/anaesthetic consultants and specialist grade registrars (n=116) and critical outreach nurses (n=7) were given access to CARM. Nine interviews were conducted in total, with eight doctors and one critical care outreach nurse.

INTERVENTIONS

Participants were given access to the CARM score, visible after login to the patients' electronic record, along with information about the development and intended use of the score.

RESULTS

Four themes and 14 subthemes emerged from reflexive thematic analysis: (1) current use (including support or challenge clinical judgement and decision making, communicating risk of mortality and professional curiosity); (2) barriers and facilitators to use (including litigation, resource needs, perception of the evidence base, strengths and limitations), (3) implementation support needs (including roll-out and integration, access, training and education); and (4) recommendations for development (including presentation and functionality and potential additional data). Barriers and facilitators to use, and recommendations for development featured highly across most interviews.

CONCLUSION

Our in situ evaluation of the pilot implementation of CARM demonstrated its scope in supporting clinical decision making and communicating risk of mortality between clinical colleagues and with service users. It suggested to us barriers to implementation of the score. Our findings may support those seeking to develop, implement or improve the adoption of risk scores.

Development and validation of automated computer-aided risk scores to predict in-hospital mortality for emergency medical admissions with COVID-19: a retrospective cohort development and validation study

OBJECTIVES

There are no established mortality risk equations specifically for unplanned emergency medical admissions which include patients with SARS-19 (COVID-19). We aim to develop and validate a computer-aided risk score (CARMc19) for predicting mortality risk by combining COVID-19 status, the first electronically recorded blood test results and the National Early Warning Score (NEWS2).

DESIGN

Logistic regression model development and validation study.

SETTING

Two acute hospitals (York Hospital-model development data; Scarborough Hospital-external validation data).

PARTICIPANTS

Adult (aged ≥16 years) medical admissions discharged over a 24-month period with electronic NEWS and blood test results recorded on admission. We used logistic regression modelling to predict the risk of in-hospital mortality using two models: (1) CARMc19_N: age+sex+NEWS2 including subcomponents+COVID19; (2) CARMc19_NB: CARMc19_N in conjunction with seven blood test results and acute kidney injury score. Model performance was evaluated according to discrimination (c-statistic), calibration (graphically) and clinical usefulness at NEWS2 thresholds of 4+, 5+, 6+.

RESULTS

The risk of in-hospital mortality following emergency medical admission was similar in development and validation datasets (8.4% vs 8.2%). The c-statistics for predicting mortality for CARMc19_NB is better than CARMc19_N in the validation dataset (CARMc19_NB=0.88 (95% CI 0.86 to 0.90) vs CARMc19_N=0.86 (95% CI 0.83 to 0.88)). Both models had good calibration (CARMc19_NB=1.01 (95% CI 0.88 to 1.14) and CARMc19_N:0.95 (95% CI 0.83 to 1.06)). At all NEWS2 thresholds (4+, 5+, 6+) model, CARMc19_NB had better sensitivity and similar specificity.

CONCLUSIONS

We have developed a validated CARMc19 scores with good performance characteristics for predicting the risk of in-hospital mortality. Since the CARMc19 scores place no additional data collection burden on clinicians, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.

A systematic review of the discrimination and absolute mortality predicted by the National Early Warning Scores according to different cut-off values and prediction windows

BACKGROUND

Although early warning scores were intended to simply identify patients in need of life-saving interventions, prediction has become their commonest metric. This review examined variation in the ability of the National Early Warning Scores (NEWS) in adult patients to predict absolute mortality at different times and cut-offs values.

METHOD

Following PRISMA guidelines, all studies reporting NEWS and NEWS2 providing enough information to fulfil the review's aims were included.

RESULTS

From 121 papers identified, the average area under the Receiver Operating Characteristic curve (AUC) for mortality declined from 0.90 at 24-hours to 0.76 at 30-days. Studies with a low overall mortality had a higher AUC for 24-hour mortality, as did general ward patients compared to patients seen earlier in their treatment. 24-hour mortality increased from 1.8% for a NEWS ≥3 to 7.8% for NEWS ≥7. Although 24-hour mortality for NEWS <3 was only 0.07% these deaths accounted for 9% of all deaths within 24-hours; for NEWS <7 24-hour mortality was 0.23%, which accounted for 44% of all 24-hour deaths. Within 30-days of a NEWS recording 22% of all deaths occurred in patients with a NEWS <3, 52% in patients with a NEWS <5, and 75% in patient with a NEWS <7.

CONCLUSION

NEWS reliably identifies patients most and least likely to die within 24-hours, which is what it was designed to do. However, many patients identified to have a low risk of imminent death die within 30-days. NEWS mortality predictions beyond 24-hours are unreliable.

Use of the first National Early Warning Score recorded within 24 hours of admission to estimate the risk of in-hospital mortality in unplanned COVID-19 patients: a retrospective cohort study

OBJECTIVES

Although the National Early Warning Score (NEWS) and its latest version NEWS2 are recommended for monitoring deterioration in patients admitted to hospital, little is known about their performance in COVID-19 patients. We aimed to compare the performance of the NEWS and NEWS2 in patients with COVID-19 versus those without during the first phase of the pandemic.

DESIGN

A retrospective cross-sectional study.

SETTING

Two acute hospitals (Scarborough and York) are combined into a single dataset and analysed collectively.

PARTICIPANTS

Adult (≥18 years) non-elective admissions discharged between 11 March 2020 and 13 June 2020 with an index or on-admission NEWS2 electronically recorded within ±24 hours of admission to predict mortality at four time points (in-hospital, 24 hours, 48 hours and 72 hours) in COVID-19 versus non-COVID-19 admissions.

RESULTS

Out of 6480 non-elective admissions, 620 (9.6%) had a diagnosis of COVID-19. They were older (73.3 vs 67.7 years), more often male (54.7% vs 50.1%), had higher index NEWS (4 vs 2.5) and NEWS2 (4.6 vs 2.8) scores and higher in-hospital mortality (32.1% vs 5.8%). The c-statistics for predicting in-hospital mortality in COVID-19 admissions was significantly lower using NEWS (0.64 vs 0.74) or NEWS2 (0.64 vs 0.74), however, these differences reduced at 72hours (NEWS: 0.75 vs 0.81; NEWS2: 0.71 vs 0.81), 48 hours (NEWS: 0.78 vs 0.81; NEWS2: 0.76 vs 0.82) and 24hours (NEWS: 0.84 vs 0.84; NEWS2: 0.86 vs 0.84). Increasing NEWS2 values reflected increased mortality, but for any given value the absolute risk was on average 24% higher (eg, NEWS2=5: 36% vs 9%).

CONCLUSIONS

The index or on-admission NEWS and NEWS2 offers lower discrimination for COVID-19 admissions versus non-COVID-19 admissions. The index NEWS2 was not proven to be better than the index NEWS. For each value of the index NEWS/NEWS2, COVID-19 admissions had a substantially higher risk of mortality than non-COVID-19 admissions which reflects the increased baseline mortality risk of COVID-19.

Accuracy of early warning scores for predicting serious adverse events in pre-hospital traumatic injury

BACKGROUND

Traumatically injured patients are at higher risk of serious adverse events. Numerous physiological scoring systems are employed as diagnostic and/or prognostic tools. The objective of this study was to evaluate the scales most commonly used by emergency medical services for the early detection of prehospital serious adverse events.

METHODS

Design. Preliminary longitudinal prospective observational study without intervention study in adults with prehospital traumatic injury.

SETTING

The study was carried out in the public health system of Castile and León (Spain), from April 1, 2018 to October 31, 2019, involving seven advanced life support units and five hospitals.

PARTICIPANTS

Traumatically injured patients over 18 years of age who were stabilized and transferred in advanced life support units to their referral hospital.

MAIN OUTCOME MEASURES

Appearance of serious adverse events at the prehospital level at the scene or during the transfer to the emergency department.

RESULTS

A total of 346 patients were included in the study. The median age was 50 years (IQR: 38-65). 32 cases (7.8%) presented serious adverse events at the prehospital level. Areas under the curve for the detection of serious adverse events were obtained with the Prehospital Index (0.979; 95% CI: 0.94-1.00) and National Early Warning Score 2 (0.956; 95% CI: 0.90-1.00); p <0.001 for all scores. The Prehospital Index had a positive probability coefficient of 78.4 (95% CI: 62.8-68.6) and the National Early Warning Score 2 obtained 52.9 (95% CI: 39.7-65.6). A comparison of the curves was not significant for any of the scores studied (p> 0.05).

CONCLUSIONS

All scoring systems were able to detect prehospital serious adverse events early in traumatic injury; therefore, any of the scoring systems could be useful and represent an ideal tool for routine use by emergency medical services in cases of traumatic injury.

Identification of Serious Adverse Events in Patients with Traumatic Brain Injuries, from Prehospital Care to Intensive-Care Unit, Using Early Warning Scores

Traumatic brain injuries are complex situations in which the emergency medical services must quickly determine the risk of deterioration using minimal diagnostic methods. The aim of this study is to analyze whether the use of early warning scores can help with decision-making in these dynamic situations by determining the patients who need the intensive care unit. A prospective, multicentric cohort study without intervention was carried out on traumatic brain injury patients aged over 18 given advanced life support and taken to the hospital. Our study included a total of 209 cases. The total number of intensive-care unit admissions was 50 cases (23.9%). Of the scores analyzed, the National Early Warning Score2 was the best result presented with an area under the curve of 0.888 (0.81–0.94; p < 0.001) and an odds ratio of 25.4 (95% confidence interval (CI):11.2–57.5). The use of early warning scores (and specifically National Early Warning Score2) can help the emergency medical services to differentiate traumatic brain injury patients with a high risk of deterioration. The emergency medical services should use the early warning scores routinely in all cases for the early detection of high-risk situations.