The effect of fractional inspired oxygen concentration on early warning score performance: A database analysis

Real-world implementation of the National Early Warning Score-2 in an acute respiratory unit

INTRODUCTION

The National Early Warning Score-2 (NEWS-2) is used to detect deteriorating patients in hospital settings. We aimed to understand how NEWS-2 functions in the real-life setting of an acute respiratory unit.

METHODS

Clinical observations data were extracted for adult patients (age ≥18 years), admitted under the care of respiratory medicine services from July to December 2019, who had at least one recorded task relating to clinical deterioration. The timing and nature of urgent out-of-hours medical reviews (escalations) were extracted through manual review of the case notes.

RESULTS

The data set comprised 765 admission episodes (48.9% women) with a mean (SD) age of 69.3 (14.8). 8971 out of 35 991 out-of-hours observation sets (24.9%) had a NEWS-2 ≥5, and 586 of these (6.5%) led to an escalation. Out of 687 escalations, 101 (14.7%) were associated with observation sets with NEWS-2<5. Rising oxygen requirement and extreme values of individual observations were associated with an increased risk of escalation. 57.6% of escalations resulted in a change in treatment. Inpatient mortality was higher in patients who were escalated at least once, compared with those who were not escalated.

CONCLUSIONS

Most observation sets with NEWS-2 scores ≥5 did not lead to a medical escalation in an acute respiratory setting out-of-hours, but more than half of escalations resulted in a change in treatment. Rising oxygen requirement is a key indicator of respiratory patient acuity which appears to influence the decision to request urgent out-of-hours medical reviews.

Predictive value of the National Early Warning Score 2 for hospitalised patients with viral respiratory illness is improved by the addition of inspired oxygen fraction as a weighted variable

OBJECTIVES

The National Early Warning Score 2 (NEWS2) is validated for predicting acute deterioration, however, the binary grading of inspired oxygen fraction (FiO2) may limit performance. We evaluated the incorporation of FiO2 as a weighted categorical variable on NEWS2 prediction of patient deterioration.

SETTING

Two hospitals at a single medical centre, Guy's and St Thomas' NHS Foundation Trust.

DESIGN

Retrospective cohort of all ward admissions, with a viral respiratory infection (SARS-CoV-2/influenza).

PARTICIPANTS

3704 adult ward admissions were analysed between 01 January 2017 and 31 December 2021.

METHODS

The NEWS-FiO2 score transformed FiO2 into a weighted categorical variable, from 0 to 3 points, substituting the original 0/2 points. The primary outcome was a composite of cardiac arrest, unplanned critical care admission or death within 24 hours of the observation. Sensitivity, positive predictive value (PPV), number needed to evaluate (NNE) and area under the receiver operating characteristic curve (AUROC) were calculated. Failure analysis for the time from trigger to outcome was compared by log-rank test.

RESULTS

The mean age was 60.4±19.4 years, 52.6% were men, with a median Charlson Comorbidity of 0 (IQR 3). The primary outcome occurred in 493 (13.3%) patients, and the weighted FiO2 score was strongly associated with the outcome (p=<0.001). In patients receiving supplemental oxygen, 78.5% of scores were reclassified correctly and the AUROC was 0.81 (95% CI 0.81 to 0.81) for NEWS-FiO2 versus 0.77 (95% CI 0.77 to 0.77) for NEWS2. This improvement persisted in the whole cohort with a significantly higher failure rate for NEWS-FiO2 (p=<0.001). At the 5-point threshold, the PPV increased by 22.0% (NNE 6.7) for only a 3.9% decrease in sensitivity.

CONCLUSION

Transforming FiO2 into a weighted categorical variable improved NEWS2 prediction for patient deterioration, significantly improving the PPV. Prospective external validation is required before institutional implementation.

Development of a dynamic prediction model for unplanned ICU admission and mortality in hospitalized patients

Frequent assessment of the severity of illness for hospitalized patients is essential in clinical settings to prevent outcomes such as in-hospital mortality and unplanned admission to the intensive care unit (ICU). Classical severity scores have been developed typically using relatively few patient features. Recently, deep learning-based models demonstrated better individualized risk assessments compared to classic risk scores, thanks to the use of aggregated and more heterogeneous data sources for dynamic risk prediction. We investigated to what extent deep learning methods can capture patterns of longitudinal change in health status using time-stamped data from electronic health records. We developed a deep learning model based on embedded text from multiple data sources and recurrent neural networks to predict the risk of the composite outcome of unplanned ICU transfer and in-hospital death. The risk was assessed at regular intervals during the admission for different prediction windows. Input data included medical history, biochemical measurements, and clinical notes from a total of 852,620 patients admitted to non-intensive care units in 12 hospitals in Denmark's Capital Region and Region Zealand during 2011-2016 (with a total of 2,241,849 admissions). We subsequently explained the model using the Shapley algorithm, which provides the contribution of each feature to the model outcome. The best model used all data modalities with an assessment rate of 6 hours, a prediction window of 14 days and an area under the receiver operating characteristic curve of 0.898. The discrimination and calibration obtained with this model make it a viable clinical support tool to detect patients at higher risk of clinical deterioration, providing clinicians insights into both actionable and non-actionable patient features.

Predicting outcome in acute respiratory admissions using patterns of National Early Warning Scores

AIMS

Accurately predicting risk of patient deterioration is vital. Altered physiology in chronic disease affects the prognostic ability of vital signs based early warning score systems. We aimed to assess the potential of early warning score patterns to improve outcome prediction in patients with respiratory disease.

METHODS

Patients admitted under respiratory medicine between April 2015 and March 2017 had their National Early Warning Score 2 (NEWS2) calculated retrospectively from vital sign observations. Prediction models (including temporal patterns) were constructed and assessed for ability to predict death within 24 hours using all observations collected not meeting exclusion criteria. The best performing model was tested on a validation cohort of admissions from April 2017 to March 2019.

RESULTS

The derivation cohort comprised 7,487 admissions and the validation cohort included 8,739 admissions. Adding the maximum score in the preceding 24 hours to the most recently recorded NEWS2 improved area under the receiver operating characteristic curve for death in 24 hours from 0.888 (95% confidence interval (CI) 0.881-0.895) to 0.902 (95% CI 0.895-0.909) in the overall respiratory population.

CONCLUSION

Combining the most recently recorded score and the maximum NEWS2 score from the preceding 24 hours demonstrated greater accuracy than using snapshot NEWS2. This simple inclusion of a scoring pattern should be considered in future iterations of early warning scoring systems.

Dynamic early warning scores for predicting clinical deterioration in patients with respiratory disease

Background

The National Early Warning Score-2 (NEWS-2) is used to detect patient deterioration in UK hospitals but fails to take account of the detailed granularity or temporal trends in clinical observations. We used data-driven methods to develop dynamic early warning scores (DEWS) to address these deficiencies, and tested their accuracy in patients with respiratory disease for predicting (1) death or intensive care unit admission, occurring within 24 h (D/ICU), and (2) clinically significant deterioration requiring urgent intervention, occurring within 4 h (CSD).

Methods

Clinical observations data were extracted from electronic records for 31,590 respiratory in-patient episodes from April 2015 to December 2020 at a large acute NHS Trust. The timing of D/ICU was extracted for all episodes. 1100 in-patient episodes were annotated manually to record the timing of CSD, defined as a specific event requiring a change in treatment. Time series features were entered into logistic regression models to derive DEWS for each of the clinical outcomes. Area under the receiver operating characteristic curve (AUROC) was the primary measure of model accuracy.

Results

AUROC (95% confidence interval) for predicting D/ICU was 0.857 (0.852–0.862) for NEWS-2 and 0.906 (0.899–0.914) for DEWS in the validation data. AUROC for predicting CSD was 0.829 (0.817–0.842) for NEWS-2 and 0.877 (0.862–0.892) for DEWS. NEWS-2 ≥ 5 had sensitivity of 88.2% and specificity of 54.2% for predicting CSD, while DEWS ≥ 0.021 had higher sensitivity of 93.6% and approximately the same specificity of 54.3% for the same outcome. Using these cut-offs, 315 out of 347 (90.8%) CSD events were detected by both NEWS-2 and DEWS, at the time of the event or within the previous 4 h; 12 (3.5%) were detected by DEWS but not by NEWS-2, while 4 (1.2%) were detected by NEWS-2 but not by DEWS; 16 (4.6%) were not detected by either scoring system.

Conclusion

We have developed DEWS that display greater accuracy than NEWS-2 for predicting clinical deterioration events in patients with respiratory disease. Prospective validation studies are required to assess whether DEWS can be used to reduce missed deteriorations and false alarms in real-life clinical settings.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12931-022-02130-6.

Artificial intelligence and clinical deterioration

PURPOSE OF REVIEW

To provide an overview of the systems being used to identify and predict clinical deterioration in hospitalised patients, with focus on the current and future role of artificial intelligence (AI).

RECENT FINDINGS

There are five leading AI driven systems in this field: the Advanced Alert Monitor (AAM), the electronic Cardiac Arrest Risk Triage (eCART) score, Hospital wide Alert Via Electronic Noticeboard, the Mayo Clinic Early Warning Score, and the Rothman Index (RI). Each uses Electronic Patient Record (EPR) data and machine learning to predict adverse events. Less mature but relevant evolutions are occurring in the fields of Natural Language Processing, Time and Motion Studies, AI Sepsis and COVID-19 algorithms.

SUMMARY

Research-based AI-driven systems to predict clinical deterioration are increasingly being developed, but few are being implemented into clinical workflows. Escobar et al. (AAM) provide the current gold standard for robust model development and implementation methodology. Multiple technologies show promise, however, the pathway to meaningfully affect patient outcomes remains challenging.

Using NEWS2 to triage newly admitted patients with COVID-19

BACKGROUND

The coronavirus disease 2019 (COVID-19) pandemic has spread globally and caused a major worldwide health crisis. Patients who are affected more seriously by COVID-19 usually deteriorate rapidly and need further intensive care.

AIMS AND OBJECTIVES

We aimed to assess the performance of the National Early Warning Score 2 (NEWS2) as a risk stratification tool to discriminate newly admitted patients with COVID-19 at risk of serious events.

DESIGN

We conducted a retrospective single-centre case-control study on 200 unselected patients consecutively admitted in March 2020 in a public general hospital in Wuhan, China.

METHODS

The following serious events were considered: mortality, unplanned intensive care unit (ICU) admission, and non-invasive ventilation treatment. Receiver operating characteristic (ROC) analysis and logistic regression analysis were used to quantify the association between outcomes and NEWS2.

RESULTS

There were 12 patients (6.0%) who had serious events, where 7 patients (3.5%) experienced unplanned ICU admissions. The area under the ROC curve (AUROC) and cut-off of NEWS2 for the composite outcome were 0.83 and 3, respectively. For patients with NEWS2 ≥ 4, the odds of being at risk for serious events was 16.4 (AUROC = 0.74), while for patients with NEWS2 ≥ 7, the odds of being at risk for serious events was 18.2 (AUROC = 0.71).

CONCLUSIONS

NEWS2 has an appropriate ability to triage newly admitted patients with COVID-19 into three levels of risk: low risk (NEWS2 = 0-3), medium risk (NEWS2 = 4-6), and high risk (NEWS2 ≥ 7).

RELEVANCE TO CLINICAL PRACTICE

Using NEWS2 may help nurses in early identification of at-risk COVID-19 patients and clinical nursing decision-making. Using NEWS2 to triage new patients with COVID-19 may help nurses provide more appropriate level of care and medical resources allocation for patients safety.

Performance Analysis of the National Early Warning Score and Modified Early Warning Score in the Adaptive COVID-19 Treatment Trial Cohort

We sought to validate prognostic scores in coronavirus disease 2019 including National Early Warning Score, Modified Early Warning Score, and age-based modifications, and define their performance characteristics.

DESIGN

We analyzed prospectively collected data from the Adaptive COVID-19 Treatment Trial. National Early Warning Score was collected daily during the trial, Modified Early Warning Score was calculated, and age applied to both scores. We assessed prognostic value for the end points of recovery, mechanical ventilation, and death for score at enrollment, average, and slope of score over the first 48 hours.

SETTING

A multisite international inpatient trial.

PATIENTS

A total of 1,062 adult nonpregnant inpatients with severe coronavirus disease 2019 pneumonia.

INTERVENTIONS

Adaptive COVID-19 Treatment Trial 1 randomized participants to receive remdesivir or placebo. The prognostic value of predictive scores was evaluated in both groups separately to assess for differential performance in the setting of remdesivir treatment.

MEASUREMENTS AND MAIN RESULTS

For mortality, baseline National Early Warning Score and Modified Early Warning Score were weakly to moderately prognostic (c-index, 0.60-0.68), and improved with addition of age (c-index, 0.66-0.74). For recovery, baseline National Early Warning Score and Modified Early Warning Score demonstrated somewhat better prognostic ability (c-index, 0.65-0.69); however, National Early Warning Score+age and Modified Early Warning Score+age further improved performance (c-index, 0.68-0.71). For deterioration, baseline National Early Warning Score and Modified Early Warning Score were weakly to moderately prognostic (c-index, 0.59-0.69) and improved with addition of age (c-index, 0.63-0.70). All prognostic performance improvements due to addition of age were significant (p < 0.05).

CONCLUSIONS

In the Adaptive COVID-19 Treatment Trial 1 cohort, National Early Warning Score and Modified Early Warning Score demonstrated moderate prognostic performance in patients with severe coronavirus disease 2019, with improvement in predictive ability for National Early Warning Score+age and Modified Early Warning Score+age. Area under receiver operating curve for National Early Warning Score and Modified Early Warning Score improved in patients receiving remdesivir versus placebo early in the pandemic for recovery and mortality. Although these scores are simple and readily obtainable in myriad settings, in our data set, they were insufficiently predictive to completely replace clinical judgment in coronavirus disease 2019 and may serve best as an adjunct to triage, disposition, and resourcing decisions.

Development and validation of early warning score systems for COVID-19 patients

COVID‐19 is a major, urgent, and ongoing threat to global health. Globally more than 24 million have been infected and the disease has claimed more than a million lives as of November 2020. Predicting which patients will need respiratory support is important to guiding individual patient treatment and also to ensuring sufficient resources are available. The ability of six common Early Warning Scores (EWS) to identify respiratory deterioration defined as the need for advanced respiratory support (high‐flow nasal oxygen, continuous positive airways pressure, non‐invasive ventilation, intubation) within a prediction window of 24 h is evaluated. It is shown that these scores perform sub‐optimally at this specific task. Therefore, an alternative EWS based on the Gradient Boosting Trees (GBT) algorithm is developed that is able to predict deterioration within the next 24 h with high AUROC 94% and an accuracy, sensitivity, and specificity of 70%, 96%, 70%, respectively. The GBT model outperformed the best EWS (LDTEWS:NEWS), increasing the AUROC by 14%. Our GBT model makes the prediction based on the current and baseline measures of routinely available vital signs and blood tests.

The ROX index has greater predictive validity than NEWS2 for deterioration in Covid-19

BACKGROUND

Patients admitted to hospital with Covid-19 are at risk of deterioration. The National Early Warning Score (NEWS2) is widely recommended, however it's validity in Covid-19 is not established and indices more specific for respiratory failure may be more appropriate. We aim to describe the physiological antecedents to deterioration, test the predictive validity of NEWS2 and compare this to the ROX index ([SpO2/FiO2]/respiratory rate).

METHOD

A single centre retrospective cohort study of adult patients who were admitted to a medical ward, between 1/3/20 and 30/5/20, with positive results for SARS-CoV-2 RNA. Physiological observations and the NEWS2 were extracted and analysed. The primary outcome was a composite of cardiac arrest, unplanned critical care admission or death within 24 hours. A generalized linear model was used to assess the association of physiological values, NEWS2 and ROX with the outcome.

FINDINGS

The primary outcome occurred in 186 patients (26%). In the preceding 24 hours, deterioration was most marked in respiratory parameters, specifically in escalating oxygen requirement; tachypnoea was a late sign, whilst cardiovascular observations remained stable. The area under the receiver operating curve was 0.815 (95% CI 0.804-0.826) for NEWS2 and 0.848 (95% CI 0.837-0.858) for ROX. Applying the optimal level of ROX, the majority of patients triggered four hours earlier than with NEWS2 of 5.

INTERPRETATION

NEWS2 may under-perform in Covid-19 due to intrinsic limitations of the design and the unique pathophysiology of the disease. A simple index utilising respiratory parameters can outperform NEWS2 in predicting the occurrence of adverse events.

Dynamic individual vital sign trajectory early warning score (DyniEWS) versus snapshot national early warning score (NEWS) for predicting postoperative deterioration

Aims

International early warning scores (EWS) including the additive National Early Warning Score (NEWS) and logistic EWS currently utilise physiological snapshots to predict clinical deterioration. We hypothesised that a dynamic score including vital sign trajectory would improve discriminatory power.

Methods

Multicentre retrospective analysis of electronic health record data from postoperative patients admitted to cardiac surgical wards in four UK hospitals. Least absolute shrinkage and selection operator-type regression (LASSO) was used to develop a dynamic model (DyniEWS) to predict a composite adverse event of cardiac arrest, unplanned intensive care re-admission or in-hospital death within 24 h.

Results

A total of 13,319 postoperative adult cardiac patients contributed 442,461 observations of which 4234 (0.96%) adverse events in 24 h were recorded. The new dynamic model (AUC = 0.80 [95% CI 0.78−0.83], AUPRC = 0.12 [0.10−0.14]) outperforms both an updated snapshot logistic model (AUC = 0.76 [0.73−0.79], AUPRC = 0.08 [0.60−0.10]) and the additive National Early Warning Score (AUC = 0.73 [0.70−0.76], AUPRC = 0.05 [0.02−0.08]). Controlling for the false alarm rates to be at current levels using NEWS cut-offs of 5 and 7, DyniEWS delivers a 7% improvement in balanced accuracy and increased sensitivities from 41% to 54% at NEWS 5 and 18% to –30% at NEWS 7.

Conclusions

Using an advanced statistical approach, we created a model that can detect dynamic changes in risk of unplanned readmission to intensive care, cardiac arrest or in-hospital mortality and can be used in real time to risk-prioritise clinical workload.

Trajectories of vital signs in patients with COVID-19

Background

The global pandemic of coronavirus disease 2019 (COVID-19) has placed a huge strain on UK hospitals. Early studies suggest that patients can deteriorate quickly after admission to hospital. The aim of this study was to model changes in vital signs for patients hospitalised with COVID-19.

Methods

This was a retrospective observational study of adult patients with COVID-19 admitted to one acute hospital trust in the UK (CV) and a cohort of patients admitted to the same hospital between 2013-2017 with viral pneumonia (VI). The primary outcome was the start of continuous positive airway pressure/non-invasive positive pressure ventilation, ICU admission or death in hospital. We used non-linear mixed-effects models to compare changes in vital sign observations prior to the primary outcome. Using observations and FiO₂ measured at discharge in the VI cohort as the model of normality, we also combined individual vital signs into a single novelty score.

Results

There were 497 cases of COVID-19, of whom 373 had been discharged from hospital. 135 (36.2%) of patients experienced the primary outcome, of whom 99 died in hospital. In-hospital mortality was over 4-times higher in the CV than the VI cohort (26.5% vs 6%). For those patients who experienced the primary outcome, CV patients became increasingly hypoxaemic, with a median estimated FiO₂ (0.75) higher than that of the VI cohort (estimated FiO₂ of 0.35). Prior to the primary outcome, blood pressure remained within normal range, and there was only a small rise in heart rate. The novelty score showed that patients with COVID-19 deteriorated more rapidly that patients with viral pneumonia.

Conclusions

Patients with COVID-19 who deteriorate in hospital experience rapidly-worsening respiratory failure, with low SpO₂ and high FiO₂, but only minor abnormalities in other vital signs. This has potential implications for the ability of early warning scores to identify deteriorating patients.

Testing a digital system that ranks the risk of unplanned intensive care unit admission in all ward patients: protocol for a prospective observational cohort study

INTRODUCTION

Traditional early warning scores (EWSs) use vital sign derangements to detect clinical deterioration in patients treated on hospital wards. Combining vital signs with demographics and laboratory results improves EWS performance. We have developed the Hospital Alerting Via Electronic Noticeboard (HAVEN) system. HAVEN uses vital signs, as well as demographic, comorbidity and laboratory data from the electronic patient record, to quantify and rank the risk of unplanned admission to an intensive care unit (ICU) within 24 hours for all ward patients. The primary aim of this study is to find additional variables, potentially missed during development, which may improve HAVEN performance. These variables will be sought in the medical record of patients misclassified by the HAVEN risk score during testing.

METHODS

This will be a prospective, observational, cohort study conducted at the John Radcliffe Hospital, part of the Oxford University Hospitals NHS Foundation Trust in the UK. Each day during the study periods, we will document all highly ranked patients (ie, those with the highest risk for unplanned ICU admission) identified by the HAVEN system. After 48 hours, we will review the progress of the identified patients. Patients who were subsequently admitted to the ICU will be removed from the study (as they will have been correctly classified by HAVEN). Highly ranked patients not admitted to ICU will undergo a structured medical notes review. Additionally, at the end of the study periods, all patients who had an unplanned ICU admission but whom HAVEN failed to rank highly will have a structured medical notes review. The review will identify candidate variables, likely associated with unplanned ICU admission, not included in the HAVEN risk score.

ETHICS AND DISSEMINATION

Approval has been granted for gathering the data used in this study from the South Central Oxford C Research Ethics Committee (16/SC/0264, 13 June 2016) and the Confidentiality Advisory Group (16/CAG/0066).

DISCUSSION

Our study will use a clinical expert conducting a structured medical notes review to identify variables, associated with unplanned ICU admission, not included in the development of the HAVEN risk score. These variables will then be added to the risk score and evaluated for potential performance gain. To the best of our knowledge, this is the first study of this type. We anticipate that documenting the HAVEN development methods will assist other research groups developing similar technology.

TRIAL REGISTRATION NUMBER

ISRCTN12518261.