Manual centile-based early warning scores derived from statistical distributions of observational vital-sign data

Development of a national maternity early warning score: centile based score development and Delphi informed escalation pathways

Objective

To derive a new maternity early warning score (MEWS) from prospectively collected data on maternity vital signs and to design clinical response pathways with a Delphi consensus exercise.

Design

Centile based score development and Delphi informed escalation pathways.

Setting

Pregnancy Physiology Pattern Prediction (4P) prospective UK cohort study, 1 August 2012 to 28 December 2016.

Participants

Pregnant people from the 4P study, recruited before 20 weeks' gestation at three UK maternity centres (Oxford, Newcastle, and London). 841, 998, and 889 women provided data in the early antenatal, antenatal, and postnatal periods.

Main outcome measures

Development of a new national MEWS, assigning numerical weights to measurements in the lower and upper extremes of distributions of individual vital signs from the 4P prospective cohort study. Comparison of escalation rates of the new national MEWS with the Scottish and Irish MEWS systems from 18 to 40 weeks' gestation. Delphi consensus exercise to agree clinical responses to raised scores.

Results

A new national MEWS was developed by assigning numerical weights to measurements in the lower and upper extremes (5%, 1%) of distributions of vital signs, except for oxygen saturation where lower centiles (10%, 2%) were used. For the new national MEWS, in a healthy population, 56% of observation sets resulted in a total score of 0 points, 26% a score of 1 point, 12% a score of 2 points, and 18% a score of ≥2 points (escalation of care is triggered at a total score of ≥2 points). Corresponding values for the Irish MEWS were 37%, 25%, 22%, and 38%, respectively; and for the Scottish MEWS, 50%, 18%, 21%, and 32%, respectively. All three MEWS were similar at the beginning of pregnancy, averaging 0.7-0.9 points. The new national MEWS had a lower mean score for the rest of pregnancy, with the mean score broadly constant (0.6-0.8 points). The new national MEWS had an even distribution of healthy population alerts across the antenatal period. In the postnatal period, heart rate threshold values were adjusted to align with postnatal changes. The centile based score derivation approach meant that each vital sign component in the new national MEWS had a similar alert rate. Suggested clinical responses to different MEWS values were agreed by consensus of an independent expert panel.

Conclusions

The centile based MEWS alerted escalation of care evenly across the antenatal period in a healthy population, while reducing alerts in healthy women compared with other MEWS systems. How well the tool predicted adverse outcomes, however, was not assessed and therefore external validation studies in large datasets are needed. Unlike other MEWS systems, the new national MEWS was developed with prospectively collected data on vital signs and used a systematic, expert informed process to design an associated escalation protocol.

Development of an enhanced scoring system to predict ICU readmission or in-hospital death within 24 hours using routine patient data from two NHS Foundation Trusts

RATIONALE

Intensive care units (ICUs) admit the most severely ill patients. Once these patients are discharged from the ICU to a step-down ward, they continue to have their vital signs monitored by nursing staff, with Early Warning Score (EWS) systems being used to identify those at risk of deterioration.

OBJECTIVES

We report the development and validation of an enhanced continuous scoring system for predicting adverse events, which combines vital signs measured routinely on acute care wards (as used by most EWS systems) with a risk score of a future adverse event calculated on discharge from the ICU.

DESIGN

A modified Delphi process identified candidate variables commonly available in electronic records as the basis for a 'static' score of the patient's condition immediately after discharge from the ICU. L1-regularised logistic regression was used to estimate the in-hospital risk of future adverse event. We then constructed a model of physiological normality using vital sign data from the day of hospital discharge. This is combined with the static score and used continuously to quantify and update the patient's risk of deterioration throughout their hospital stay.

SETTING

Data from two National Health Service Foundation Trusts (UK) were used to develop and (externally) validate the model.

PARTICIPANTS

A total of 12 394 vital sign measurements were acquired from 273 patients after ICU discharge for the development set, and 4831 from 136 patients in the validation cohort.

RESULTS

Outcome validation of our model yielded an area under the receiver operating characteristic curve of 0.724 for predicting ICU readmission or in-hospital death within 24 hours. It showed an improved performance with respect to other competitive risk scoring systems, including the National EWS (0.653).

CONCLUSIONS

We showed that a scoring system incorporating data from a patient's stay in the ICU has better performance than commonly used EWS systems based on vital signs alone.

TRIAL REGISTRATION NUMBER

ISRCTN32008295.

Correlation of vital sign centiles with in-hospital outcomes among adults encountered by emergency medical services

BACKGROUND

Vital signs are a critical component of the prehospital assessment. Prior work has suggested that vital signs may vary in their distribution by age. These differences in vital signs may have implications on in-hospital outcomes or be utilized within prediction models. We sought to (1) identify empirically derived (unadjusted) cut points for vital signs for adult patients encountered by emergency medical services (EMS), (2) evaluate differences in age-adjusted cutoffs for vital signs in this population, and (3) evaluate unadjusted and age-adjusted vital signs measures with in-hospital outcomes.

METHODS

We used two multiagency EMS data sets to derive (National EMS Information System from 2018) and assess agreement (ESO, Inc., from 2019 to 2021) of vital signs cutoffs among adult EMS encounters. We compared unadjusted to age-adjusted cutoffs. For encounters within the ESO sample that had in-hospital data, we compared the association of unadjusted cutoffs and age-adjusted cutoffs with hospitalization and in-hospital mortality.

RESULTS

We included 13,405,858 and 18,682,684 encounters in the derivation and validation samples, respectively. Both extremely high and extremely low vital signs demonstrated stepwise increases in admission and in-hospital mortality. When evaluating age-based centiles with vital signs, a gradual decline was noted at all extremes of heart rate (HR) with increasing age. Extremes of systolic blood pressure at upper and lower margins were greater in older age groups relative to younger age groups. Respiratory rate (RR) cut points were similar for all adult age groups. Compared to unadjusted vital signs, age-adjusted vital signs had slightly increased accuracy for HR and RR but lower accuracy for SBP for outcomes of mortality and hospitalization.

CONCLUSIONS

We describe cut points for vital signs for adults in the out-of-hospital setting that are associated with both mortality and hospitalization. While we found age-based differences in vital signs cutoffs, this adjustment only slightly improved model performance for in-hospital outcomes.

Safer and more efficient vital signs monitoring protocols to identify the deteriorating patients in the general hospital ward: an observational study

Background

The frequency at which patients should have their vital signs (e.g. blood pressure, pulse, oxygen saturation) measured on hospital wards is currently unknown. Current National Health Service monitoring protocols are based on expert opinion but supported by little empirical evidence. The challenge is finding the balance between insufficient monitoring (risking missing early signs of deterioration and delays in treatment) and over-observation of stable patients (wasting resources needed in other aspects of care).

Objective

Provide an evidence-based approach to creating monitoring protocols based on a patient's risk of deterioration and link these to nursing workload and economic impact.

Design

Our study consisted of two parts: (1) an observational study of nursing staff to ascertain the time to perform vital sign observations; and (2) a retrospective study of historic data on patient admissions exploring the relationships between National Early Warning Score and risk of outcome over time. These were underpinned by opinions and experiences from stakeholders.

Setting and participants

Observational study: observed nursing staff on 16 randomly selected adult general wards at four acute National Health Service hospitals. Retrospective study: extracted, linked and analysed routinely collected data from two large National Health Service acute trusts; data from over 400,000 patient admissions and 9,000,000 vital sign observations.

Results

Observational study found a variety of practices, with two hospitals having registered nurses take the majority of vital sign observations and two favouring healthcare assistants or student nurses. However, whoever took the observations spent roughly the same length of time. The average was 5:01 minutes per observation over a 'round', including time to locate and prepare the equipment and travel to the patient area. Retrospective study created survival models predicting the risk of outcomes over time since the patient was last observed. For low-risk patients, there was little difference in risk between 4 hours and 24 hours post observation.

Conclusions

We explored several different scenarios with our stakeholders (clinicians and patients), based on how 'risk' could be managed in different ways. Vital sign observations are often done more frequently than necessary from a bald assessment of the patient's risk, and we show that a maximum threshold of risk could theoretically be achieved with less resource. Existing resources could therefore be redeployed within a changed protocol to achieve better outcomes for some patients without compromising the safety of the rest. Our work supports the approach of the current monitoring protocol, whereby patients' National Early Warning Score 2 guides observation frequency. Existing practice is to observe higher-risk patients more frequently and our findings have shown that this is objectively justified. It is worth noting that important nurse-patient interactions take place during vital sign monitoring and should not be eliminated under new monitoring processes. Our study contributes to the existing evidence on how vital sign observations should be scheduled. However, ultimately, it is for the relevant professionals to decide how our work should be used.

Study registration

This study is registered as ISRCTN10863045.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: 17/05/03) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 6. See the NIHR Funding and Awards website for further award information.

Vital signs and medical emergency response (MER) activation predict in-hospital mortality in general surgery patients: a study of 15 969 admissions

BACKGROUND

The applicability of the vital signs prompting medical emergency response (MER) activation has not previously been examined specifically in a large general surgical cohort. This study aimed to characterize the distribution, and predictive performance, of four vital signs selected based on Australian guidelines (oxygen saturation, respiratory rate, systolic blood pressure and heart rate); with those of the MER activation criteria.

METHODS

A retrospective cohort study was conducted including patients admitted under general surgical services of two hospitals in South Australia over 2 years. Likelihood ratios for patients meeting MER activation criteria, or a vital sign in the most extreme 1% for general surgery inpatients (<0.5th percentile or > 99.5th percentile), were calculated to predict in-hospital mortality.

RESULTS

15 969 inpatient admissions were included comprising 2 254 617 total vital sign observations. The 0.5th and 99.5th centile for heart rate was 48 and 133, systolic blood pressure 85 and 184, respiratory rate 10 and 31, and oxygen saturations 89% and 100%, respectively. MER activation criteria with the highest positive likelihood ratio for in-hospital mortality were heart rate ≤ 39 (37.65, 95% CI 27.71-49.51), respiratory rate ≥ 31 (15.79, 95% CI 12.82-19.07), and respiratory rate ≤ 7 (10.53, 95% CI 6.79-14.84). These MER activation criteria likelihood ratios were similar to those derived when applying a threshold of the most extreme 1% of vital signs.

CONCLUSIONS

This study demonstrated that vital signs within Australian guidelines, and escalation to MER activation, appropriately predict in-hospital mortality in a large cohort of patients admitted to general surgical services in South Australia.

Development and External Validation of the International Early Warning Score for Improved Age- and Sex-Adjusted In-Hospital Mortality Prediction in the Emergency Department

OBJECTIVES

Early Warning Scores (EWSs) have a great potential to assist clinical decision-making in the emergency department (ED). However, many EWS contain methodological weaknesses in development and validation and have poor predictive performance in older patients. The aim of this study was to develop and externally validate an International Early Warning Score (IEWS) based on a recalibrated National Early warning Score (NEWS) model including age and sex and evaluate its performance independently at arrival to the ED in three age categories (18-65, 66-80, > 80 yr).

DESIGN

International multicenter cohort study.

SETTING

Data was used from three Dutch EDs. External validation was performed in two EDs in Denmark.

PATIENTS

All consecutive ED patients greater than or equal to 18 years in the Netherlands Emergency department Evaluation Database (NEED) with at least two registered vital signs were included, resulting in 95,553 patients. For external validation, 14,809 patients were included from a Danish Multicenter Cohort (DMC).

MEASUREMENTS AND MAIN RESULTS

Model performance to predict in-hospital mortality was evaluated by discrimination, calibration curves and summary statistics, reclassification, and clinical usefulness by decision curve analysis. In-hospital mortality rate was 2.4% ( n = 2,314) in the NEED and 2.5% ( n = 365) in the DMC. Overall, the IEWS performed significantly better than NEWS with an area under the receiving operating characteristic of 0.89 (95% CIs, 0.89-0.90) versus 0.82 (0.82-0.83) in the NEED and 0.87 (0.85-0.88) versus 0.82 (0.80-0.84) at external validation. Calibration for NEWS predictions underestimated risk in older patients and overestimated risk in the youngest, while calibration improved for IEWS with a substantial reclassification of patients from low to high risk and a standardized net benefit of 5-15% in the relevant risk range for all age categories.

CONCLUSIONS

The IEWS substantially improves in-hospital mortality prediction for all ED patients greater than or equal to18 years.

Development of a dynamic prediction model for unplanned ICU admission and mortality in hospitalized patients

Frequent assessment of the severity of illness for hospitalized patients is essential in clinical settings to prevent outcomes such as in-hospital mortality and unplanned admission to the intensive care unit (ICU). Classical severity scores have been developed typically using relatively few patient features. Recently, deep learning-based models demonstrated better individualized risk assessments compared to classic risk scores, thanks to the use of aggregated and more heterogeneous data sources for dynamic risk prediction. We investigated to what extent deep learning methods can capture patterns of longitudinal change in health status using time-stamped data from electronic health records. We developed a deep learning model based on embedded text from multiple data sources and recurrent neural networks to predict the risk of the composite outcome of unplanned ICU transfer and in-hospital death. The risk was assessed at regular intervals during the admission for different prediction windows. Input data included medical history, biochemical measurements, and clinical notes from a total of 852,620 patients admitted to non-intensive care units in 12 hospitals in Denmark's Capital Region and Region Zealand during 2011-2016 (with a total of 2,241,849 admissions). We subsequently explained the model using the Shapley algorithm, which provides the contribution of each feature to the model outcome. The best model used all data modalities with an assessment rate of 6 hours, a prediction window of 14 days and an area under the receiver operating characteristic curve of 0.898. The discrimination and calibration obtained with this model make it a viable clinical support tool to detect patients at higher risk of clinical deterioration, providing clinicians insights into both actionable and non-actionable patient features.

Development and validation of nomogram for unplanned ICU admission in patients with dilated cardiomyopathy

Objective

Unplanned admission to the intensive care unit (ICU) is the major in-hospital adverse event for patients with dilated cardiomyopathy (DCM). We aimed to establish a nomogram of individualized risk prediction for unplanned ICU admission in DCM patients.

Methods

A total of 2,214 patients diagnosed with DCM from the First Affiliated Hospital of Xinjiang Medical University from January 01, 2010, to December 31, 2020, were retrospectively analyzed. Patients were randomly divided into training and validation groups at a 7:3 ratio. The least absolute shrinkage and selection operator and multivariable logistic regression analysis were used for nomogram model development. The area under the receiver operating characteristic curve, calibration curves, and decision curve analysis (DCA) were used to evaluate the model. The primary outcome was defined as unplanned ICU admission.

Results

A total of 209 (9.44%) patients experienced unplanned ICU admission. The variables in our final nomogram included emergency admission, previous stroke, New York Heart Association Class, heart rate, neutrophil count, and levels of N-terminal pro b-type natriuretic peptide. In the training group, the nomogram showed good calibration (Hosmer-Lemeshow χ 2 = 14.40, P = 0.07) and good discrimination, with an optimal-corrected C-index of 0.76 (95% confidence interval: 0.72-0.80). DCA confirmed the clinical net benefit of the nomogram model, and the nomogram maintained excellent performances in the validation group.

Conclusion

This is the first risk prediction model for predicting unplanned ICU admission in patients with DCM by simply collecting clinical information. This model may assist physicians in identifying individuals at a high risk of unplanned ICU admission for DCM inpatients.

Tracking the National Early Warning Score 2 from Prehospital Care to the Emergency Department: A Prospective, Ambulance-Based, Observational Study

Aim of the study: To assess the prognostic ability of the National Early Warning Score 2 (NEWS2) at three time points of care -at the emergency scene (NEWS2-1), just before starting the transfer by ambulance to the hospital (NEWS2- 2), and at the hospital triage box (NEWS2-3)- to estimate in-hospital mortality after two days since the index event.Methods: Prospective, multicenter, ambulance-based, cohort ongoing study in adults (>18 years) consecutively attended by advanced life support (ALS) and evacuated with high-priority to the emergency departments (ED) between October 2018 and May 2021. Vital sign measures were used to calculate the NEWS2 score at each time point, then this score was entered in a logistic regression model as the single predictor. Two outcomes were considered: first, all-cause mortality of the patients within 2 days of presentation to EMS, and second, unplanned ICU admission. The calibration and scores comparison was performed by representing the predicted vs the observed risk curves according to NEWS score value.Results: 4943 patients were enrolled. Median age was 69 years (interquartile range 53- 81). The NEWS2-3 presented the better performance for all-cause two-day in-hospital mortality with an AUC of 0.941 (95% CI: 0.917-0.964), showing statistical differences with both the NEWS2-1 (0.872 (95% CI: 0.833-0.911); p < 0.003) and with the NEWS2- 2 (0.895 (95% CI: 0.866-0.925; p < 0.05). The calibration and scores comparison results showed that the NEWS2-3 was the best predictive score followed by the NEWS2-2 and the NEWS2-1, respectively.Conclusions: The NEWS2 has an excellent predictive performance. The score showed a very consistent response over time with the difference between "at the emergency scene" and "pre-evacuation" presenting the sharpest change with decreased threshold values, thus displaying a drop in the risk of acute clinical impairment.

Financial Risk Intelligent Early Warning System of a Municipal Company Based on Genetic Tabu Algorithm and Big Data Analysis

This paper proposes an intelligent early warning system for financial risk of listed companies based on genetic tabu algorithm and big data analysis. Establish the role application, analyze and complete the financial risk intelligent early warning system of listed companies and query the analysis results. Through the early warning source data management module, financial early warning module and countermeasure management module, the hardware part of the system is designed, the functional modules are divided, the relationship between the internal functions of the system is clear, and the hierarchical structure and call relationship between the modules are established. The key technology of CBR retrieve the matching case. Genetic tabu algorithm establish fitness function, search fitness measure, optimize attribute weight, and design system software. The experimental results show that the financial risk intelligent early warning system has high early warning accuracy, can effectively shorten the early warning time, and meet the needs of financial risk management of listed companies.

Patient Deterioration on General Care Units: A Concept Analysis

Patient deterioration is a phenomenon that occurs from the inability to recognize it or respond to a change in condition. Despite the published reports on recognizing a deteriorating patient on general care floors, a gap remains in the ability of nurses to describe the concept, affecting patient outcomes. Walker and Avant's approach was applied to analyze patient deterioration. The aim of this article was to explore and clarify the meaning of patient deterioration and identify attributes, antecedents, and consequences. The defining attributes were compared to early warning scores. An operational definition was developed and its value to nurses established.

Emergency Department Length of Stay and Mortality in Critically Injured Patients

OBJECTIVE

Multicenter data from 2 decades ago demonstrated that critically ill and injured patients spending more than 6 hours in the emergency department (ED) before transfer to the intensive care unit (ICU) had higher mortality rates. A contemporary analysis of ED length of stay in critically injured patients at American College of Surgeons' Trauma Quality Improvement Program (ACS-TQIP) centers was performed to test whether prolonged ED length of stay is still associated with mortality.

METHODS

This was an observational cohort study of critically injured patients admitted directly to ICU from the ED in ACS-TQIP centers from 2010-2015. Spending more than 6 hours in the ED was defined as prolonged ED length of stay. Patients with prolonged ED length of stay were matched to those with non-prolonged ED length of stay and mortality was compared.

MAIN RESULTS

A total of 113,097 patients were directly admitted from the ED to the ICU following injury. The median ED length of stay was 167 minutes. Prolonged ED length of stay occurred in 15,279 (13.5%) of patients. Women accounted for 29.4% of patients with prolonged ED length of stay but only 25.8% of patients with non-prolonged ED length of stay, P < 0.0001. Mortality rates were similar after matching-4.5% among patients with prolonged ED length of stay versus 4.2% among matched controls. Multivariable logistic regression of the matched cohorts demonstrated prolonged ED length of stay was not associated with mortality. However, women had higher adjusted mortality compared to men Odds Ratio = 1.41, 95% Confidence Interval 1.28 -1.61, P < 0.0001.

CONCLUSION

Prolonged ED length of stay is no longer associated with mortality among critically injured patients. Women are more likely to have prolonged ED length of stay and mortality.

Association of the All-Patient Refined Diagnosis-Related Groups Severity of Illness and Risk of Mortality Classification with Outcomes

OBJECTIVES

Diagnosis-related groups (DRGs) is a patient classification system used to characterize the types of patients that the hospital manages and to compare the resources needed during hospitalization. The DRG classification is based on International Classification of Diseases diagnoses, procedures, demographics, discharge status, and complications or comorbidities and compares hospital resources and outcomes used to determine how much Medicare pays the hospital for each "product/medical condition." The All-Patient Refined DRG (APR-DRG) incorporated severity of illness (SOI) and risk of mortality (ROM) into the DRG system to adjust for patient complexity to compare resource utilization, complication rates, and lengths of stay.

METHODS

This study included 18,478 adult patients admitted to a tertiary care center in Lubbock, Texas during a 1-year period. We recorded the APR-DRG SOI and ROM and some clinical information on these patients, including age, sex, admission shock index, admission glucose and lactate levels, diagnoses based on International Classification of Diseases, Tenth Revision discharge coding, length of stay, and mortality. We compared the levels of SOI and ROM across this clinical information.

RESULTS

As the levels of SOI and ROM increase (which indicates increased disease severity and risk of mortality), age, glucose levels, lactate levels, shock index, length of stay, and mortality increased significantly (P < 0.001). Multiple logistic regression analysis demonstrated that each unit increase in ROM and SOI level was significantly associated with an 11.45 and a 10.37 times increase in the odds of in-hospital mortality, respectively. The C-statistics for the corresponding models are 0.947 and 0.929, respectively. When both ROM and SOI were included in the model, the magnitudes of increase in odds of in-hospital mortality were 5.61 and 1.17 times for ROM and SOI, respectively. The C-statistic is 0.949.

CONCLUSIONS

This study indicates that the APR-DRG SOI and ROM scores provide a classification system that is associated with mortality and correlates with other clinical variables, such as the shock index and lactate levels, which are available on admission.

Detecting Deteriorating Patients in the Hospital: Development and Validation of a Novel Scoring System

Rationale: Late recognition of patient deterioration in hospital is associated with worse outcomes, including higher mortality. Despite the widespread introduction of early warning score (EWS) systems and electronic health records, deterioration still goes unrecognized. Objectives: To develop and externally validate a Hospital- wide Alerting via Electronic Noticeboard (HAVEN) system to identify hospitalized patients at risk of reversible deterioration. Methods: This was a retrospective cohort study of patients 16 years of age or above admitted to four UK hospitals. The primary outcome was cardiac arrest or unplanned admission to the ICU. We used patient data (vital signs, laboratory tests, comorbidities, and frailty) from one hospital to train a machine-learning model (gradient boosting trees). We internally and externally validated the model and compared its performance with existing scoring systems (including the National EWS, laboratory-based acute physiology score, and electronic cardiac arrest risk triage score). Measurements and Main Results: We developed the HAVEN model using 230,415 patient admissions to a single hospital. We validated HAVEN on 266,295 admissions to four hospitals. HAVEN showed substantially higher discrimination (c-statistic, 0.901 [95% confidence interval, 0.898-0.903]) for the primary outcome within 24 hours of each measurement than other published scoring systems (which range from 0.700 [0.696-0.704] to 0.863 [0.860-0.865]). With a precision of 10%, HAVEN was able to identify 42% of cardiac arrests or unplanned ICU admissions with a lead time of up to 48 hours in advance, compared with 22% by the next best system. Conclusions: The HAVEN machine-learning algorithm for early identification of in-hospital deterioration significantly outperforms other published scores such as the National EWS.

Development and validation of early warning score systems for COVID-19 patients

COVID‐19 is a major, urgent, and ongoing threat to global health. Globally more than 24 million have been infected and the disease has claimed more than a million lives as of November 2020. Predicting which patients will need respiratory support is important to guiding individual patient treatment and also to ensuring sufficient resources are available. The ability of six common Early Warning Scores (EWS) to identify respiratory deterioration defined as the need for advanced respiratory support (high‐flow nasal oxygen, continuous positive airways pressure, non‐invasive ventilation, intubation) within a prediction window of 24 h is evaluated. It is shown that these scores perform sub‐optimally at this specific task. Therefore, an alternative EWS based on the Gradient Boosting Trees (GBT) algorithm is developed that is able to predict deterioration within the next 24 h with high AUROC 94% and an accuracy, sensitivity, and specificity of 70%, 96%, 70%, respectively. The GBT model outperformed the best EWS (LDTEWS:NEWS), increasing the AUROC by 14%. Our GBT model makes the prediction based on the current and baseline measures of routinely available vital signs and blood tests.

A simple index predicting mortality in acutely hospitalized patients

BACKGROUND

Mortality rates used to evaluate and improve the quality of hospital care are adjusted for comorbidity and disease severity. Comorbidity, measured by International Classification of Diseases codes, do not reflect the severity of the medical condition, that requires clinical assessments not available in electronic databases, and/or laboratory data with clinically relevant ranges to permit extrapolation from one setting to the next.

AIM

To propose a simple index predicting mortality in acutely hospitalized patients.

DESIGN

Retrospective cohort study with internal and external validation.

METHODS

The study populations were all acutely admitted patients in 2015-16, and in January 2019-November 2019 to internal medicine, cardiology and intensive care departments at the Laniado Hospital in Israel, and in 2002-19, at St. James Hospital, Ireland. Predictor variables were age and admission laboratory tests. The outcome variable was in-hospital mortality. Using logistic regression of the data in the 2015-16 Israeli cohort, we derived an index that included age groups and significant laboratory data.

RESULTS

In the Israeli 2015-16 cohort, the index predicted mortality rates from 0.2% to 32.0% with a c-statistic (area under the receiver operator characteristic curve) of 0.86. In the Israeli 2019 validation cohort, the index predicted mortality rates from 0.3% to 38.9% with a c-statistic of 0.87. An abbreviated index performed similarly in the Irish 2002-19 cohort.

CONCLUSIONS

Hospital mortality can be predicted by age and selected admission laboratory data without acquiring information from the patient's medical records. This permits an inexpensive comparison of performance of hospital departments.

Postpartum-Specific Vital Sign Reference Ranges

OBJECTIVE

To estimate normal ranges for postpartum maternal vital signs.

METHODS

We conducted a multicenter prospective longitudinal cohort study in the United Kingdom. We recruited women before 20 weeks of gestation without significant comorbidities and with accurately dated singleton pregnancies. Women recorded their own blood pressure, heart rate, oxygen saturation and temperature daily for 2 weeks postpartum. Trained midwives measured participants' vital signs including respiratory rate around postpartum days 1, 7, and 14.

RESULTS

From August 2012 to September 2016, we screened 4,279 pregnant women; 1,054 met eligibility criteria and chose to take part. Postpartum vital sign data were available for 909 women (86.2%). Median, or 50th centile (3rd-97th centile), systolic and diastolic blood pressures increased from the day of birth: 116 mm Hg (88-147) and 74 mm Hg (59-93) to a maximum median of 121 mm Hg (102-143) and 79 mm Hg (63-94) on days 5 and 6 postpartum, respectively, an increase of 5 mm Hg (95% CI 3-7) and 5 mm Hg (95% CI 4-6), respectively. Median (3rd-97th centile) systolic and diastolic blood pressure returned to 116 mm Hg (98-137) and 75 mm Hg (61-91) by day 14 postpartum. The median (3rd-97th centile) heart rate was highest on the day of birth, 84 beats per minute (bpm) (59-110) decreasing to a minimum of 75 bpm (55-101) 14 days postpartum. Oxygen saturation, respiratory rate, and temperature did not change in the 2 weeks postbirth. Median (3rd-97th centile) day-of-birth oxygen saturation was 96% (93-98). Median (3rd-97th centile) day-of-birth respiratory rate was 15 breaths per minute (10-22). Median (3rd-97th centile) day-of-birth temperature was 36.7°C (35.6-37.6).

CONCLUSION

We present widely relevant, postpartum, day-specific reference ranges which may facilitate early detection of abnormal blood pressure, heart rate, respiratory rate, oxygen saturation and temperature during the puerperium. Our findings could inform construction of an evidence-based modified obstetric early warning system to better identify unwell postpartum women.

CLINICAL TRIAL REGISTRATION

ISRCTN, 10838017.

Machine Learning for Clinical Outcome Prediction

Clinical decision-making in healthcare is already being influenced by predictions or recommendations made by data-driven machines. Numerous machine learning applications have appeared in the latest clinical literature, especially for outcome prediction models, with outcomes ranging from mortality and cardiac arrest to acute kidney injury and arrhythmia. In this review article, we summarize the state-of-the-art in related works covering data processing, inference, and model evaluation, in the context of outcome prediction models developed using data extracted from electronic health records. We also discuss limitations of prominent modeling assumptions and highlight opportunities for future research.

Dynamic individual vital sign trajectory early warning score (DyniEWS) versus snapshot national early warning score (NEWS) for predicting postoperative deterioration

Aims

International early warning scores (EWS) including the additive National Early Warning Score (NEWS) and logistic EWS currently utilise physiological snapshots to predict clinical deterioration. We hypothesised that a dynamic score including vital sign trajectory would improve discriminatory power.

Methods

Multicentre retrospective analysis of electronic health record data from postoperative patients admitted to cardiac surgical wards in four UK hospitals. Least absolute shrinkage and selection operator-type regression (LASSO) was used to develop a dynamic model (DyniEWS) to predict a composite adverse event of cardiac arrest, unplanned intensive care re-admission or in-hospital death within 24 h.

Results

A total of 13,319 postoperative adult cardiac patients contributed 442,461 observations of which 4234 (0.96%) adverse events in 24 h were recorded. The new dynamic model (AUC = 0.80 [95% CI 0.78−0.83], AUPRC = 0.12 [0.10−0.14]) outperforms both an updated snapshot logistic model (AUC = 0.76 [0.73−0.79], AUPRC = 0.08 [0.60−0.10]) and the additive National Early Warning Score (AUC = 0.73 [0.70−0.76], AUPRC = 0.05 [0.02−0.08]). Controlling for the false alarm rates to be at current levels using NEWS cut-offs of 5 and 7, DyniEWS delivers a 7% improvement in balanced accuracy and increased sensitivities from 41% to 54% at NEWS 5 and 18% to –30% at NEWS 7.

Conclusions

Using an advanced statistical approach, we created a model that can detect dynamic changes in risk of unplanned readmission to intensive care, cardiac arrest or in-hospital mortality and can be used in real time to risk-prioritise clinical workload.

Early warning score validation methodologies and performance metrics: a systematic review

BACKGROUND

Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation.

METHODS

A systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity.

RESULTS

The key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies.

CONCLUSIONS

Methodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.

Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology

OBJECTIVE

To provide an overview and critical appraisal of early warning scores for adult hospital patients.

DESIGN

Systematic review.

DATA SOURCES

Medline, CINAHL, PsycInfo, and Embase until June 2019.

ELIGIBILITY CRITERIA FOR STUDY SELECTION

Studies describing the development or external validation of an early warning score for adult hospital inpatients.

RESULTS

13 171 references were screened and 95 articles were included in the review. 11 studies were development only, 23 were development and external validation, and 61 were external validation only. Most early warning scores were developed for use in the United States (n=13/34, 38%) and the United Kingdom (n=10/34, 29%). Death was the most frequent prediction outcome for development studies (n=10/23, 44%) and validation studies (n=66/84, 79%), with different time horizons (the most frequent was 24 hours). The most common predictors were respiratory rate (n=30/34, 88%), heart rate (n=28/34, 83%), oxygen saturation, temperature, and systolic blood pressure (all n=24/34, 71%). Age (n=13/34, 38%) and sex (n=3/34, 9%) were less frequently included. Key details of the analysis populations were often not reported in development studies (n=12/29, 41%) or validation studies (n=33/84, 39%). Small sample sizes and insufficient numbers of event patients were common in model development and external validation studies. Missing data were often discarded, with just one study using multiple imputation. Only nine of the early warning scores that were developed were presented in sufficient detail to allow individualised risk prediction. Internal validation was carried out in 19 studies, but recommended approaches such as bootstrapping or cross validation were rarely used (n=4/19, 22%). Model performance was frequently assessed using discrimination (development n=18/22, 82%; validation n=69/84, 82%), while calibration was seldom assessed (validation n=13/84, 15%). All included studies were rated at high risk of bias.

CONCLUSIONS

Early warning scores are widely used prediction models that are often mandated in daily clinical practice to identify early clinical deterioration in hospital patients. However, many early warning scores in clinical use were found to have methodological weaknesses. Early warning scores might not perform as well as expected and therefore they could have a detrimental effect on patient care. Future work should focus on following recommended approaches for developing and evaluating early warning scores, and investigating the impact and safety of using these scores in clinical practice.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42017053324.

Development and validation of early warning score system: A systematic literature review

OBJECTIVES

This review aims to: 1) evaluate the quality of model reporting, 2) provide an overview of methodology for developing and validating Early Warning Score Systems (EWSs) for adult patients in acute care settings, and 3) highlight the strengths and limitations of the methodologies, as well as identify future directions for EWS derivation and validation studies.

METHODOLOGY

A systematic search was conducted in PubMed, Cochrane Library, and CINAHL. Only peer reviewed articles and clinical guidelines regarding developing and validating EWSs for adult patients in acute care settings were included. 615 articles were extracted and reviewed by five of the authors. Selected studies were evaluated based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist. The studies were analyzed according to their study design, predictor selection, outcome measurement, methodology of modeling, and validation strategy.

RESULTS

A total of 29 articles were included in the final analysis. Twenty-six articles reported on the development and validation of a new EWS, while three reported on validation and model modification. Only eight studies met more than 75% of the items in the TRIPOD checklist. Three major techniques were utilized among the studies to inform their predictive algorithms: 1) clinical-consensus models (n = 6), 2) regression models (n = 15), and 3) tree models (n = 5). The number of predictors included in the EWSs varied from 3 to 72 with a median of seven. Twenty-eight models included vital signs, while 11 included lab data. Pulse oximetry, mental status, and other variables extracted from electronic health records (EHRs) were among other frequently used predictors. In-hospital mortality, unplanned transfer to the intensive care unit (ICU), and cardiac arrest were commonly used clinical outcomes. Twenty-eight studies conducted a form of model validation either within the study or against other widely-used EWSs. Only three studies validated their model using an external database separate from the derived database.

CONCLUSION

This literature review demonstrates that the characteristics of the cohort, predictors, and outcome selection, as well as the metrics for model validation, vary greatly across EWS studies. There is no consensus on the optimal strategy for developing such algorithms since data-driven models with acceptable predictive accuracy are often site-specific. A standardized checklist for clinical prediction model reporting exists, but few studies have included reporting aligned with it in their publications. Data-driven models are subjected to biases in the use of EHR data, thus it is particularly important to provide detailed study protocols and acknowledge, leverage, or reduce potential biases of the data used for EWS development to improve transparency and generalizability.

Logistic early warning scores to predict death, cardiac arrest or unplanned intensive care unit re-admission after cardiac surgery

NHS England recently mandated that the National Early Warning Score of vital signs be used in all acute hospital trusts in the UK despite limited validation in the postoperative setting. We undertook a multicentre UK study of 13,631 patients discharged from intensive care after risk-stratified cardiac surgery in four centres, all of which used VitalPACTM to electronically collect postoperative National Early Warning Score vital signs. We analysed 540,127 sets of vital signs to generate a logistic score, the discrimination of which we compared with the national additive score for the composite outcome of: in-hospital death; cardiac arrest; or unplanned intensive care admission. There were 578 patients (4.2%) with an outcome that followed 4300 sets of observations (0.8%) in the preceding 24 h: 499 out of 578 (86%) patients had unplanned re-admissions to intensive care. Discrimination by the logistic score was significantly better than the additive score. Respective areas (95%CI) under the receiver-operating characteristic curve with 24-h and 6-h vital signs were: 0.779 (0.771-0.786) vs. 0.754 (0.746-0.761), p < 0.001; and 0.841 (0.829-0.853) vs. 0.813 (0.800-0.825), p < 0.001, respectively. Our proposed logistic Early Warning Score was better than the current National Early Warning Score at discriminating patients who had an event after cardiac surgery from those who did not.

Early warning score adjusted for age to predict the composite outcome of mortality, cardiac arrest or unplanned intensive care unit admission using observational vital-sign data: a multicentre development and validation

OBJECTIVES

Early warning scores (EWS) alerting for in-hospital deterioration are commonly developed using routinely collected vital-sign data from the whole in-hospital population. As these in-hospital populations are dominated by those over the age of 45 years, resultant scores may perform less well in younger age groups. We developed and validated an age-specific early warning score (ASEWS) derived from statistical distributions of vital signs.

DESIGN

Observational cohort study.

SETTING

Oxford University Hospitals (OUH) July 2013 to March 2018 and Portsmouth Hospitals (PH) NHS Trust January 2010 to March 2017 within the Hospital Alerting Via Electronic Noticeboard database.

PARTICIPANTS

Hospitalised patients with electronically documented vital-sign observations OUTCOME: Composite outcome of unplanned intensive care unit admission, mortality and cardiac arrest.

METHODS AND RESULTS

Statistical distributions of vital signs were used to develop an ASEWS to predict the composite outcome within 24 hours. The OUH development set consisted of 2 538 099 vital-sign observation sets from 142 806 admissions (mean age (SD): 59.8 (20.3)). We compared the performance of ASEWS to the National Early Warning Score (NEWS) and our previous EWS (MCEWS) on an OUH validation set consisting of 581 571 observation sets from 25 407 emergency admissions (mean age (SD): 63.0 (21.4)) and a PH validation set consisting of 5 865 997 observation sets from 233 632 emergency admissions (mean age (SD): 64.3 (21.1)). ASEWS performed better in the 16-45 years age group in the OUH validation set (AUROC 0.820 (95% CI 0.815 to 0.824)) and PH validation set (AUROC 0.840 (95% CI 0.839 to 0.841)) than NEWS (AUROC 0.763 (95% CI 0.758 to 0.768) and AUROC 0.836 (95% CI 0.835 to 0.838) respectively) and MCEWS (AUROC 0.808 (95% CI 0.803 to 0.812) and AUROC 0.833 (95% CI 0.831 to 0.834) respectively). Differences in performance were not consistent in the elder age group.

CONCLUSIONS

Accounting for age-related vital sign changes can more accurately detect deterioration in younger patients.

Increasing Cardiovascular Data Sampling Frequency and Referencing It to Baseline Improve Hemorrhage Detection

Supplemental Digital Content is available in the text.

We hypothesize that knowledge of a stable personalized baseline state and increased data sampling frequency would markedly improve the ability to detect progressive hypovolemia during hemorrhage earlier and with a lower false positive rate than when using less granular data.

Deep Interpretable Early Warning System for the Detection of Clinical Deterioration

Assessment of physiological instability preceding adverse events on hospital wards has been previously investigated through clinical early warning score systems. Early warning scores are simple to use yet they consider data as independent and identically distributed random variables. Deep learning applications are able to learn from sequential data, however they lack interpretability and are thus difficult to deploy in clinical settings. We propose the 'Deep Early Warning System' (DEWS), an interpretable end-to-end deep learning model that interpolates temporal data and predicts the probability of an adverse event, defined as the composite outcome of cardiac arrest, mortality or unplanned ICU admission. The model was developed and validated using routinely collected vital signs of patients admitted to the the Oxford University Hospitals between 21st March 2014 and 31st March 2018. We extracted 45 314 vital-sign measurements as a balanced training set and 359 481 vital-sign measurements as an imbalanced testing set to mimic a real-life setting of emergency admissions. DEWS achieved superior accuracy than the state-of-the-art that is currently implemented in clinical settings, the National Early Warning Score, in terms of the overall area under the receiver operating characteristic curve (AUROC) (0.880 vs. 0.866) and when evaluated independently for each of the three outcomes. Our attention-based architecture was able to recognize 'historical' trends in the data that are most correlated with the predicted probability. With high sensitivity, improved clinical utility and increased interpretability, our model can be easily deployed in clinical settings to supplement existing EWS systems.

The effect of fractional inspired oxygen concentration on early warning score performance: A database analysis

OBJECTIVES

To calculate fractional inspired oxygen concentration (FiO2) thresholds in ward patients and add these to the National Early Warning Score (NEWS). To evaluate the performance of NEWS-FiO2 against NEWS when predicting in-hospital death and unplanned intensive care unit (ICU) admission.

METHODS

A multi-centre, retrospective, observational cohort study was carried out in five hospitals from two UK NHS Trusts. Adult admissions with at least one complete set of vital sign observations recorded electronically were eligible. The primary outcome measure was an 'adverse event' which comprised either in-hospital death or unplanned ICU admission. Discrimination was assessed using the Area Under the Receiver Operating Characteristic curve (AUROC).

RESULTS

A cohort of 83,304 patients from a total of 271,363 adult admissions were prescribed oxygen. In this cohort, NEWS-FiO2 (AUROC 0.823, 95% CI 0.819-0.824) outperformed NEWS (AUORC 0.811, 95% CI 0.809-0.814) when predicting in-hospital death or unplanned ICU admission within 24 h of a complete set of vital sign observations.

CONCLUSIONS

NEWS-FiO2 generates a performance gain over NEWS when studied in ward patients requiring oxygen. This warrants further study, particularly in patients with respiratory disorders.