Predicting out of intensive care unit cardiopulmonary arrest or death using electronic medical record data

Early Warning Systems for Critical Illness Outside the Intensive Care Unit

Early warning systems (EWSs) are designed and deployed to create a rapid assessment and response for patients with clinical deterioration outside the intensive care unit (ICU). These models incorporate patient-level data such as vital signs and laboratory values to detect or prevent adverse clinical events, such as vital signs and laboratories to allow detection and prevention of adverse clinical events such as cardiac arrest, intensive care transfer, or sepsis. The applicability, development, clinical utility, and general perception of EWS in clinical practice vary widely. Here, we review the field as it has grown from early vital sign-based scoring systems to contemporary multidimensional algorithms and predictive technologies for clinical decompensation outside the ICU.

Clinical Predictive Modeling of Heart Failure: Domain Description, Models' Characteristics and Literature Review

This study attempts to identify and briefly describe the current directions in applied and theoretical clinical prediction research. Context-rich chronic heart failure syndrome (CHFS) telemedicine provides the medical foundation for this effort. In the chronic stage of heart failure, there are sudden exacerbations of syndromes with subsequent hospitalizations, which are called acute decompensation of heart failure (ADHF). These decompensations are the subject of diagnostic and prognostic predictions. The primary purpose of ADHF predictions is to clarify the current and future health status of patients and subsequently optimize therapeutic responses. We proposed a simplified discrete-state disease model as an attempt at a typical summarization of a medical subject before starting predictive modeling. The study tries also to structure the essential common characteristics of quantitative models in order to understand the issue in an application context. The last part provides an overview of prediction works in the field of CHFS. These three parts provide the reader with a comprehensive view of quantitative clinical predictive modeling in heart failure telemedicine with an emphasis on several key general aspects. The target community is medical researchers seeking to align their clinical studies with prognostic or diagnostic predictive modeling, as well as other predictive researchers. The study was written by a non-medical expert.

Development and Validation of a Machine Learning Algorithm Using Clinical Pages to Predict Imminent Clinical Deterioration

BACKGROUND

Early detection of clinical deterioration among hospitalized patients is a clinical priority for patient safety and quality of care. Current automated approaches for identifying these patients perform poorly at identifying imminent events.

OBJECTIVE

Develop a machine learning algorithm using pager messages sent between clinical team members to predict imminent clinical deterioration.

DESIGN

We conducted a large observational study using long short-term memory machine learning models on the content and frequency of clinical pages.

PARTICIPANTS

We included all hospitalizations between January 1, 2018 and December 31, 2020 at Vanderbilt University Medical Center that included at least one page message to physicians. Exclusion criteria included patients receiving palliative care, hospitalizations with a planned intensive care stay, and hospitalizations in the top 2% longest length of stay.

MAIN MEASURES

Model classification performance to identify in-hospital cardiac arrest, transfer to intensive care, or Rapid Response activation in the next 3-, 6-, and 12-hours. We compared model performance against three common early warning scores: Modified Early Warning Score, National Early Warning Score, and the Epic Deterioration Index.

KEY RESULTS

There were 87,783 patients (mean [SD] age 54.0 [18.8] years; 45,835 [52.2%] women) who experienced 136,778 hospitalizations. 6214 hospitalized patients experienced a deterioration event. The machine learning model accurately identified 62% of deterioration events within 3-hours prior to the event and 47% of events within 12-hours. Across each time horizon, the model surpassed performance of the best early warning score including area under the receiver operating characteristic curve at 6-hours (0.856 vs. 0.781), sensitivity at 6-hours (0.590 vs. 0.505), specificity at 6-hours (0.900 vs. 0.878), and F-score at 6-hours (0.291 vs. 0.220).

CONCLUSIONS

Machine learning applied to the content and frequency of clinical pages improves prediction of imminent deterioration. Using clinical pages to monitor patient acuity supports improved detection of imminent deterioration without requiring changes to clinical workflow or nursing documentation.

External Validation and Comparison of a General Ward Deterioration Index Between Diversely Different Health Systems

OBJECTIVES

Implementing a predictive analytic model in a new clinical environment is fraught with challenges. Dataset shifts such as differences in clinical practice, new data acquisition devices, or changes in the electronic health record (EHR) implementation mean that the input data seen by a model can differ significantly from the data it was trained on. Validating models at multiple institutions is therefore critical. Here, using retrospective data, we demonstrate how Predicting Intensive Care Transfers and other UnfoReseen Events (PICTURE), a deterioration index developed at a single academic medical center, generalizes to a second institution with significantly different patient population.

DESIGN

PICTURE is a deterioration index designed for the general ward, which uses structured EHR data such as laboratory values and vital signs.

SETTING

The general wards of two large hospitals, one an academic medical center and the other a community hospital.

SUBJECTS

The model has previously been trained and validated on a cohort of 165,018 general ward encounters from a large academic medical center. Here, we apply this model to 11,083 encounters from a separate community hospital.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The hospitals were found to have significant differences in missingness rates (> 5% difference in 9/52 features), deterioration rate (4.5% vs 2.5%), and racial makeup (20% non-White vs 49% non-White). Despite these differences, PICTURE's performance was consistent (area under the receiver operating characteristic curve [AUROC], 0.870; 95% CI, 0.861-0.878), area under the precision-recall curve (AUPRC, 0.298; 95% CI, 0.275-0.320) at the first hospital; AUROC 0.875 (0.851-0.902), AUPRC 0.339 (0.281-0.398) at the second. AUPRC was standardized to a 2.5% event rate. PICTURE also outperformed both the Epic Deterioration Index and the National Early Warning Score at both institutions.

CONCLUSIONS

Important differences were observed between the two institutions, including data availability and demographic makeup. PICTURE was able to identify general ward patients at risk of deterioration at both hospitals with consistent performance (AUROC and AUPRC) and compared favorably to existing metrics.

NURSE-LED proactive rounding and automatic early-warning score systems to prevent resuscitation incidences among Adults in ward-based Hospitalised patients

Objectives

In this study, we investigated the impact of critical care outreach implemented to overcome the problem of rapid response system (RRS) activation. The aim was to evaluate the impact of nurse-led proactive rounding on the rate of adverse events in a hospital setting using an automatic early-warning score system, without a call-activated team.

Methods

This observational study was conducted at a university hospital in Japan. Beginning in September 2019, critical care outreach via nurse-led proactive rounding of the general ward was conducted, using an automatic early-warning score system. We retrospectively assessed the computerised records of all inpatient days (N = 497,284) of adult inpatients admitted to the hospital from September 2017 to 2020. We compared the adverse event occurrences before and after implementation of the critical care outreach program. The main outcome measures were: unexpected death in the general ward, code blue (an in-hospital resuscitation request code directed towards all staff via broadcast) for non-intensive care unit inpatients and unexpected intensive care unit admissions from the general ward. The secondary outcome was the proportion of patients who received respiratory rate measurement.

Results

The incidence rate ratios of the occurrence of unexpected deaths (0.19, 95% confidence interval: 0.04-0.57) and code blue in the general ward (0.15, 95% confidence interval: 0.025-0.50) decreased. There was no change in unexpected intensive care unit admissions from the general ward (1.25, confidence interval: 0.84-1.82). The proportion of patients who received respiratory rate measurement increased (10.2% vs 16.2%).

Conclusion

Our results suggest that in RRSs, drastic control of the failure of the mechanism to activate a response team may produce positive outcomes. Proactive rounding that bypasses the mechanism to activate a response team component of RRSs may relieve ward nurses of activation failure responsibility and help them overcome the hierarchical hospital structure.

Dealing With Missing, Imbalanced, and Sparse Features During the Development of a Prediction Model for Sudden Death Using Emergency Medicine Data: Machine Learning Approach

BACKGROUND

In emergency departments (EDs), early diagnosis and timely rescue, which are supported by prediction modes using ED data, can increase patients' chances of survival. Unfortunately, ED data usually contain missing, imbalanced, and sparse features, which makes it challenging to build early identification models for diseases.

OBJECTIVE

This study aims to propose a systematic approach to deal with the problems of missing, imbalanced, and sparse features for developing sudden-death prediction models using emergency medicine (or ED) data.

METHODS

We proposed a 3-step approach to deal with data quality issues: a random forest (RF) for missing values, k-means for imbalanced data, and principal component analysis (PCA) for sparse features. For continuous and discrete variables, the decision coefficient R² and the κ coefficient were used to evaluate performance, respectively. The area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) were used to estimate the model's performance. To further evaluate the proposed approach, we carried out a case study using an ED data set obtained from the Hainan Hospital of Chinese PLA General Hospital. A logistic regression (LR) prediction model for patient condition worsening was built.

RESULTS

A total of 1085 patients with rescue records and 17,959 patients without rescue records were selected and significantly imbalanced. We extracted 275, 402, and 891 variables from laboratory tests, medications, and diagnosis, respectively. After data preprocessing, the median R² of the RF continuous variable interpolation was 0.623 (IQR 0.647), and the median of the κ coefficient for discrete variable interpolation was 0.444 (IQR 0.285). The LR model constructed using the initial diagnostic data showed poor performance and variable separation, which was reflected in the abnormally high odds ratio (OR) values of the 2 variables of cardiac arrest and respiratory arrest (201568034532 and 1211118945, respectively) and an abnormal 95% CI. Using processed data, the recall of the model reached 0.746, the F₁-score was 0.73, and the AUROC was 0.708.

CONCLUSIONS

The proposed systematic approach is valid for building a prediction model for emergency patients.

Development and Validation of the Medical Emergency Team-Risk Prediction Model for Clinical Deterioration in Acute Hospital Patients, at Time of an Emergency Admission

OBJECTIVES

To develop and validate a prediction model to estimate the risk of Medical Emergency Team (MET) review, within 48 hours of an emergency admission, using information routinely available at the time of hospital admission.

DESIGN

Development and validation of a multivariable risk model using prospectively collected data. Transparent Reporting of a multivariable model for Individual Prognosis Or Diagnosis recommendations were followed to develop and report the prediction model.

SETTING

A 560-bed teaching hospital, with a 22-bed ICU and 24-hour Emergency Department in Melbourne, Australia.

PATIENTS

A total of 45,170 emergency admissions of 30,064 adult patients (≥18 yr), with an inpatient length of stay greater than 24 hours, admitted under acute medical or surgical hospital services between 2015 and 2017.

MEASUREMENTS AND MAIN RESULTS

The outcome was MET review within 48 hours of emergency admission. Thirty candidate variables were selected from a routinely collected hospital dataset based on their availability to clinicians at the time of admission. The final model included nine variables: age; comorbid alcohol-related behavioral diagnosis; history of heart failure, chronic obstructive pulmonary disease (COPD), or renal disease; admitted from residential care; Charlson Comorbidity Index score 1 or 2, or 3+; at least one planned and one emergency admission in the last year; and admission diagnosis and one interaction (past history of COPD × admission diagnosis). The discrimination of the model was comparable in the training (C-statistics 0.82; 95% CI, 0.81-0.83) and the validation set (0.81; 0.80-0.83). Calibration was reasonable for training and validation sets.

CONCLUSIONS

Using only nine predictor variables available to clinicians at the time of admission, the MET-risk model can predict the risk of MET review during the first 48 hours of an emergency admission. Model utility in improving patient outcomes requires further investigation.

Risk Management In Intensive Care Units With Artificial Intelligence Technologies: Systematic Review of Prediction Models Using Electronic Health Records

Background and aim: Clinical risk assessments should be made to protect patients from negative outcomes, and the definition, frequency and severity of the risk should be determined. The information contained in the electronic health records (EHRs) can use in different areas such as risk prediction, estimation of treatment effect ect. Many prediction models using artificial intelligence (AI) technologies that can be used in risk assessment have been developed. The aim of this study is to bring together the researches on prediction models developed with AI technologies using the EHRs of patients hospitalized in the intensive care unit (ICU) and to evaluate them in terms of risk management in healthcare. Methods: The study restricted the search to the Web of Science, Pubmed, Science Direct, and Medline databases to retrieve research articles published in English in 2010 and after. Studies with a prediction model using data obtained from EHRs in the ICU are included. The study focused solely on research conducted in ICU to predict a health condition that poses a significant risk to patient safety using artificial intellegence (AI) technologies. Results: Recognized prediction subcategories were mortality (n=6), sepsis (n=4), pressure ulcer (n=4), acute kidney injury (n=3), and other areas (n=10). It has been found that EHR-based prediction models are good risk management and decision support tools and adoption of such models in ICUs may reduce the prevalence of adverse conditions. Conclusions: The article results remarks that developed models was found to have higher performance and better selectivity than previously developed risk models, so they are better at predicting risks and serious adverse events in ICU. It is recommended to use AI based prediction models developed using EHRs in risk management studies. Future work is still needed to researches to predict different health conditions risks.

Artificial intelligence and clinical deterioration

PURPOSE OF REVIEW

To provide an overview of the systems being used to identify and predict clinical deterioration in hospitalised patients, with focus on the current and future role of artificial intelligence (AI).

RECENT FINDINGS

There are five leading AI driven systems in this field: the Advanced Alert Monitor (AAM), the electronic Cardiac Arrest Risk Triage (eCART) score, Hospital wide Alert Via Electronic Noticeboard, the Mayo Clinic Early Warning Score, and the Rothman Index (RI). Each uses Electronic Patient Record (EPR) data and machine learning to predict adverse events. Less mature but relevant evolutions are occurring in the fields of Natural Language Processing, Time and Motion Studies, AI Sepsis and COVID-19 algorithms.

SUMMARY

Research-based AI-driven systems to predict clinical deterioration are increasingly being developed, but few are being implemented into clinical workflows. Escobar et al. (AAM) provide the current gold standard for robust model development and implementation methodology. Multiple technologies show promise, however, the pathway to meaningfully affect patient outcomes remains challenging.

Evaluation of a digital system to predict unplanned admissions to the intensive care unit: A mixed-methods approach

Background

We have developed the Hospital Alerting Via Electronic Noticeboard (HAVEN) which aims to identify hospitalised patients most at risk of reversible deterioration. HAVEN combines patients’ vital-sign measurements with laboratory results, demographics and comorbidities using a machine learnt algorithm.

Objectives

The aim of this study was to identify variables or concepts that could improve HAVEN predictive performance.

Methods

This was an embedded, mixed methods study. Eligible patients with the five highest HAVEN scores in the hospital (i.e., ‘HAVEN Top 5′) had their medical identification details recorded. We conducted a structured medical note review on these patients 48 hours post their identifiers being recorded. Methods of constant comparison were used during data collection and to analyse patient data.

Results

The 129 patients not admitted to ICU then underwent constant comparison review, which produced three main groups. Group 1 were patients referred to specialist services (n = 37). Group 2 responded to ward-based treatment, (n = 38). Group 3 were frail and had documented treatment limitations (n = 47).

Conclusions

Digital-only validation methods code the cohort not admitted to ICU as ‘falsely positive’ in sensitivity analyses however this approach limits the evaluation of model performance. Our study suggested that coding for patients referred to other specialist teams, those with treatment limitations in place, along with those who are deteriorating but then respond to ward-based therapies, would give a more accurate measure of the value of the scores, especially in relation to cost-effectiveness of resource utilisation.

Algorithms for Prediction of Clinical Deterioration on the General Wards: A Scoping Review

OBJECTIVE

The primary objective of this scoping review was to identify and describe state-of-the-art models that use vital sign monitoring to predict clinical deterioration on the general ward. The secondary objective was to identify facilitators, barriers, and effects of implementing these models.

DATA SOURCES

PubMed, Embase, and CINAHL databases until November 2020.

STUDY SELECTION

We selected studies that compared vital signs-based automated real-time predictive algorithms to current track-and-trace protocols in regard to the outcome of clinical deterioration in a general ward population.

DATA EXTRACTION

Study characteristics, predictive characteristics and barriers, facilitators, and effects.

RESULTS

We identified 1741 publications, 21 of which were included in our review. Two of the these were clinical trials, 2 were prospective observational studies, and the remaining 17 were retrospective studies. All of the studies focused on hospitalized adult patients. The reported area under the receiver operating characteristic curves ranged between 0.65 and 0.95 for the outcome of clinical deterioration. Positive predictive value and sensitivity ranged between 0.223 and 0.773 and from 7.2% to 84.0%, respectively. Input variables differed widely, and predicted endpoints were inconsistently defined. We identified 57 facilitators and 48 barriers to the implementation of these models. We found 68 reported effects, 57 of which were positive.

CONCLUSION

Predictive algorithms can detect clinical deterioration on the general ward earlier and more accurately than conventional protocols, which in one recent study led to lower mortality. Consensus is needed on input variables, predictive time horizons, and definitions of endpoints to better facilitate comparative research.

Utilizing timestamps of longitudinal electronic health record data to classify clinical deterioration events

OBJECTIVE

To propose an algorithm that utilizes only timestamps of longitudinal electronic health record data to classify clinical deterioration events.

MATERIALS AND METHODS

This retrospective study explores the efficacy of machine learning algorithms in classifying clinical deterioration events among patients in intensive care units using sequences of timestamps of vital sign measurements, flowsheets comments, order entries, and nursing notes. We design a data pipeline to partition events into discrete, regular time bins that we refer to as timesteps. Logistic regressions, random forest classifiers, and recurrent neural networks are trained on datasets of different length of timesteps, respectively, against a composite outcome of death, cardiac arrest, and Rapid Response Team calls. Then these models are validated on a holdout dataset.

RESULTS

A total of 6720 intensive care unit encounters meet the criteria and the final dataset includes 830 578 timestamps. The gated recurrent unit model utilizes timestamps of vital signs, order entries, flowsheet comments, and nursing notes to achieve the best performance on the time-to-outcome dataset, with an area under the precision-recall curve of 0.101 (0.06, 0.137), a sensitivity of 0.443, and a positive predictive value of 0. 092 at the threshold of 0.6.

DISCUSSION AND CONCLUSION

This study demonstrates that our recurrent neural network models using only timestamps of longitudinal electronic health record data that reflect healthcare processes achieve well-performing discriminative power.

Contact and Remote Breathing Rate Monitoring Techniques: A Review

Breathing rate monitoring is a must for hospitalized patients with the current coronavirus disease 2019 (COVID-19). We review in this paper recent implementations of breathing monitoring techniques, where both contact and remote approaches are presented. It is known that with non-contact monitoring, the patient is not tied to an instrument, which improves patients' comfort and enhances the accuracy of extracted breathing activity, since the distress generated by a contact device is avoided. Remote breathing monitoring allows screening people infected with COVID-19 by detecting abnormal respiratory patterns. However, non-contact methods show some disadvantages such as the higher set-up complexity compared to contact ones. On the other hand, many reported contact methods are mainly implemented using discrete components. While, numerous integrated solutions have been reported for non-contact techniques, such as continuous wave (CW) Doppler radar and ultrawideband (UWB) pulsed radar. These radar chips are discussed and their measured performances are summarized and compared.

Demonstrating the consequences of learning missingness patterns in early warning systems for preventative health care: A novel simulation and solution

When using tree-based methods to develop predictive analytics and early warning systems for preventive healthcare, it is important to use an appropriate imputation method to prevent learning the missingness pattern. To demonstrate this, we developed a novel simulation that generated synthetic electronic health record data using a variational autoencoder with a custom loss function, which took into account the high missing rate of electronic health data. We showed that when tree-based methods learn missingness patterns (correlated with adverse events) in electronic health record data, this leads to decreased performance if the system is used in a new setting that has different missingness patterns. Performance is worst in this scenario when the missing rate between those with and without an adverse event is the greatest. We found that randomized and Bayesian regression imputation methods mitigate the issue of learning the missingness pattern for tree-based methods. We used this information to build a novel early warning system for predicting patient deterioration in general wards and telemetry units: PICTURE (Predicting Intensive Care Transfers and other UnfoReseen Events). To develop, tune, and test PICTURE, we used labs and vital signs from electronic health records of adult patients over four years (n = 133,089 encounters). We analyzed primary outcomes of unplanned intensive care unit transfer, emergency vasoactive medication administration, cardiac arrest, and death. We compared PICTURE with existing early warning systems and logistic regression at multiple levels of granularity. When analyzing PICTURE on the testing set using all observations within a hospital encounter (event rate = 3.4%), PICTURE had an area under the receiver operating characteristic curve (AUROC) of 0.83 and an adjusted (event rate = 4%) area under the precision-recall curve (AUPR) of 0.27, while the next best tested method-regularized logistic regression-had an AUROC of 0.80 and an adjusted AUPR of 0.22. To ensure system interpretability, we applied a state-of-the-art prediction explainer that provided a ranked list of features contributing most to the prediction. Though it is currently difficult to compare machine learning-based early warning systems, a rudimentary comparison with published scores demonstrated that PICTURE is on par with state-of-the-art machine learning systems. To facilitate more robust comparisons and development of early warning systems in the future, we have released our variational autoencoder's code and weights so researchers can (a) test their models on data similar to our institution and (b) make their own synthetic datasets.

Wireless monitoring and real-time adaptive predictive indicator of deterioration

To assist in the early warning of deterioration in hospitalised children we studied the feasibility of collecting continuous wireless physiological data using Lifetouch (ECG-derived heart and respiratory rate) and WristOx2 (pulse-oximetry and derived pulse rate) sensors. We compared our bedside paediatric early warning (PEW) score and a machine learning automated approach: a Real-time Adaptive Predictive Indicator of Deterioration (RAPID) to identify children experiencing significant clinical deterioration. 982 patients contributed 7,073,486 min during 1,263 monitoring sessions. The proportion of intended monitoring time was 93% for Lifetouch and 55% for WristOx2. Valid clinical data was 63% of intended monitoring time for Lifetouch and 50% WristOx2. 29 patients experienced 36 clinically significant deteriorations. The RAPID Index detected significant deterioration more frequently (77% to 97%) and earlier than the PEW score ≥ 9/26. High sensitivity and negative predictive value for the RAPID Index was associated with low specificity and low positive predictive value. We conclude that it is feasible to collect clinically valid physiological data wirelessly for 50% of intended monitoring time. The RAPID Index identified more deterioration, before the PEW score, but has a low specificity. By using the RAPID Index with a PEW system some life-threatening events may be averted.

PPG Derived Respiratory Rate Estimation in Daily living Conditions

Respiratory rate (RR) derived from photoplethysmogram (PPG) during daily activities can be corrupted due to movement and other artefacts. We have investigated the use of ensemble empirical mode decomposition (EEMD) based smart fusion approach for improving the RR extraction from PPG. PPG was recorded while subjects performed five different activities: sitting, standing, climbing and descending stairs, walking, and running. RR was obtained using EEMD and smart fusion. The median absolute error (AE) of the proposed method is superior, median AE = 3.05 (range 3.01 to 3.18) breath/min in estimating RR during five different activities. Therefore, the proposed method can be implemented for overcoming the artefact problems when recording continuous RR monitoring during activities of daily living.

Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology

OBJECTIVE

To provide an overview and critical appraisal of early warning scores for adult hospital patients.

DESIGN

Systematic review.

DATA SOURCES

Medline, CINAHL, PsycInfo, and Embase until June 2019.

ELIGIBILITY CRITERIA FOR STUDY SELECTION

Studies describing the development or external validation of an early warning score for adult hospital inpatients.

RESULTS

13 171 references were screened and 95 articles were included in the review. 11 studies were development only, 23 were development and external validation, and 61 were external validation only. Most early warning scores were developed for use in the United States (n=13/34, 38%) and the United Kingdom (n=10/34, 29%). Death was the most frequent prediction outcome for development studies (n=10/23, 44%) and validation studies (n=66/84, 79%), with different time horizons (the most frequent was 24 hours). The most common predictors were respiratory rate (n=30/34, 88%), heart rate (n=28/34, 83%), oxygen saturation, temperature, and systolic blood pressure (all n=24/34, 71%). Age (n=13/34, 38%) and sex (n=3/34, 9%) were less frequently included. Key details of the analysis populations were often not reported in development studies (n=12/29, 41%) or validation studies (n=33/84, 39%). Small sample sizes and insufficient numbers of event patients were common in model development and external validation studies. Missing data were often discarded, with just one study using multiple imputation. Only nine of the early warning scores that were developed were presented in sufficient detail to allow individualised risk prediction. Internal validation was carried out in 19 studies, but recommended approaches such as bootstrapping or cross validation were rarely used (n=4/19, 22%). Model performance was frequently assessed using discrimination (development n=18/22, 82%; validation n=69/84, 82%), while calibration was seldom assessed (validation n=13/84, 15%). All included studies were rated at high risk of bias.

CONCLUSIONS

Early warning scores are widely used prediction models that are often mandated in daily clinical practice to identify early clinical deterioration in hospital patients. However, many early warning scores in clinical use were found to have methodological weaknesses. Early warning scores might not perform as well as expected and therefore they could have a detrimental effect on patient care. Future work should focus on following recommended approaches for developing and evaluating early warning scores, and investigating the impact and safety of using these scores in clinical practice.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42017053324.

Development and validation of early warning score system: A systematic literature review

OBJECTIVES

This review aims to: 1) evaluate the quality of model reporting, 2) provide an overview of methodology for developing and validating Early Warning Score Systems (EWSs) for adult patients in acute care settings, and 3) highlight the strengths and limitations of the methodologies, as well as identify future directions for EWS derivation and validation studies.

METHODOLOGY

A systematic search was conducted in PubMed, Cochrane Library, and CINAHL. Only peer reviewed articles and clinical guidelines regarding developing and validating EWSs for adult patients in acute care settings were included. 615 articles were extracted and reviewed by five of the authors. Selected studies were evaluated based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist. The studies were analyzed according to their study design, predictor selection, outcome measurement, methodology of modeling, and validation strategy.

RESULTS

A total of 29 articles were included in the final analysis. Twenty-six articles reported on the development and validation of a new EWS, while three reported on validation and model modification. Only eight studies met more than 75% of the items in the TRIPOD checklist. Three major techniques were utilized among the studies to inform their predictive algorithms: 1) clinical-consensus models (n = 6), 2) regression models (n = 15), and 3) tree models (n = 5). The number of predictors included in the EWSs varied from 3 to 72 with a median of seven. Twenty-eight models included vital signs, while 11 included lab data. Pulse oximetry, mental status, and other variables extracted from electronic health records (EHRs) were among other frequently used predictors. In-hospital mortality, unplanned transfer to the intensive care unit (ICU), and cardiac arrest were commonly used clinical outcomes. Twenty-eight studies conducted a form of model validation either within the study or against other widely-used EWSs. Only three studies validated their model using an external database separate from the derived database.

CONCLUSION

This literature review demonstrates that the characteristics of the cohort, predictors, and outcome selection, as well as the metrics for model validation, vary greatly across EWS studies. There is no consensus on the optimal strategy for developing such algorithms since data-driven models with acceptable predictive accuracy are often site-specific. A standardized checklist for clinical prediction model reporting exists, but few studies have included reporting aligned with it in their publications. Data-driven models are subjected to biases in the use of EHR data, thus it is particularly important to provide detailed study protocols and acknowledge, leverage, or reduce potential biases of the data used for EWS development to improve transparency and generalizability.

Testing a digital system that ranks the risk of unplanned intensive care unit admission in all ward patients: protocol for a prospective observational cohort study

INTRODUCTION

Traditional early warning scores (EWSs) use vital sign derangements to detect clinical deterioration in patients treated on hospital wards. Combining vital signs with demographics and laboratory results improves EWS performance. We have developed the Hospital Alerting Via Electronic Noticeboard (HAVEN) system. HAVEN uses vital signs, as well as demographic, comorbidity and laboratory data from the electronic patient record, to quantify and rank the risk of unplanned admission to an intensive care unit (ICU) within 24 hours for all ward patients. The primary aim of this study is to find additional variables, potentially missed during development, which may improve HAVEN performance. These variables will be sought in the medical record of patients misclassified by the HAVEN risk score during testing.

METHODS

This will be a prospective, observational, cohort study conducted at the John Radcliffe Hospital, part of the Oxford University Hospitals NHS Foundation Trust in the UK. Each day during the study periods, we will document all highly ranked patients (ie, those with the highest risk for unplanned ICU admission) identified by the HAVEN system. After 48 hours, we will review the progress of the identified patients. Patients who were subsequently admitted to the ICU will be removed from the study (as they will have been correctly classified by HAVEN). Highly ranked patients not admitted to ICU will undergo a structured medical notes review. Additionally, at the end of the study periods, all patients who had an unplanned ICU admission but whom HAVEN failed to rank highly will have a structured medical notes review. The review will identify candidate variables, likely associated with unplanned ICU admission, not included in the HAVEN risk score.

ETHICS AND DISSEMINATION

Approval has been granted for gathering the data used in this study from the South Central Oxford C Research Ethics Committee (16/SC/0264, 13 June 2016) and the Confidentiality Advisory Group (16/CAG/0066).

DISCUSSION

Our study will use a clinical expert conducting a structured medical notes review to identify variables, associated with unplanned ICU admission, not included in the development of the HAVEN risk score. These variables will then be added to the risk score and evaluated for potential performance gain. To the best of our knowledge, this is the first study of this type. We anticipate that documenting the HAVEN development methods will assist other research groups developing similar technology.

TRIAL REGISTRATION NUMBER

ISRCTN12518261.

Patient centred variables with univariate associations with unplanned ICU admission: a systematic review

Background

Multiple predictive scores using Electronic Patient Record data have been developed for hospitalised patients at risk of clinical deterioration. Methods used to select patient centred variables for inclusion in these scores varies. We performed a systematic review to describe univariate associations with unplanned Intensive Care Unit (ICU) admission with the aim of assisting model development for future scores that predict clinical deterioration.

Methods

Data sources were MEDLINE, EMBASE, CINAHL, CENTRAL and the Cochrane Database of Systematic Reviews. Included studies were published since 2000 describing an association between patient centred variables and unplanned ICU admission determined using univariate analysis. Two authors independently screened titles, abstracts and full texts against inclusion and exclusion criteria. DistillerSR (Evidence Partners, Canada, Ottawa, Ontario) software was used to manage the data and identify duplicate search results. All screening and data extraction forms were implemented within DistillerSR. Study quality was assessed using an adapted version of the Newcastle-Ottawa Scale. Variables were analysed for strength of association with unplanned ICU admission.

Results

The database search yielded 1520 unique studies; 1462 were removed after title and abstract review; 57 underwent full text screening; 16 studies were included. One hundred and eighty nine variables with an evaluated univariate association with unplanned ICU admission were described.

Discussion

Being male, increasing age, a history of congestive cardiac failure or diabetes, a diagnosis of hepatic disease or having abnormal vital signs were all strongly associated with ICU admission.

Conclusion

These findings will assist variable selection during the development of future models predicting unplanned ICU admission.

Trial registration

This study is a component of a larger body of work registered in the ISRCTN registry (ISRCTN12518261).

Electronic supplementary material

The online version of this article (10.1186/s12911-019-0820-1) contains supplementary material, which is available to authorized users.

Statistical Modeling and Aggregate-Weighted Scoring Systems in Prediction of Mortality and ICU Transfer: A Systematic Review

BACKGROUND

The clinical deterioration of patientsin general hospital wards is an important safety issue. Aggregate-weighted early warning systems (EWSs) may not detect risk until patients present with acute decline.

PURPOSE

We aimed to compare the prognostic test accuracy and clinical workloads generated by EWSs using statistical modeling (multivariable regression or machine learning) versus aggregate-weighted tools.

DATA SOURCES

We searched PubMed and CINAHL using terms that described clinical deterioration and use of an advanced EWS.

STUDY SELECTION

The outcome was clinical deterioration (intensive care unit transfer or death) of adult patients on general hospital wards. We included studies published from January 1, 2012 to September 15, 2018.

DATA EXTRACTION

Following 2015 PRIMSA systematic review protocol guidelines; 2015 TRIPOD criteria for predictive model evaluation; and the Cochrane Collaboration guidelines, we reported model performance, adjusted positive predictive value (PPV), and conducted simulations of workup-to-detection ratios.

DATA SYNTHESIS

Of 285 articles, six studies reported the model performance of advanced EWSs, and five were of high quality. All EWSs using statistical modeling identified at-risk patients with greater precision than aggregate-weighted EWSs (mean AUC 0.80 vs 0.73). EWSs using statistical modeling generated 4.9 alerts to find one true positive case versus 7.1 alerts in aggregate-weighted EWSs; a nearly 50% relative workload increase for aggregate-weighted EWSs.

CONCLUSIONS

Compared with aggregate-weighted tools, EWSs using statistical modeling consistently demonstrated superior prognostic performance and generated less workload to identify and treat one true positive case. A standardized approach to reporting EWS model performance is needed, including outcome definitions, pretest probability, observed and adjusted PPV, and workup-to-detection ratio.

Predicting cardiac arrests in pediatric intensive care units

BACKGROUND

Early identification of children at risk for cardiac arrest would allow for skill training associated with improved outcomes and provides a prevention opportunity.

OBJECTIVE

Develop and assess a predictive model for cardiopulmonary arrest using data available in the first 4 h.

METHODS

Data from PICU patients from 8 institutions included descriptive, severity of illness, cardiac arrest, and outcomes.

RESULTS

Of the 10074 patients, 120 satisfying inclusion criteria sustained a cardiac arrest and 67 (55.9%) died. In univariate analysis, patients with cardiac arrest prior to admission were over 6 times and those with cardiac arrests during the first 4 h were over 50 times more likely to have a subsequent arrest. The multivariate logistic regression model performance was excellent (area under the ROC curve = 0.85 and Hosmer-Lemeshow statistic, p = 0.35). The variables with the highest odds ratio's for sustaining a cardiac arrest in the multivariable model were admission from an inpatient unit (8.23 (CI: 4.35-15.54)), and cardiac arrest in the first 4 h (6.48 (CI: 2.07-20.36). The average risk predicted by the model was highest (11.6%) among children sustaining an arrest during hours >4-12 and continued to be high even for days after the risk assessment period; the average predicted risk was 9.5% for arrests that occurred after 8 PICU days.

CONCLUSIONS

Patients at high risk of cardiac arrest can be identified with routinely available data after 4 h. The cardiac arrest may occur relatively close to the risk assessment period or days later.

Advancing In-Hospital Clinical Deterioration Prediction Models

BACKGROUND

Early warning systems lack robust evidence that they improve patients' outcomes, possibly because of their limitation of predicting binary rather than time-to-event outcomes.

OBJECTIVES

To compare the prediction accuracy of 2 statistical modeling strategies (logistic regression and Cox proportional hazards regression) and 2 machine learning strategies (random forest and random survival forest) for in-hospital cardiopulmonary arrest.

METHODS

Retrospective cohort study with prediction model development from deidentified electronic health records at an urban academic medical center.

RESULTS

The classification models (logistic regression and random forest) had statistical recall and precision similar to or greater than those of the time-to-event models (Cox proportional hazards regression and random survival forest). However, the time-to-event models provided predictions that could potentially better indicate to clinicians whether and when a patient is likely to experience cardiopulmonary arrest.

CONCLUSIONS

As early warning scoring systems are refined, they must use the best analytical methods that both model the underlying phenomenon and provide an understandable prediction.

Participatory design of probability-based decision support tools for in-hospital nurses

Objective

To describe nurses' preferences for the design of a probability-based clinical decision support (PB-CDS) tool for in-hospital clinical deterioration.

Methods

A convenience sample of bedside nurses, charge nurses, and rapid response nurses (n = 20) from adult and pediatric hospitals completed participatory design sessions with researchers in a simulation laboratory to elicit preferred design considerations for a PB-CDS tool. Following theme-based content analysis, we shared findings with user interface designers and created a low-fidelity prototype.

Results

Three major themes and several considerations for design elements of a PB-CDS tool surfaced from end users. Themes focused on "painting a picture" of the patient condition over time, promoting empowerment, and aligning probability information with what a nurse already believes about the patient. The most notable design element consideration included visualizing a temporal trend of the predicted probability of the outcome along with user-selected overlapping depictions of vital signs, laboratory values, and outcome-related treatments and interventions. Participants expressed that the prototype adequately operationalized requests from the design sessions.

Conclusions

Participatory design served as a valuable method in taking the first step toward developing PB-CDS tools for nurses. This information about preferred design elements of tools that support, rather than interrupt, nurses' cognitive workflows can benefit future studies in this field as well as nurses' practice.

Automated Medication Error Risk Assessment System (Auto-MERAS)

This study developed the Automated Medical Error Risk Assessment System (Auto-MERAS), which was incorporated into the electronic health record system. The system itself maintained high predictive validity for medication errors at the area under the receiver operating characteristic curves of above 0.80 at the time of development and validation. This study has found possibilities to predict the risk of medication errors that are sensitive to situational and environmental risks without additional data entry from nurses.

A portable respiratory rate estimation system with a passive single-lead electrocardiogram acquisition module

BACKGROUND

Among vital signs of acutely ill hospital patients, respiratory rate (RR) is a highly accurate predictor of health deterioration.

OBJECTIVE

This study proposes a system that consists of a passive and non-invasive single-lead electrocardiogram (ECG) acquisition module and an ECG-derived respiratory (EDR) algorithm in the working prototype of a mobile application.

METHOD

Before estimating RR that produces the EDR rate, ECG signals were evaluated based on the signal quality index (SQI). The SQI algorithm was validated quantitatively using the PhysioNet/Computing in Cardiology Challenge 2011 training data set. The RR extraction algorithm was validated by adopting 40 MIT PhysioNet Multiparameter Intelligent Monitoring in Intensive Care II data set.

RESULTS

The estimated RR showed a mean absolute error (MAE) of 1.4 compared with the ``gold standard'' RR. The proposed system was used to record 20 ECGs of healthy subjects and obtained the estimated RR with MAE of 0.7 bpm.

CONCLUSION

Results indicate that the proposed hardware and algorithm could replace the manual counting method, uncomfortable nasal airflow sensor, chest band, and impedance pneumotachography often used in hospitals. The system also takes advantage of the prevalence of smartphone usage and increase the monitoring frequency of the current ECG of patients with critical illnesses.

Automating Venous Thromboembolism Risk Calculation Using Electronic Health Record Data upon Hospital Admission: The Automated Padua Prediction Score

BACKGROUND

Venous thromboembolism (VTE) risk scores assist providers in determining the relative benefit of prophylaxis for individual patients. While automated risk calculation using simpler electronic health record (EHR) data is feasible, it lacks clinical nuance and may be less predictive. Automated calculation of the Padua Prediction Score (PPS), requiring more complex input such as recent medical events and clinical status, may save providers time and increase risk score use.

OBJECTIVE

We developed the Automated Padua Prediction Score (APPS) to auto-calculate a VTE risk score using EHR data drawn from prior encounters and the first 4 hours of admission. We compared APPS to standard practice of clinicians manually calculating the PPS to assess VTE risk.

DESIGN

Cohort study of 30,726 hospitalized patients. APPS was compared to manual calculation of PPS by chart review from 300 randomly selected patients.

MEASUREMENTS

Prediction of hospital-acquired VTE not present on admission.

RESULTS

Compared to manual PPS calculation, no significant difference in average score was found (5.5 vs. 5.1, P = 0.073), and area under curve (AUC) was similar (0.79 vs. 0.76). Hospital- acquired VTE occurred in 260 (0.8%) of 30,726 patients. Those without VTE averaged APPS of 4.9 (standard deviation [SD], 2.6) and those with VTE averaged 7.7 (SD, 2.6). APPS had AUC = 0.81 (confidence interval [CI], 0.79-0.83) in patients receiving no pharmacologic prophylaxis and AUC = 0.78 (CI, 0.76- 0.82) in patients receiving pharmacologic prophylaxis.

CONCLUSIONS

Automated calculation of VTE risk had similar ability to predict hospital-acquired VTE as manual calculation despite differences in how often specific scoring criteria were considered present by the 2 methods. Journal of Hospital Medicine 2017;12: 231- 237.

Translational bioinformatics in the era of real-time biomedical, health care and wellness data streams

Monitoring and modeling biomedical, health care and wellness data from individuals and converging data on a population scale have tremendous potential to improve understanding of the transition to the healthy state of human physiology to disease setting. Wellness monitoring devices and companion software applications capable of generating alerts and sharing data with health care providers or social networks are now available. The accessibility and clinical utility of such data for disease or wellness research are currently limited. Designing methods for streaming data capture, real-time data aggregation, machine learning, predictive analytics and visualization solutions to integrate wellness or health monitoring data elements with the electronic medical records (EMRs) maintained by health care providers permits better utilization. Integration of population-scale biomedical, health care and wellness data would help to stratify patients for active health management and to understand clinically asymptomatic patients and underlying illness trajectories. In this article, we discuss various health-monitoring devices, their ability to capture the unique state of health represented in a patient and their application in individualized diagnostics, prognosis, clinical or wellness intervention. We also discuss examples of translational bioinformatics approaches to integrating patient-generated data with existing EMRs, personal health records, patient portals and clinical data repositories. Briefly, translational bioinformatics methods, tools and resources are at the center of these advances in implementing real-time biomedical and health care analytics in the clinical setting. Furthermore, these advances are poised to play a significant role in clinical decision-making and implementation of data-driven medicine and wellness care.

Toward a Robust Estimation of Respiratory Rate From Pulse Oximeters

Goal: Current methods for estimating respiratory rate (RR) from the photoplethysmogram (PPG) typically fail to distinguish between periods of high- and low-quality input data, and fail to perform well on independent “validation” datasets. The lack of robustness of existing methods directly results in a lack of penetration of such systems into clinical practice. The present work proposes an alternative method to improve the robustness of the estimation of RR from the PPG. Methods: The proposed algorithm is based on the use of multiple autoregressive models of different orders for determining the dominant respiratory frequency in the three respiratory-induced variations (frequency, amplitude, and intensity) derived from the PPG. The algorithm was tested on two different datasets comprising 95 eight-minute PPG recordings (in total) acquired from both children and adults in different clinical settings, and its performance using two window sizes (32 and 64 seconds) was compared with that of existing methods in the literature. Results: The proposed method achieved comparable accuracy to existing methods in the literature, with mean absolute errors (median, 25\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{} $\text {th}$\end{document}–75\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$\text {th}$\end{document} percentiles for a window size of 32 seconds) of 1.5 (0.3–3.3) and 4.0 (1.8–5.5) breaths per minute (for each dataset respectively), whilst providing RR estimates for a greater proportion of windows (over 90% of the input data are kept). Conclusion: Increased robustness of RR estimation by the proposed method was demonstrated. Significance: This work demonstrates that the use of large publicly available datasets is essential for improving the robustness of wearable-monitoring algorithms for use in clinical practice.

Initial Development of a Computer Algorithm to Identify Patients With Breast and Lung Cancer Having Poor Prognosis in a Safety Net Hospital

BACKGROUND

Physicians often have difficulty with prognostication and identification of patients who are in need of counseling about options for care at the end of life. Consequently, the objective of this study was to describe the initial stages in development of a computerized algorithm that will identify breast and lung cancer patients most in need of counseling about care options, including advance care planning, palliative care, and hospice.

METHODS

Clinical and non-clinical data were extracted from the electronic medical record of breast and lung cancer patients admitted to a large, urban hospital for the year 2010. These data were used to create an electronic (e-EOL) algorithm designed to identify advanced cancer patients who could benefit from in-depth discussion about end-of-life care options.

RESULTS

There were 369 eligible breast (42%) and lung (58%) cancer patients identified by ICD-9 code. The e-EOL algorithm identified 53 (14%) patients that met assigned criteria (presence of metastatic disease and albumin < 2.5 g/dl). The sensitivity, specificity, and positive predictive value of the first generation algorithm were 21%, 96%, and 91% when compared to physician expert chart review. Survival analysis showed that 6-month survival for algorithm positive cases was 46% versus 78% for algorithm negative cases, and 1-year survival was 32% versus 72%, respectively.

CONCLUSIONS

Initial testing of the e-EOL algorithm appears to be promising. Other markers of advanced illness will added to the algorithm to improve its test operating characteristics so it may be used to identify patients with poor prognosis in real time.

The Drug Data to Knowledge Pipeline: Large-Scale Claims Data Classification for Pharmacologic Insight

In biomedical informatics, assigning drug codes to categories is a common step in the analysis pipeline. Unfortunately, incomplete mappings are the norm rather than the exception with coverage values less than 85% not uncommon. Here, we perform this linking task on a nationwide insurance claims database with over 13 million members who were dispensed, according to National Drug Codes (NDCs), over 50,000 unique product forms of medication. The chosen approach employs Cerner Multum's VantageRx and the U.S. National Library of Medicine's RxMix. As a result, 94.0% of the NDCs were successfully mapped to categories used by common drug terminologies, e.g., Anatomical Therapeutic Chemical (ATC). Implemented as an SQL database and scripts, the approach is generic and can be setup for a new data set in a few hours. Thus, the method is a viable option for large-scale drug classification.

Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review

OBJECTIVE

Electronic health records (EHRs) are an increasingly common data source for clinical risk prediction, presenting both unique analytic opportunities and challenges. We sought to evaluate the current state of EHR based risk prediction modeling through a systematic review of clinical prediction studies using EHR data.

METHODS

We searched PubMed for articles that reported on the use of an EHR to develop a risk prediction model from 2009 to 2014. Articles were extracted by two reviewers, and we abstracted information on study design, use of EHR data, model building, and performance from each publication and supplementary documentation.

RESULTS

We identified 107 articles from 15 different countries. Studies were generally very large (median sample size = 26 100) and utilized a diverse array of predictors. Most used validation techniques (n = 94 of 107) and reported model coefficients for reproducibility (n = 83). However, studies did not fully leverage the breadth of EHR data, as they uncommonly used longitudinal information (n = 37) and employed relatively few predictor variables (median = 27 variables). Less than half of the studies were multicenter (n = 50) and only 26 performed validation across sites. Many studies did not fully address biases of EHR data such as missing data or loss to follow-up. Average c-statistics for different outcomes were: mortality (0.84), clinical prediction (0.83), hospitalization (0.71), and service utilization (0.71).

CONCLUSIONS

EHR data present both opportunities and challenges for clinical risk prediction. There is room for improvement in designing such studies.

EHDViz: clinical dashboard development using open-source technologies

OBJECTIVE

To design, develop and prototype clinical dashboards to integrate high-frequency health and wellness data streams using interactive and real-time data visualisation and analytics modalities.

MATERIALS AND METHODS

We developed a clinical dashboard development framework called electronic healthcare data visualization (EHDViz) toolkit for generating web-based, real-time clinical dashboards for visualising heterogeneous biomedical, healthcare and wellness data. The EHDViz is an extensible toolkit that uses R packages for data management, normalisation and producing high-quality visualisations over the web using R/Shiny web server architecture. We have developed use cases to illustrate utility of EHDViz in different scenarios of clinical and wellness setting as a visualisation aid for improving healthcare delivery.

RESULTS

Using EHDViz, we prototyped clinical dashboards to demonstrate the contextual versatility of EHDViz toolkit. An outpatient cohort was used to visualise population health management tasks (n=14,221), and an inpatient cohort was used to visualise real-time acuity risk in a clinical unit (n=445), and a quantified-self example using wellness data from a fitness activity monitor worn by a single individual was also discussed (n-of-1). The back-end system retrieves relevant data from data source, populates the main panel of the application and integrates user-defined data features in real-time and renders output using modern web browsers. The visualisation elements can be customised using health features, disease names, procedure names or medical codes to populate the visualisations. The source code of EHDViz and various prototypes developed using EHDViz are available in the public domain at http://ehdviz.dudleylab.org.

CONCLUSIONS

Collaborative data visualisations, wellness trend predictions, risk estimation, proactive acuity status monitoring and knowledge of complex disease indicators are essential components of implementing data-driven precision medicine. As an open-source visualisation framework capable of integrating health assessment, EHDViz aims to be a valuable toolkit for rapid design, development and implementation of scalable clinical data visualisation dashboards.

Should predictive scores based on vital signs be used in the same way as those based on laboratory data? A hypothesis generating retrospective evaluation of in-hospital mortality by four different scoring systems

BACKGROUND

few studies have compared the discrimination of predictive scores of in-hospital mortality that used vital signs with those using laboratory results in different patient populations.

METHODS

a hypothesis generating retrospective observational cohort study. A score that only used vital signs was compared with three other scores that used laboratory changes in 44,985 medical and 20,432 surgical patients.

RESULTS

the discrimination of the score based only on vital signs was highest for the prediction of in-hospital death within 24h. In contrast the, albeit lower, discrimination of scores based only on laboratory data remained constant for the prediction of death up to 30 days after hospital admission. Moreover, the discrimination of scores based only on laboratory data was higher in surgical than in medical patients.

CONCLUSION

in acutely ill medical patients a vital sign based score appears to predict mortality within 24h better than scores using laboratory data. This may be because in acutely ill patients vital sign changes indicate how well a patient is responding to a current insult. In contrast, for patients without acute illness laboratory data may be a more valuable indication of the patient's capacity to respond to insults in the future.

Postoperative 30-day mortality in patients undergoing surgery for colorectal cancer: development of a prognostic model using administrative claims data

PURPOSE

To develop a prognostic model to predict 30-day mortality following colorectal cancer (CRC) surgery using the Surveillance, Epidemiology, and End Results (SEER)-Medicare-linked data and to assess whether race/ethnicity, neighborhood, and hospital characteristics influence model performance.

METHODS

We included patients aged 66 years and older from the linked 2000-2005 SEER-Medicare database. Outcome included 30-day mortality, both in-hospital and following discharge. Potential prognostic factors included tumor, treatment, sociodemographic, hospital, and neighborhood characteristics (census-tract-poverty rate). We performed a multilevel logistic regression analysis to account for nesting of CRC patients within hospitals. Model performance was assessed using the area under the receiver operating characteristic curve (AUC) for discrimination and the Hosmer-Lemeshow goodness-of-fit test for calibration.

RESULTS

In a model that included all prognostic factors, important predictors of 30-day mortality included age at diagnosis, cancer stage, and mode of presentation. Race/ethnicity, census-tract-poverty rate, and hospital characteristics were independently associated with 30-day mortality, but they did not influence model performance. Our SEER-Medicare model achieved moderate discrimination (AUC = 0.76), despite suboptimal calibration.

CONCLUSIONS

We developed a prognostic model that included tumor, treatment, sociodemographic, hospital, and neighborhood predictors. Race/ethnicity, neighborhood, and hospital characteristics did not improve model performance compared with previously developed models.

Measuring the modified early warning score and the Rothman index: advantages of utilizing the electronic medical record in an early warning system

Early detection of an impending cardiac or pulmonary arrest is an important focus for hospitals trying to improve quality of care. Unfortunately, all current early warning systems suffer from high false-alarm rates. Most systems are based on the Modified Early Warning Score (MEWS); 4 of its 5 inputs are vital signs. The purpose of this study was to compare the accuracy of MEWS against the Rothman Index (RI), a patient acuity score based upon summation of excess risk functions that utilize additional data from the electronic medical record (EMR). MEWS and RI scores were computed retrospectively for 32,472 patient visits. Nursing assessments, a category of EMR inputs only used by the RI, showed sharp differences 24 hours before death. Receiver operating characteristic curves for 24-hour mortality demonstrated superior RI performance with c-statistics, 0.82 and 0.93, respectively. At the point where MEWS triggers an alarm, we identified the RI point corresponding to equal sensitivity and found the positive likelihood ratio (LR+) for MEWS was 7.8, and for the RI was 16.9 with false alarms reduced by 53%. At the RI point corresponding to equal LR+, the sensitivity for MEWS was 49% and 77% for RI, capturing 54% more of those patients who will die within 24 hours.