A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice. Crit Care Med. 2019;47:1485-1492. [PMID: 31389839 DOI: 10.1097/ccm.0000000000003891]

Machine learning model predicts short-term mortality among prehospital patients: A prospective development study from Finland

Aim

To show whether adding blood glucose to the National Early Warning Score (NEWS) parameters in a machine learning model predicts 30-day mortality more precisely than the standard NEWS in a prehospital setting.

Methods

In this study, vital sign data prospectively collected from 3632 unselected prehospital patients in June 2015 were used to compare the standard NEWS to random forest models for predicting 30-day mortality. The NEWS parameters and blood glucose levels were used to develop the random forest models. Predictive performance on an unknown patient population was estimated with a ten-fold stratified cross-validation method.

Results

All NEWS parameters and blood glucose levels were reported in 2853 (79%) eligible patients. Within 30 days after contact with ambulance staff, 97 (3.4%) of the analysed patients had died. The area under the receiver operating characteristic curve for the 30-day mortality of the evaluated models was 0.682 (95% confidence interval [CI], 0.619-0.744) for the standard NEWS, 0.735 (95% CI, 0.679-0.787) for the random forest-trained NEWS parameters only and 0.758 (95% CI, 0.705-0.807) for the random forest-trained NEWS parameters and blood glucose. The models predicted secondary outcomes similarly, but adding blood glucose into the random forest model slightly improved its performance in predicting short-term mortality.

Conclusions

Among unselected prehospital patients, a machine learning model including blood glucose and NEWS parameters had a fair performance in predicting 30-day mortality.

Clinician checklist for assessing suitability of machine learning applications in healthcare

Machine learning algorithms are being used to screen and diagnose disease, prognosticate and predict therapeutic responses. Hundreds of new algorithms are being developed, but whether they improve clinical decision making and patient outcomes remains uncertain. If clinicians are to use algorithms, they need to be reassured that key issues relating to their validity, utility, feasibility, safety and ethical use have been addressed. We propose a checklist of 10 questions that clinicians can ask of those advocating for the use of a particular algorithm, but which do not expect clinicians, as non-experts, to demonstrate mastery over what can be highly complex statistical and computational concepts. The questions are: (1) What is the purpose and context of the algorithm? (2) How good were the data used to train the algorithm? (3) Were there sufficient data to train the algorithm? (4) How well does the algorithm perform? (5) Is the algorithm transferable to new clinical settings? (6) Are the outputs of the algorithm clinically intelligible? (7) How will this algorithm fit into and complement current workflows? (8) Has use of the algorithm been shown to improve patient care and outcomes? (9) Could the algorithm cause patient harm? and (10) Does use of the algorithm raise ethical, legal or social concerns? We provide examples where an algorithm may raise concerns and apply the checklist to a recent review of diagnostic imaging applications. This checklist aims to assist clinicians in assessing algorithm readiness for routine care and identify situations where further refinement and evaluation is required prior to large-scale use.

Demystifying machine learning: a primer for physicians

Machine learning is a tool for analysing digitised data sets and formulating predictions that can optimise clinical decision-making. It aims to identify complex patterns in large data sets and encode them into models that can then classify new unseen cases or make predictions on new data. Machine learning methods take several forms and individual models can be of many different types. More than 50 models have been approved for use in routine healthcare, and the numbers continue to grow exponentially. The reliability and robustness of any model depends on multiple factors, including the quality and quantity of the data used to develop the models, and the selection of features in the data considered most important to maximising accuracy. In ensuring models are safe, effective and reproducible in routine care, physicians need to have some understanding of how these models are developed and evaluated, and to collaborate with data and computer scientists in their design and validation. This narrative review introduces principles, methods and examples of machine learning in a way that does not require mastery of highly complex statistical and computational concepts.

Early Detection of Septic Shock Onset Using Interpretable Machine Learners

BACKGROUND

Developing a decision support system based on advances in machine learning is one area for strategic innovation in healthcare. Predicting a patient's progression to septic shock is an active field of translational research. The goal of this study was to develop a working model of a clinical decision support system for predicting septic shock in an acute care setting for up to 6 h from the time of admission in an integrated healthcare setting.

METHOD

Clinical data from Electronic Health Record (EHR), at encounter level, were used to build a predictive model for progression from sepsis to septic shock up to 6 h from the time of admission; that is, T = 1, 3, and 6 h from admission. Eight different machine learning algorithms (Random Forest, XGBoost, C5.0, Decision Trees, Boosted Logistic Regression, Support Vector Machine, Logistic Regression, Regularized Logistic, and Bayes Generalized Linear Model) were used for model development. Two adaptive sampling strategies were used to address the class imbalance. Data from two sources (clinical and billing codes) were used to define the case definition (septic shock) using the Centers for Medicare & Medicaid Services (CMS) Sepsis criteria. The model assessment was performed using Area under Receiving Operator Characteristics (AUROC), sensitivity, and specificity. Model predictions for each feature window (1, 3 and 6 h from admission) were consolidated.

RESULTS

Retrospective data from April 2005 to September 2018 were extracted from the EHR, Insurance Claims, Billing, and Laboratory Systems to create a dataset for septic shock detection. The clinical criteria and billing information were used to label patients into two classes-septic shock patients and sepsis patients at three different time points from admission, creating two different case-control cohorts. Data from 45,425 unique in-patient visits were used to build 96 prediction models comparing clinical-based definition versus billing-based information as the gold standard. Of the 24 consolidated models (based on eight machine learning algorithms and three feature windows), four models reached an AUROC greater than 0.9. Overall, all the consolidated models reached an AUROC of at least 0.8820 or higher. Based on the AUROC of 0.9483, the best model was based on Random Forest, with a sensitivity of 83.9% and specificity of 88.1%. The sepsis detection window at 6 h outperformed the 1 and 3-h windows. The sepsis definition based on clinical variables had improved performance when compared to the sepsis definition based on only billing information.

CONCLUSION

This study corroborated that machine learning models can be developed to predict septic shock using clinical and administrative data. However, the use of clinical information to define septic shock outperformed models developed based on only administrative data. Intelligent decision support tools can be developed and integrated into the EHR and improve clinical outcomes and facilitate the optimization of resources in real-time.

Development and Validation of a Model to Predict Pediatric Septic Shock Using Data Known 2 Hours After Hospital Arrival

Objective:

To use Electronic Health Record (EHR) data from the first two hours of care to derive and validate a model to predict hypotensive septic shock in children with infection.

Design:

Derivation-validation study using an existing registry

Setting:

Six emergency care sites within a regional pediatric healthcare system. Three datasets of unique visits were designated:

Patients:

Patients in whom clinicians were concerned about serious infection from 60 days-17 years were included; those with septic shock in the first two hours were excluded. There were 2318 included visits; 197 developed septic shock (8.5%).

Interventions:

Lasso with tenfold cross-validation was used for variable selection; logistic regression was then used to construct a model from those variables in the training set. Variables were derived from EHR data known in the first two hours, including vital signs, medical history, demographics, laboratory information. Test characteristics at two thresholds were evaluated: 1) optimizing sensitivity and specificity, 2) set to 90% sensitivity.

Measurements and Main Results:

Septic shock was defined as systolic hypotension and vasoactive use or ≥30 ml/kg isotonic crystalloid administration in the first 24 hours. A model was created using twenty predictors, with an area under the receiver operating curve in the training set of 0.85 (0.82-0.88); 0.83 [0.78-0.89] in the temporal test set; 0.83 [0.60-1.00] in the geographic test set. Sensitivity and specificity varied based on cutpoint; when sensitivity in the training set was set to 90% (83%, 94%), specificity was 62% (60%, 65%).

Conclusions:

This model predicted risk of septic shock in children with suspected infection 2 hours after arrival, a critical timepoint for emergent treatment and transfer decisions. Varied cutpoints could be used to customize sensitivity to clinical context.

COVID-19 stokes inflammasomes

The poor success rate of treating patients with aggressive sepsis in SARS-CoV-2 infections has highlighted again the challenges of managing systemic inflammatory conditions. In this issue of JEM, Rodrigues et al. (https://doi.org/10.1084/jem.20201707) discuss the role of inflammasome activation in COVID-19 disease severity, opening new possibilities for therapeutic management of sepsis syndromes.

Coronavirus Disease 2019 Calls for Predictive Analytics Monitoring-A New Kind of Illness Scoring System

Coronavirus disease 2019 can lead to sudden and severe respiratory failure that mandates endotracheal intubation, a procedure much more safely performed under elective rather than emergency conditions. Early warning of rising risk of this event could benefit both patients and healthcare providers by reducing the high risk of emergency intubation. Current illness severity scoring systems, which usually update only when clinicians measure vital signs or laboratory values, are poorly suited for early detection of this kind of rapid clinical deterioration. We propose that continuous predictive analytics monitoring, a new approach to bedside management, is more useful. The principles of this new practice anchor in analysis of continuous bedside monitoring data, training models on diagnosis-specific paths of deterioration using clinician-identified events, and continuous display of trends in risks rather than alerts when arbitrary thresholds are exceeded.

Random forest machine learning method outperforms prehospital National Early Warning Score for predicting one-day mortality: A retrospective study

Aim of the study

The National Early Warning Score (NEWS) is a validated method for predicting clinical deterioration in hospital wards, but its performance in prehospital settings remains controversial. Modern machine learning models may outperform traditional statistical analyses for predicting short-term mortality. Thus, we aimed to compare the mortality prediction accuracy of NEWS and random forest machine learning using prehospital vital signs.

Methods

In this retrospective study, all electronic ambulance mission reports between 2008 and 2015 in a single EMS system were collected. Adult patients (≥ 18 years) were included in the analysis. Random forest models with and without blood glucose were compared to the traditional NEWS for predicting one-day mortality. A ten-fold cross-validation method was applied to train and validate the random forest models.

Results

A total of 26,458 patients were included in the study of whom 278 (1.0%) died within one day of ambulance mission. The area under the receiver operating characteristic curve for one-day mortality was 0.836 (95% CI, 0.810−0.860) for NEWS, 0.858 (95% CI, 0.832−0.883) for a random forest trained with NEWS variables only and 0.868 (0.843−0.892) for a random forest trained with NEWS variables and blood glucose.

Conclusion

A random forest algorithm trained with NEWS variables was superior to traditional NEWS for predicting one-day mortality in adult prehospital patients, although the risk of selection bias must be acknowledged. The inclusion of blood glucose in the model further improved its predictive performance.

Continuous and automatic mortality risk prediction using vital signs in the intensive care unit: a hybrid neural network approach

Mortality risk prediction can greatly improve the utilization of resources in intensive care units (ICUs). Existing schemes in ICUs today require laborious manual input of many complex parameters. In this work, we present a scheme that uses variations in vital signs over a 24-h period to make mortality risk assessments for 3-day, 7-day, and 14-day windows. We develop a hybrid neural network model that combines convolutional (CNN) layers with bidirectional long short-term memory (BiLSTM) to predict mortality from statistics describing the variation of heart rate, blood pressure, respiratory rate, blood oxygen levels, and temperature. Our scheme performs strongly compared to state-of-the-art schemes in the literature for mortality prediction, with our highest-performing model achieving an area under the receiver-operator curve of 0.884. We conclude that the use of a hybrid CNN-BiLSTM network is highly effective in determining mortality risk for the 3, 7, and 14 day windows from vital signs. As vital signs are routinely recorded, in many cases automatically, our scheme could be implemented such that highly accurate mortality risk could be predicted continuously and automatically, reducing the burden on healthcare providers and improving patient outcomes.

Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study

Background

Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care.

Objective

This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows.

Methods

We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data.

Results

A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes.

Conclusions

This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.

Development and Verification of a Digital Twin Patient Model to Predict Specific Treatment Response During the First 24 Hours of Sepsis

To develop and verify a digital twin model of critically ill patient using the causal artificial intelligence approach to predict the response to specific treatment during the first 24 hours of sepsis.

DESIGN

Directed acyclic graphs were used to define explicitly the causal relationship among organ systems and specific treatments used. A hybrid approach of agent-based modeling, discrete-event simulation, and Bayesian network was used to simulate treatment effect across multiple stages and interactions of major organ systems (cardiovascular, neurologic, renal, respiratory, gastrointestinal, inflammatory, and hematology). Organ systems were visualized using relevant clinical markers. The application was iteratively revised and debugged by clinical experts and engineers. Agreement statistics was used to test the performance of the model by comparing the observed patient response versus the expected response (primary and secondary) predicted by digital twin.

SETTING

Medical ICU of a large quaternary- care academic medical center in the United States.

PATIENTS OR SUBJECTS

Adult (> 18 year yr old), medical ICU patients were included in the study.

INTERVENTIONS

No additional interventions were made beyond the standard of care for this study.

MEASUREMENTS AND MAIN RESULTS

During the verification phase, model performance was prospectively tested on 145 observations in a convenience sample of 29 patients. Median age was 60 years (54-66 d) with a median Sequential Organ Failure Assessment score of 9.5 (interquartile range, 5.0-14.0). The most common source of sepsis was pneumonia, followed by hepatobiliary. The observations were made during the first 24 hours of the ICU admission with one-step interventions, comparing the output in the digital twin with the real patient response. The agreement between the observed versus and the expected response ranged from fair (kappa coefficient of 0.41) for primary response to good (kappa coefficient of 0.65) for secondary response to the intervention. The most common error detected was coding error in 50 observations (35%), followed by expert rule error in 29 observations (20%) and timing error in seven observations (5%).

CONCLUSIONS

We confirmed the feasibility of development and prospective testing of causal artificial intelligence model to predict the response to treatment in early stages of critical illness. The availability of qualitative and quantitative data and a relatively short turnaround time makes the ICU an ideal environment for development and testing of digital twin patient models. An accurate digital twin model will allow the effect of an intervention to be tested in a virtual environment prior to use on real patients.

Gut Dysbiosis Contributes to the Imbalance of Treg and Th17 Cells in Graves' Disease Patients by Propionic Acid

BACKGROUND

Graves' disease (GD) is a typical organ-specific autoimmune disease. Intestinal flora plays a pivotal role in immune homeostasis and autoimmune disease development. However, the association and mechanism between intestinal flora and GD remain elusive.

OBJECTIVE

To investigate the association and mechanism between intestinal flora and GD.

METHODS

We recruited 58 initially untreated GD patients and 63 healthy individuals in the study. The composition and metabolic characteristics of the intestinal flora in GD patients and the causal relationship between intestinal flora and GD pathogenesis were assessed using 16S rRNA gene sequencing, targeted/untargeted metabolomics, and fecal microbiota transplantation.

RESULTS

The composition, metabolism, and inter-relationships of the intestinal flora were also changed, particularly the significantly reduced short-chain fatty acid (SCFA)-producing bacteria and SCFAs. The YCH46 strain of Bacteroides fragilis could produce propionic acid and increase Treg cell numbers while decreasing Th17 cell numbers. Transplanting the intestinal flora of GD patients significantly increased GD incidence in the GD mouse model. Additionally, there were 3 intestinal bacteria genera (Bacteroides, Alistipes, Prevotella) could distinguish GD patients from healthy individuals with 85% accuracy.

CONCLUSIONS

Gut dysbiosis contributes to a Treg/Th17 imbalance through the pathway regulated by propionic acid and promotes the occurrence of GD, together with other pathogenic factors. Bacteroides, Alistipes, and Prevotella have great potential to serve as adjunct markers for GD diagnosis. This study provided valuable clues for improving immune dysfunction of GD patients using B. fragilis and illuminated the prospects of microecological therapy for GD as an adjunct treatment.

Validation of a machine learning algorithm for early severe sepsis prediction: a retrospective study predicting severe sepsis up to 48 h in advance using a diverse dataset from 461 US hospitals

BACKGROUND

Severe sepsis and septic shock are among the leading causes of death in the United States and sepsis remains one of the most expensive conditions to diagnose and treat. Accurate early diagnosis and treatment can reduce the risk of adverse patient outcomes, but the efficacy of traditional rule-based screening methods is limited. The purpose of this study was to develop and validate a machine learning algorithm (MLA) for severe sepsis prediction up to 48 h before onset using a diverse patient dataset.

METHODS

Retrospective analysis was performed on datasets composed of de-identified electronic health records collected between 2001 and 2017, including 510,497 inpatient and emergency encounters from 461 health centers collected between 2001 and 2015, and 20,647 inpatient and emergency encounters collected in 2017 from a community hospital. MLA performance was compared to commonly used disease severity scoring systems and was evaluated at 0, 4, 6, 12, 24, and 48 h prior to severe sepsis onset.

RESULTS

270,438 patients were included in analysis. At time of onset, the MLA demonstrated an AUROC of 0.931 (95% CI 0.914, 0.948) and a diagnostic odds ratio (DOR) of 53.105 on a testing dataset, exceeding MEWS (0.725, P < .001; DOR 4.358), SOFA (0.716; P < .001; DOR 3.720), and SIRS (0.655; P < .001; DOR 3.290). For prediction 48 h prior to onset, the MLA achieved an AUROC of 0.827 (95% CI 0.806, 0.848) on a testing dataset. On an external validation dataset, the MLA achieved an AUROC of 0.948 (95% CI 0.942, 0.954) at the time of onset, and 0.752 at 48 h prior to onset.

CONCLUSIONS

The MLA accurately predicts severe sepsis onset up to 48 h in advance using only readily available vital signs extracted from the existing patient electronic health records. Relevant implications for clinical practice include improved patient outcomes from early severe sepsis detection and treatment.

Practical Machine Learning-Based Sepsis Prediction

Sepsis is a life-threatening clinical syndrome and one of the most expensive conditions treated in hospitals. It is challenging to detect due to the nonspecific clinical signs and the absence of gold standard diagnostics. However, early recognition of sepsis and optimal treatments for sepsis are of paramount importance to improve the condition's management and patient outcomes. This paper aims to delineate key aspects of current sepsis detection systems, including their dependency on clinical expert and laboratory biometric features requiring ongoing critical care intervention, the efficacy of vital sign measures, and the effect of the study population with respect to the precision of sepsis prediction. The AUROC performances of XGBoost models trained on a heterogenous ICU patient group (n=3932) showed significant degradations (p<0.05) as the expert and laboratory biomarker features are removed systematically and vital sign features taken in ICU settings are left. The performance of XGBoost models trained only with vital sign features on a more homogeneous group of ICU patients (n=1927) had a significantly (P<0.05) improved AUPRC to moderate level. The presented results highlight the importance of making a practical machine learning system for sepsis prediction by considering the availability of dominant features as well as personalizing sepsis prediction by configuring it to the specific demographics of a targeted population.

Survival prediction of patients with sepsis from age, sex, and septic episode number alone

Sepsis is a life-threatening condition caused by an exaggerated reaction of the body to an infection, that leads to organ failure or even death. Since sepsis can kill a patient even in just one hour, survival prediction is an urgent priority among the medical community: even if laboratory tests and hospital analyses can provide insightful information about the patient, in fact, they might not come in time to allow medical doctors to recognize an immediate death risk and treat it properly. In this context, machine learning can be useful to predict survival of patients within minutes, especially when applied to few medical features easily retrievable. In this study, we show that it is possible to achieve this goal by applying computational intelligence algorithms to three features of patients with sepsis, recorded at hospital admission: sex, age, and septic episode number. We applied several data mining methods to a cohort of 110,204 admissions of patients, and obtained high prediction scores both on this complete dataset (top precision-recall area under the curve PR AUC = 0.966) and on its subset related to the recent Sepsis-3 definition (top PR AUC = 0.860). Additionally, we tested our models on an external validation cohort of 137 patients, and achieved good results in this case too (top PR AUC = 0.863), confirming the generalizability of our approach. Our results can have a huge impact on clinical settings, allowing physicians to forecast the survival of patients by sex, age, and septic episode number alone.

A Pilot Study of End-Tidal Carbon Dioxide in Prediction of Inhospital Cardiac Arrests

A validated means to predict inhospital cardiac arrest is lacking. The purpose of this study was to evaluate the changes in end-tidal carbon dioxide, as it correlates with the progression to inhospital cardiac arrest in ICU patients.

Development of Machine Learning Models to Validate a Medication Regimen Complexity Scoring Tool for Critically Ill Patients

INTRODUCTION

The Medication Regimen Complexity -Intensive Care Unit (MRC-ICU) is the first tool for measuring medication regimen complexity in critically ill patients. This study tested machine learning (ML) models to investigate the relationship between medication regimen complexity and patient outcomes.

METHODS

This study was a single-center, retrospective observational evaluation of 130 adults admitted to the medical ICU. The MRC-ICU score was utilized to improve the inpatient model's prediction accuracy. Three models were proposed: model I, demographic data without medication data; model II, demographic data and medication regimen complexity variables; and model III: demographic data and the MRC-ICU score. A total of 6 ML classifiers was developed: k-nearest neighbor (KNN), naïve Bayes (NB), random forest, support vector machine, neural network, and logistic classifier (LC). They were developed and tested using electronic health record data to predict inpatient mortality.

RESULTS

The results demonstrated that adding medication regimen complexity variables (model II) and the MRC-ICU score (model III) improved inpatient mortality prediction.. The LC outperformed the other classifiers (KNN and NB), with an overall accuracy of 83%, sensitivity (Se) of 87%, specificity of 67%, positive predictive value of 93%, and negative predictive value of 46%. The APACHE III score and the MRC-ICU score at the 24-hour interval were the 2 most important variables.

CONCLUSION AND RELEVANCE

Inclusion of the MRC-ICU score improved the prediction of patient outcomes on the previously established APACHE III score. This novel, proof-of-concept methodology shows promise for future application of the MRC-ICU scoring tool for patient outcome predictions.

Using Machine Learning to Make Predictions in Patients Who Fall

BACKGROUND

As the population ages, the incidence of traumatic falls has been increasing. We hypothesize that a machine learning algorithm can more accurately predict mortality after a fall compared with a standard logistic regression (LR) model based on immediately available admission data. Secondary objectives were to predict who would be discharged home and determine which variables had the largest effect on prediction.

METHODS

All patients who were admitted for fall between 2012 and 2017 at our level 1 trauma center were reviewed. Fourteen variables describing patient demographics, injury characteristics, and physiology were collected at the time of admission and were used for prediction modeling. Algorithms assessed included LR, decision tree classifier (DTC), and random forest classifier (RFC). Area under the receiver operating characteristic curve (AUC) values were calculated for each algorithm for mortality and discharge to home.

RESULTS

About 4725 patients met inclusion criteria. The mean age was 61 ± 20.5 y, Injury Severity Score 8 ± 7, length of stay 5.8 ± 7.6 d, intensive care unit length of stay 1.8± 5.2 d, and ventilator days 0.7 ± 4.2 d. The mortality rate was 3% and three times greater for elderly (aged 65 y and older) patients (5.0% versus 1.6%, P < 0.001). The AUC for predicting mortality for LR, DTC, and RFC was 0.78, 0.64, and 0.86, respectively. The AUC for predicting discharge to home for LR, DTC, and RFC was 0.72, 0.61, and 0.74, respectively. The top five variables that contribute to the prediction of mortality in descending order of importance are the Glasgow Coma Score (GCS) motor, GCS verbal, respiratory rate, GCS eye, and temperature.

CONCLUSIONS

RFC can accurately predict mortality and discharge home after a fall. This predictive model can be implemented at the time of patient arrival and may help identify candidates for targeted intervention as well as improve prognostication and resource utilization.

Demonstrating the consequences of learning missingness patterns in early warning systems for preventative health care: A novel simulation and solution

When using tree-based methods to develop predictive analytics and early warning systems for preventive healthcare, it is important to use an appropriate imputation method to prevent learning the missingness pattern. To demonstrate this, we developed a novel simulation that generated synthetic electronic health record data using a variational autoencoder with a custom loss function, which took into account the high missing rate of electronic health data. We showed that when tree-based methods learn missingness patterns (correlated with adverse events) in electronic health record data, this leads to decreased performance if the system is used in a new setting that has different missingness patterns. Performance is worst in this scenario when the missing rate between those with and without an adverse event is the greatest. We found that randomized and Bayesian regression imputation methods mitigate the issue of learning the missingness pattern for tree-based methods. We used this information to build a novel early warning system for predicting patient deterioration in general wards and telemetry units: PICTURE (Predicting Intensive Care Transfers and other UnfoReseen Events). To develop, tune, and test PICTURE, we used labs and vital signs from electronic health records of adult patients over four years (n = 133,089 encounters). We analyzed primary outcomes of unplanned intensive care unit transfer, emergency vasoactive medication administration, cardiac arrest, and death. We compared PICTURE with existing early warning systems and logistic regression at multiple levels of granularity. When analyzing PICTURE on the testing set using all observations within a hospital encounter (event rate = 3.4%), PICTURE had an area under the receiver operating characteristic curve (AUROC) of 0.83 and an adjusted (event rate = 4%) area under the precision-recall curve (AUPR) of 0.27, while the next best tested method-regularized logistic regression-had an AUROC of 0.80 and an adjusted AUPR of 0.22. To ensure system interpretability, we applied a state-of-the-art prediction explainer that provided a ranked list of features contributing most to the prediction. Though it is currently difficult to compare machine learning-based early warning systems, a rudimentary comparison with published scores demonstrated that PICTURE is on par with state-of-the-art machine learning systems. To facilitate more robust comparisons and development of early warning systems in the future, we have released our variational autoencoder's code and weights so researchers can (a) test their models on data similar to our institution and (b) make their own synthetic datasets.

Clinical Implementation of Predictive Models Embedded within Electronic Health Record Systems: A Systematic Review

Predictive analytics using electronic health record (EHR) data have rapidly advanced over the last decade. While model performance metrics have improved considerably, best practices for implementing predictive models into clinical settings for point-of-care risk stratification are still evolving. Here, we conducted a systematic review of articles describing predictive models integrated into EHR systems and implemented in clinical practice. We conducted an exhaustive database search and extracted data encompassing multiple facets of implementation. We assessed study quality and level of evidence. We obtained an initial 3393 articles for screening, from which a final set of 44 articles was included for data extraction and analysis. The most common clinical domains of implemented predictive models were related to thrombotic disorders/anticoagulation (25%) and sepsis (16%). The majority of studies were conducted in inpatient academic settings. Implementation challenges included alert fatigue, lack of training, and increased work burden on the care team. Of 32 studies that reported effects on clinical outcomes, 22 (69%) demonstrated improvement after model implementation. Overall, EHR-based predictive models offer promising results for improving clinical outcomes, although several gaps in the literature remain, and most study designs were observational. Future studies using randomized controlled trials may help improve the generalizability of findings.

Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study

Background

Successful integrations of machine learning into routine clinical care are exceedingly rare, and barriers to its adoption are poorly characterized in the literature.

Objective

This study aims to report a quality improvement effort to integrate a deep learning sepsis detection and management platform, Sepsis Watch, into routine clinical care.

Methods

In 2016, a multidisciplinary team consisting of statisticians, data scientists, data engineers, and clinicians was assembled by the leadership of an academic health system to radically improve the detection and treatment of sepsis. This report of the quality improvement effort follows the learning health system framework to describe the problem assessment, design, development, implementation, and evaluation plan of Sepsis Watch.

Results

Sepsis Watch was successfully integrated into routine clinical care and reshaped how local machine learning projects are executed. Frontline clinical staff were highly engaged in the design and development of the workflow, machine learning model, and application. Novel machine learning methods were developed to detect sepsis early, and implementation of the model required robust infrastructure. Significant investment was required to align stakeholders, develop trusting relationships, define roles and responsibilities, and to train frontline staff, leading to the establishment of 3 partnerships with internal and external research groups to evaluate Sepsis Watch.

Conclusions

Machine learning models are commonly developed to enhance clinical decision making, but successful integrations of machine learning into routine clinical care are rare. Although there is no playbook for integrating deep learning into clinical care, learnings from the Sepsis Watch integration can inform efforts to develop machine learning technologies at other health care delivery systems.

How good is our diagnostic intuition? Clinician prediction of bacteremia in critically ill children

Background

Clinical intuition and nonanalytic reasoning play a major role in clinical hypothesis generation; however, clinicians’ intuition about whether a critically ill child is bacteremic has not been explored. We endeavored to assess pediatric critical care clinicians’ ability to predict bacteremia and to evaluate what affected the accuracy of those predictions.

Methods

We conducted a retrospective review of clinicians’ responses to a sepsis screening tool (“Early Sepsis Detection Tool” or “ESDT”) over 6 months. The ESDT was completed during the initial evaluation of a possible sepsis episode. If a culture was ordered, they were asked to predict if the culture would be positive or negative. Culture results were compared to predictions for each episode as well as vital signs and laboratory data from the preceding 24 h.

Results

From January to July 2017, 266 ESDTs were completed. Of the 135 blood culture episodes, 15% of cultures were positive. Clinicians correctly predicted patients with bacteremia in 82% of cases, but the positive predictive value was just 28% as there was a tendency to overestimate the presence of bacteremia. The negative predictive value was 96%. The presence of bandemia, thrombocytopenia, and abnormal CRP were associated with increased likelihood of correct positive prediction.

Conclusions

Clinicians are accurate in predicting critically ill children whose blood cultures, obtained for symptoms of sepsis, will be negative. Clinicians frequently overestimate the presence of bacteremia. The combination of evidence-based practice guidelines and bedside judgment should be leveraged to optimize diagnosis of bacteremia.

Bloodstream Infections and Delayed Antibiotic Coverage Are Associated With Negative Hospital Outcomes in Hematopoietic Stem Cell Transplant Recipients

BACKGROUND

Bloodstream infections (BSIs) are common after hematopoietic stem cell transplantation (HSCT) and are associated with increased long-term morbidity and mortality. However, short-term outcomes related to BSI in this population remain unknown. More specifically, it is unclear whether choices related to empiric antimicrobials for potentially infected patients are associated with patient outcomes.

RESEARCH QUESTION

Are potential delays in appropriate antibiotics associated with hospital outcomes among HSCT recipients with BSI?

STUDY DESIGN AND METHODS

We conducted a retrospective cohort study at a large comprehensive inpatient academic cancer center between January 2014 and June 2017. We identified all admissions for HSCT and prior recipients of HSCT. We defined potential delay in appropriate antibiotics as > 24 h between positive blood culture results and the initial dose of an antimicrobial with activity against the pathogen.

RESULTS

We evaluated 2,751 hospital admissions from 1,086 patients. Of these admissions, 395 (14.4%) involved one or more BSIs. Of these 395 hospitalizations, 44 (11.1%) involved potential delays in appropriate antibiotics. The incidence of mortality was higher in BSI hospitalizations than in those without BSI (23% vs 4.5%; P < .001). In multivariable analysis, BSI was an independent predictor of mortality (OR, 8.14; 95% CI, 5.06-13.1; P < .001). Mortality was higher for admissions with potentially delayed appropriate antibiotics than for those with appropriate antibiotics (48% vs 20%; P < .001). Potential delay in antibiotics was also an independent predictor of mortality in multivariable analysis (OR, 13.8; 95% CI, 5.27-35.9; P < .001).

INTERPRETATION

BSIs were common and independently associated with increased morbidity and mortality. Delays in administration of appropriate antimicrobials were identified as an important factor in hospital morbidity and mortality. These findings may have important implications for our current practice of empiric antibiotic treatment in HSCT patients.

Artificial intelligence and computer simulation models in critical illness

Widespread implementation of electronic health records has led to the increased use of artificial intelligence (AI) and computer modeling in clinical medicine. The early recognition and treatment of critical illness are central to good outcomes but are made difficult by, among other things, the complexity of the environment and the often non-specific nature of the clinical presentation. Increasingly, AI applications are being proposed as decision supports for busy or distracted clinicians, to address this challenge. Data driven "associative" AI models are built from retrospective data registries with missing data and imprecise timing. Associative AI models lack transparency, often ignore causal mechanisms, and, while potentially useful in improved prognostication, have thus far had limited clinical applicability. To be clinically useful, AI tools need to provide bedside clinicians with actionable knowledge. Explicitly addressing causal mechanisms not only increases validity and replicability of the model, but also adds transparency and helps gain trust from the bedside clinicians for real world use of AI models in teaching and patient care.

Enhancing sepsis management through machine learning techniques: A review

Sepsis is a major public health problem and a leading cause of death in the world, where delay in the beginning of treatment, along with clinical guidelines non-adherence have been proved to be associated with higher mortality. Machine Learning is increasingly being adopted in developing innovative Clinical Decision Support Systems in many areas of medicine, showing a great potential for automatic prediction of diverse patient conditions, as well as assistance in clinical decision making. In this context, this work conducts a narrative review to provide an overview of how specific Machine Learning techniques can be used to improve sepsis management, discussing the main tasks addressed, the most popular methods and techniques, as well as the obtained results, in terms of both intelligent system accuracy and clinical outcomes improvement.

Clinician Perception of a Machine Learning-Based Early Warning System Designed to Predict Severe Sepsis and Septic Shock

OBJECTIVE

To assess clinician perceptions of a machine learning-based early warning system to predict severe sepsis and septic shock (Early Warning System 2.0).

DESIGN

Prospective observational study.

SETTING

Tertiary teaching hospital in Philadelphia, PA.

PATIENTS

Non-ICU admissions November-December 2016.

INTERVENTIONS

During a 6-week study period conducted 5 months after Early Warning System 2.0 alert implementation, nurses and providers were surveyed twice about their perceptions of the alert's helpfulness and impact on care, first within 6 hours of the alert, and again 48 hours after the alert.

MEASUREMENTS AND MAIN RESULTS

For the 362 alerts triggered, 180 nurses (50% response rate) and 107 providers (30% response rate) completed the first survey. Of these, 43 nurses (24% response rate) and 44 providers (41% response rate) completed the second survey. Few (24% nurses, 13% providers) identified new clinical findings after responding to the alert. Perceptions of the presence of sepsis at the time of alert were discrepant between nurses (13%) and providers (40%). The majority of clinicians reported no change in perception of the patient's risk for sepsis (55% nurses, 62% providers). A third of nurses (30%) but few providers (9%) reported the alert changed management. Almost half of nurses (42%) but less than a fifth of providers (16%) found the alert helpful at 6 hours.

CONCLUSIONS

In general, clinical perceptions of Early Warning System 2.0 were poor. Nurses and providers differed in their perceptions of sepsis and alert benefits. These findings highlight the challenges of achieving acceptance of predictive and machine learning-based sepsis alerts.

Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission: a prospective multicentre clinical outcomes evaluation of real-world patient data from US hospitals

BACKGROUND

Severe sepsis and septic shock are among the leading causes of death in the USA. While early prediction of severe sepsis can reduce adverse patient outcomes, sepsis remains one of the most expensive conditions to diagnose and treat.

OBJECTIVE

The purpose of this study was to evaluate the effect of a machine learning algorithm for severe sepsis prediction on in-hospital mortality, hospital length of stay and 30-day readmission.

DESIGN

Prospective clinical outcomes evaluation.

SETTING

Evaluation was performed on a multiyear, multicentre clinical data set of real-world data containing 75 147 patient encounters from nine hospitals across the continental USA, ranging from community hospitals to large academic medical centres.

PARTICIPANTS

Analyses were performed for 17 758 adult patients who met two or more systemic inflammatory response syndrome criteria at any point during their stay ('sepsis-related' patients).

INTERVENTIONS

Machine learning algorithm for severe sepsis prediction.

OUTCOME MEASURES

In-hospital mortality, length of stay and 30-day readmission rates.

RESULTS

Hospitals saw an average 39.5% reduction of in-hospital mortality, a 32.3% reduction in hospital length of stay and a 22.7% reduction in 30-day readmission rate for sepsis-related patient stays when using the machine learning algorithm in clinical outcomes analysis.

CONCLUSIONS

Reductions of in-hospital mortality, hospital length of stay and 30-day readmissions were observed in real-world clinical use of the machine learning-based algorithm. The predictive algorithm may be successfully used to improve sepsis-related outcomes in live clinical settings.

TRIAL REGISTRATION NUMBER

NCT03960203.

Artificial Intelligence in the Intensive Care Unit

This article is one of ten reviews selected from the Annual Update in Intensive Care and Emergency Medicine 2020. Other selected articles can be found online at https://www.biomedcentral.com/collections/annualupdate2020. Further information about the Annual Update in Intensive Care and Emergency Medicine is available from http://www.springer.com/series/8901.

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy

PURPOSE

Early clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis.

METHODS

A systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance.

RESULTS

After screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68-0.99 in the ICU, to 0.96-0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance.

CONCLUSION

This systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside.