Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU

Can Prehospital Data Improve Early Identification of Sepsis in Emergency Department? An Integrative Review of Machine Learning Approaches

BACKGROUND

Sepsis is associated with high mortality, especially during the novel coronavirus disease 2019 (COVID-19) pandemic. Along with high monetary health care costs for sepsis treatment, there is a lasting impact on lives of sepsis survivors and their caregivers. Early identification is necessary to reduce the negative impact of sepsis and to improve patient outcomes. Prehospital data are among the earliest information collected by health care systems. Using these untapped sources of data in machine learning (ML)-based approaches can identify patients with sepsis earlier in emergency department (ED).

OBJECTIVES

This integrative literature review aims to discuss the importance of utilizing prehospital data elements in ED, summarize their current use in developing ML-based prediction models, and specifically identify those data elements that can potentially contribute to early identification of sepsis in ED when used in ML-based approaches.

METHOD

Literature search strategy includes following two separate searches: (1) use of prehospital data in ML models in ED; and (2) ML models that are developed specifically to predict/detect sepsis in ED. In total, 24 articles are used in this review.

RESULTS

A summary of prehospital data used to identify time-sensitive conditions earlier in ED is provided. Literature related to use of ML models for early identification of sepsis in ED is limited and no studies were found related to ML models using prehospital data in prediction/early identification of sepsis in ED. Among those using ED data, ML models outperform traditional statistical models. In addition, the use of the free-text elements and natural language processing (NLP) methods could result in better prediction of sepsis in ED.

CONCLUSION

This study reviews the use of prehospital data in early decision-making in ED and suggests that researchers utilize such data elements for prediction/early identification of sepsis in ML-based approaches.

Clinical Characteristics and Death Risk Factors of Severe Sepsis in Children

Sepsis is a systemic inflammatory response syndrome caused by viral infection. The circulatory dysfunction caused by sepsis is also called septic shock or septic shock. The main characteristics are rapid onset, rapid changes, and involvement. Multiple organs in the body make diagnosis difficult, which seriously threatens the survival of patients. As many as one million people worldwide die every year because of SIRS, it is also the leading cause of death among children in hospital ICUs. This article is aimed at studying the clinical characteristics of severe sepsis in children and the risk factors for death. Based on the analysis of the pathogenesis of sepsis and the treatment of septic shock, 65 cases of children with PICU sepsis admitted to a hospital were selected. Data, to study its clinical characteristics and risk factors for death. The results of the study showed that despite the interaction among the removal factors of the three indexes of serum lactic acid value, PCIS level, and the number of organs involved in MODS, they are still related to the mortality of children with severe sepsis.

Evaluating machine learning models for sepsis prediction: A systematic review of methodologies

Studies for sepsis prediction using machine learning are developing rapidly in medical science recently. In this review, we propose a set of new evaluation criteria and reporting standards to assess 21 qualified machine learning models for quality analysis based on PRISMA. Our assessment shows that (1.) the definition of sepsis is not consistent among the studies; (2.) data sources and data preprocessing methods, machine learning models, feature engineering, and inclusion types vary widely among the studies; (3.) the closer to the onset of sepsis, the higher the value of AUROC is; (4.) the improvement in AUROC is primarily due to using machine learning as a feature engineering tool; (5.) deep neural networks coupled with Sepsis-3 diagnostic criteria tend to yield better results on the time series data collected from patients with sepsis. The new evaluation criteria and reporting standards will facilitate the development of improved machine learning models for clinical applications.

•

New evaluation/reporting standard for sepsis prediction machine learning models

•

Major limitations in the current models for sepsis prediction have been identified

•

We strongly suggest using machine learning as a feature engineering tool

•

Recommending multilayer neural networks and Sepsis 3.0 for yield better result

Multi-Task Learning with Recurrent Neural Networks for ARDS Prediction using only EHR Data: Model Development and Validation Study (Preprint)

Background

Acute respiratory distress syndrome (ARDS) is a condition that is often considered to have broad and subjective diagnostic criteria and is associated with significant mortality and morbidity. Early and accurate prediction of ARDS and related conditions such as hypoxemia and sepsis could allow timely administration of therapies, leading to improved patient outcomes.

Objective

The aim of this study is to perform an exploration of how multilabel classification in the clinical setting can take advantage of the underlying dependencies between ARDS and related conditions to improve early prediction of ARDS in patients.

Methods

The electronic health record data set included 40,703 patient encounters from 7 hospitals from April 20, 2018, to March 17, 2021. A recurrent neural network (RNN) was trained using data from 5 hospitals, and external validation was conducted on data from 2 hospitals. In addition to ARDS, 12 target labels for related conditions such as sepsis, hypoxemia, and COVID-19 were used to train the model to classify a total of 13 outputs. As a comparator, XGBoost models were developed for each of the 13 target labels. Model performance was assessed using the area under the receiver operating characteristic curve. Heat maps to visualize attention scores were generated to provide interpretability to the neural networks. Finally, cluster analysis was performed to identify potential phenotypic subgroups of patients with ARDS.

Results

The single RNN model trained to classify 13 outputs outperformed the individual XGBoost models for ARDS prediction, achieving an area under the receiver operating characteristic curve of 0.842 on the external test sets. Models trained on an increasing number of tasks resulted in improved performance. Earlier prediction of ARDS nearly doubled the rate of in-hospital survival. Cluster analysis revealed distinct ARDS subgroups, some of which had similar mortality rates but different clinical presentations.

Conclusions

The RNN model presented in this paper can be used as an early warning system to stratify patients who are at risk of developing one of the multiple risk outcomes, hence providing practitioners with the means to take early action.

Prediction of Bacteremia Based on 12-Year Medical Data Using a Machine Learning Approach: Effect of Medical Data by Extraction Time

Early detection of bacteremia is important to prevent antibiotic abuse. Therefore, we aimed to develop a clinically applicable bacteremia prediction model using machine learning technology. Data from two tertiary medical centers’ electronic medical records during a 12-year-period were extracted. Multi-layer perceptron (MLP), random forest, and gradient boosting algorithms were applied for machine learning analysis. Clinical data within 12 and 24 hours of blood culture were analyzed and compared. Out of 622,771 blood cultures, 38,752 episodes of bacteremia were identified. In MLP with 128 hidden layer nodes, the area under the receiver operating characteristic curve (AUROC) of the prediction performance in 12- and 24-h data models was 0.762 (95% confidence interval (CI); 0.7617–0.7623) and 0.753 (95% CI; 0.7520–0.7529), respectively. AUROC of causative-pathogen subgroup analysis predictive value for Acinetobacter baumannii bacteremia was the highest at 0.839 (95% CI; 0.8388–0.8394). Compared to primary bacteremia, AUROC of sepsis caused by pneumonia was highest. Predictive performance of bacteremia was superior in younger age groups. Bacteremia prediction using machine learning technology appeared possible for acute infectious diseases. This model was more suitable especially to pneumonia caused by Acinetobacter baumannii. From the 24-h blood culture data, bacteremia was predictable by substituting only the continuously variable values.

Implementation and evaluation of sepsis surveillance and decision support in medical ICU and emergency department

OBJECTIVE

To improve the timely diagnosis and treatment of sepsis many institutions implemented automated sepsis alerts. Poor specificity, time delays, and a lack of actionable information lead to limited adoption by bedside clinicians and no change in practice or clinical outcomes. We aimed to compare sepsis care compliance before and after a multi-year implementation of a sepsis surveillance coupled with decision support in a tertiary care center.

DESIGN

Single center before and after study.

SETTING

Large academic Medical Intensive Care Unit (MICU) and Emergency Department (ED).

POPULATION

Patients 18 years of age or older admitted to *** Hospital MICU and ED from 09/4/2011 to 05/01/2018 with severe sepsis or septic shock.

INTERVENTIONS

Electronic medical record-based sepsis surveillance system augmented by clinical decision support and completion feedback.

MEASUREMENTS AND MAIN RESULTS

There were 1950 patients admitted to the MICU with the diagnosis of severe sepsis or septic shock during the study period. The baseline characteristics were similar before (N = 854) and after (N = 1096) implementation of sepsis surveillance. The performance of the alert was modest with a sensitivity of 79.9%, specificity of 76.9%, positive predictive value (PPV) 27.9%, and negative predictive value (NPV) 97.2%. There were 3424 unique alerts and 1131 confirmed sepsis patients after the sniffer implementation. During the study period average care bundle compliance was higher; however after taking into account improvements in compliance leading up to the intervention, there was no association between intervention and improved care bundle compliance (Odds ratio: 1.16; 95% CI: 0.71 to 1.89; p-value 0.554). Similarly, the intervention was not associated with improvement in hospital mortality (Odds ratio: 1.55; 95% CI: 0.95 to 2.52; p-value: 0.078).

CONCLUSIONS

A sepsis surveillance system incorporating decision support or completion feedback was not associated with improved sepsis care and patient outcomes.

Clinical Value of the Lactate/Albumin Ratio and Lactate/Albumin Ratio × Age Score in the Assessment of Prognosis in Patients With Sepsis

Objective: To examine the clinical significance of the blood lactate (Lac)/serum albumin (Alb) ratio and the Lac/Alb × age score for assessing the severity and prognosis of patients with sepsis.

Methods: A total of 8,029 patients with sepsis, aged >18 years were enrolled between June 2001 to October 2012 from the latest version of the Medical Information Mart for Intensive Care III (MIMIC-III v.1.4). The general data of the patients were obtained from hospital records and included gender, age, body mass index (BMI), laboratory indices, the sequential organ failure assessment (SOFA) score, and simplified acute physiology score II (SAPS II). The patients were graded and scored according to their age and then divided into a survival or death group based on their prognosis. The Lac/Alb ratio after ICU admission was calculated and compared between the two groups. The risk factors for death in patients with sepsis were determined using multivariate logistic regression analysis, while mortality was examined using receiver operating characteristic (ROC) curve and survival curve plots. Finally, the values of the Lac/Alb ratio and Lac/Alb × age score for assessing prognosis of patients with sepsis were analyzed and compared.

Results: After items with default values were excluded, a total of 4,555 patients with sepsis were enrolled (2,526 males and 2,029 females). 2,843 cases were classified as the death group and 1,712 cases in the survival group. (1) The mean age, BMI, SOFA and SAPS II scores were higher in the death group than those in the survival group. Significant differences in baseline data between the two groups were also observed. (2) The patients in the death group were divided further into four subgroups according to the quartile of the Lac/Alb ratio from low to high. Comparison of the four subgroups showed that the death rate rose with an increase in the Lac/Alb ratio, while analysis of the survival curve revealed that patients with a higher Lac/Alb ratio had a worse prognosis. (3) Multivariate logistic regression analysis showed that age ≥ 60 years, overweight (BMI ≥ 24 kg/m²), Lac/Alb ratio ≥ 0.16, SOFA score ≥ 2 points, and SAPS II ≥ 40 points were independent risk factors for death in patients with septic. (4) ROC curve analysis indicated that the SAPS II, Lac/Alb x age score, SOFA, and Lac/Alb ratio were the best predictors of death in patients with sepsis. The Lac/Alb × age score was characterized by its simple acquisition and ability to quickly analyze the prognosis of patients.

Conclusion: (1)A high Lac/Alb ratio is an independent risk factor for death in patients with sepsis. (2) Although the prognosis of sepsis can be accurately and comprehensively assessed by multi-dimensional analysis of multiple indices, the Lac/Alb×age score is more accurate and convenient for providing a general assessment of prognosis, so is worthy of further clinical recognition.

Applying artificial neural network for early detection of sepsis with intentionally preserved highly missing real-world data for simulating clinical situation

Purpose

Some predictive systems using machine learning models have been developed to predict sepsis; however, they were mostly built with a low percent of missing values, which does not correspond with the actual clinical situation. In this study, we developed a machine learning model with a high rate of missing and erroneous data to enable prediction under missing, noisy, and erroneous inputs, as in the actual clinical situation.

Materials and methods

The proposed artificial neural network model was implemented using the MATLAB ANN toolbox, based on stochastic gradient descent. The dataset was collected over the past decade with approval from the appropriate institutional review boards, and the sepsis status was identified and labeled using Sepsis-3 clinical criteria. The imputation method was built by last observation carried forward and mean value, aimed to simulate clinical situation.

Results

The mean area under the receiver operating characteristic (ROC) curve (AUC) of classifying sepsis and nonsepsis patients was 0.82 and 0.786 at 0 h and 40 h prior to onset, respectively. The highest model performance was found for one-hourly data, demonstrating that our ANN model can perform adequately with limited hourly data provided.

Conclusions

Our model has the moderate ability to predict sepsis up to 40 h in advance under simulated clinical situation with real-world data.

The impact of recency and adequacy of historical information on sepsis predictions using machine learning

Sepsis is a major public and global health concern. Every hour of delay in detecting sepsis significantly increases the risk of death, highlighting the importance of accurately predicting sepsis in a timely manner. A growing body of literature has examined developing new or improving the existing machine learning (ML) approaches for timely and accurate predictions of sepsis. This study contributes to this literature by providing clear insights regarding the role of the recency and adequacy of historical information in predicting sepsis using ML. To this end, we implemented a deep learning model using a bidirectional long short-term memory (BiLSTM) algorithm and compared it with six other ML algorithms based on numerous combinations of the prediction horizons (to capture information recency) and observation windows (to capture information adequacy) using different measures of predictive performance. Our results indicated that the BiLSTM algorithm outperforms all other ML algorithms and provides a great separability of the predicted risk of sepsis among septic versus non-septic patients. Moreover, decreasing the prediction horizon (in favor of information recency) always boosts the predictive performance; however, the impact of expanding the observation window (in favor of information adequacy) depends on the prediction horizon and the purpose of prediction. More specifically, when the prediction is responsive to the positive label (i.e., Sepsis), increasing historical data improves the predictive performance when the prediction horizon is short-moderate.

Machine Learning Model to Identify Sepsis Patients in the Emergency Department: Algorithm Development and Validation

Accurate stratification of sepsis can effectively guide the triage of patient care and shared decision making in the emergency department (ED). However, previous research on sepsis identification models focused mainly on ICU patients, and discrepancies in model performance between the development and external validation datasets are rarely evaluated. The aim of our study was to develop and externally validate a machine learning model to stratify sepsis patients in the ED. We retrospectively collected clinical data from two geographically separate institutes that provided a different level of care at different time periods. The Sepsis-3 criteria were used as the reference standard in both datasets for identifying true sepsis cases. An eXtreme Gradient Boosting (XGBoost) algorithm was developed to stratify sepsis patients and the performance of the model was compared with traditional clinical sepsis tools; quick Sequential Organ Failure Assessment (qSOFA) and Systemic Inflammatory Response Syndrome (SIRS). There were 8296 patients (1752 (21%) being septic) in the development and 1744 patients (506 (29%) being septic) in the external validation datasets. The mortality of septic patients in the development and validation datasets was 13.5% and 17%, respectively. In the internal validation, XGBoost achieved an area under the receiver operating characteristic curve (AUROC) of 0.86, exceeding SIRS (0.68) and qSOFA (0.56). The performance of XGBoost deteriorated in the external validation (the AUROC of XGBoost, SIRS and qSOFA was 0.75, 0.57 and 0.66, respectively). Heterogeneity in patient characteristics, such as sepsis prevalence, severity, age, comorbidity and infection focus, could reduce model performance. Our model showed good discriminative capabilities for the identification of sepsis patients and outperformed the existing sepsis identification tools. Implementation of the ML model in the ED can facilitate timely sepsis identification and treatment. However, dataset discrepancies should be carefully evaluated before implementing the ML approach in clinical practice. This finding reinforces the necessity for future studies to perform external validation to ensure the generalisability of any developed ML approaches.

Timing of antibiotic therapy in the ICU

Severe or life threatening infections are common among patients in the intensive care unit (ICU). Most infections in the ICU are bacterial or fungal in origin and require antimicrobial therapy for clinical resolution. Antibiotics are the cornerstone of therapy for infected critically ill patients. However, antibiotics are often not optimally administered resulting in less favorable patient outcomes including greater mortality. The timing of antibiotics in patients with life threatening infections including sepsis and septic shock is now recognized as one of the most important determinants of survival for this population. Individuals who have a delay in the administration of antibiotic therapy for serious infections can have a doubling or more in their mortality. Additionally, the timing of an appropriate antibiotic regimen, one that is active against the offending pathogens based on in vitro susceptibility, also influences survival. Thus not only is early empiric antibiotic administration important but the selection of those agents is crucial as well. The duration of antibiotic infusions, especially for β-lactams, can also influence antibiotic efficacy by increasing antimicrobial drug exposure for the offending pathogen. However, due to mounting antibiotic resistance, aggressive antimicrobial de-escalation based on microbiology results is necessary to counterbalance the pressures of early broad-spectrum antibiotic therapy. In this review, we examine time related variables impacting antibiotic optimization as it relates to the treatment of life threatening infections in the ICU. In addition to highlighting the importance of antibiotic timing in the ICU we hope to provide an approach to antimicrobials that also minimizes the unnecessary use of these agents. Such approaches will increasingly be linked to advances in molecular microbiology testing and artificial intelligence/machine learning. Such advances should help identify patients needing empiric antibiotic therapy at an earlier time point as well as the specific antibiotics required in order to avoid unnecessary administration of broad-spectrum antibiotics.

Machine learning identification of specific changes in myeloid cell phenotype during bloodstream infections

The early identification of bacteremia is critical for ensuring appropriate treatment of nosocomial infections in intensive care unit (ICU) patients. The aim of this study was to use flow cytometric data of myeloid cells as a biomarker of bloodstream infection (BSI). An eight-color antibody panel was used to identify seven monocyte and two dendritic cell subsets. In the learning cohort, immunophenotyping was applied to (1) control subjects, (2) postoperative heart surgery patients, as a model of noninfectious inflammatory responses, and (3) blood culture-positive patients. Of the complex changes in the myeloid cell phenotype, a decrease in myeloid and plasmacytoid dendritic cell numbers, increase in CD14⁺CD16⁺ inflammatory monocyte numbers, and upregulation of neutrophils CD64 and CD123 expression were prominent in BSI patients. An extreme gradient boosting (XGBoost) algorithm called the “infection detection and ranging score” (iDAR), ranging from 0 to 100, was developed to identify infection-specific changes in 101 phenotypic variables related to neutrophils, monocytes and dendritic cells. The tenfold cross-validation achieved an area under the receiver operating characteristic (AUROC) of 0.988 (95% CI 0.985–1) for the detection of bacteremic patients. In an out-of-sample, in-house validation, iDAR achieved an AUROC of 0.85 (95% CI 0.71–0.98) in differentiating localized from bloodstream infection and 0.95 (95% CI 0.89–1) in discriminating infected from noninfected ICU patients. In conclusion, a machine learning approach was used to translate the changes in myeloid cell phenotype in response to infection into a score that could identify bacteremia with high specificity in ICU patients.

Application of a 72 h National Early Warning Score and Incorporation with Sequential Organ Failure Assessment for Predicting Sepsis Outcomes and Risk Stratification in an Intensive Care Unit: A Derivation and Validation Cohort Study

We investigated the best timing for using the National Early Warning Score 2 (NEWS2) for predicting sepsis outcomes and whether combining the NEWS2 and the Sequential Organ Failure Assessment (SOFA) was applicable for mortality risk stratification in intensive care unit (ICU) patients with severe sepsis. All adult patients who met the Third International Consensus Definitions for Sepsis and Septic Shock criteria between August 2013 and January 2017 with complete clinical parameters and laboratory data were enrolled as a derivation cohort. The primary outcomes were the 7-, 14-, 21-, and 28-day mortalities. Furthermore, another group of patients under the same setting between January 2020 and March 2020 were also enrolled as a validation cohort. In the derivation cohort, we included 699 consecutive adult patients. The 72 h NEWS2 had good discrimination for predicting 7-, 14-, 21-, and 28-day mortalities (AUC: 0.780, 0.724, 0.700, and 0.667, respectively) and was not inferior to the SOFA (AUC: 0.740, 0.680, 0.684, and 0.677, respectively). With the new combined NESO tool, the hazard ratio was 1.854 (1.203-2.950) for the intermediate-risk group and 6.810 (3.927-11.811) for the high-risk group relative to the low-risk group. This finding was confirmed in the validation cohort using a separated survival curve for 28-day mortality. The 72 h NEWS2 alone was non-inferior to the admission SOFA or day 3 SOFA for predicting sepsis outcomes. The NESO tool was found to be useful for 7-, 14-, 21-, and 28-day mortality risk stratification in patients with severe sepsis.

Artificial intelligence sepsis prediction algorithm learns to say "I don't know"

Sepsis is a leading cause of morbidity and mortality worldwide. Early identification of sepsis is important as it allows timely administration of potentially life-saving resuscitation and antimicrobial therapy. We present COMPOSER (COnformal Multidimensional Prediction Of SEpsis Risk), a deep learning model for the early prediction of sepsis, specifically designed to reduce false alarms by detecting unfamiliar patients/situations arising from erroneous data, missingness, distributional shift and data drifts. COMPOSER flags these unfamiliar cases as indeterminate rather than making spurious predictions. Six patient cohorts (515,720 patients) curated from two healthcare systems in the United States across intensive care units (ICU) and emergency departments (ED) were used to train and externally and temporally validate this model. In a sequential prediction setting, COMPOSER achieved a consistently high area under the curve (AUC) (ICU: 0.925-0.953; ED: 0.938-0.945). Out of over 6 million prediction windows roughly 20% and 8% were identified as indeterminate amongst non-septic and septic patients, respectively. COMPOSER provided early warning within a clinically actionable timeframe (ICU: 12.2 [3.2 22.8] and ED: 2.1 [0.8 4.5] hours prior to first antibiotics order) across all six cohorts, thus allowing for identification and prioritization of patients at high risk for sepsis.

Metagenomic evidence for a polymicrobial signature of sepsis

Our understanding of the host component of sepsis has made significant progress. However, detailed study of the microorganisms causing sepsis, either as single pathogens or microbial assemblages, has received far less attention. Metagenomic data offer opportunities to characterize the microbial communities found in septic and healthy individuals. In this study we apply gradient-boosted tree classifiers and a novel computational decontamination technique built upon SHapley Additive exPlanations (SHAP) to identify microbial hallmarks which discriminate blood metagenomic samples of septic patients from that of healthy individuals. Classifiers had high performance when using the read assignments to microbial genera [area under the receiver operating characteristic (AUROC=0.995)], including after removal of species ‘culture-confirmed’ as the cause of sepsis through clinical testing (AUROC=0.915). Models trained on single genera were inferior to those employing a polymicrobial model and we identified multiple co-occurring bacterial genera absent from healthy controls. While prevailing diagnostic paradigms seek to identify single pathogens, our results point to the involvement of a polymicrobial community in sepsis. We demonstrate the importance of the microbial component in characterising sepsis, which may offer new biological insights into the aetiology of sepsis, and ultimately support the development of clinical diagnostic or even prognostic tools.

Cardiogenic shock and machine learning: A systematic review on prediction through clinical decision support softwares

BACKGROUND AND AIM

Cardiogenic shock (CS) withholds a significantly high mortality rate between 40% and 60% despite advances in diagnosis and medical/surgical intervention. To date, machine learning (ML) is being implemented to integrate numerous data to optimize early diagnostic predictions and suggest clinical courses. This systematic review summarizes the area under the curve (AUC) receiver operating characteristics (ROCs) accuracy for the early prediction of CS.

METHODS

A systematic review was conducted within databases of PubMed, ScienceDirect, Clinical Key/MEDLINE, Embase, GoogleScholar, and Cochrane. Cohort studies that assessed the accuracy of early detection of CS using ML software were included. Data extraction was focused on AUC-ROC values directed towards the early detection of CS.

RESULTS

A total of 943 studies were included for systematic review. From the reviewed studies, 2.2% (N = 21) evaluated patient outcomes, of which 14.3% (N = 3) were assessed. The collective patient cohort (N = 698) consisted of 314 (45.0%) females, with an average age and body mass index of 64.1 years and 28.1 kg/m² , respectively. Collectively, 159 (22.8%) mortalities were reported following early CS detection. Altogether, the AUC-ROC value was 0.82 (α = .05), deeming it of superb sensitivity and specificity.

CONCLUSIONS

From the present comprehensively gathered data, this study accounts the use of ML software for the early detection of CS in a clinical setting as a valid tool to predict patients at risk of CS. The complexity of ML and its parallel lack of clinical evidence implies that further prospective randomized control trials are needed to draw definitive conclusions before standardizing the use of these technologies.

BRIEF SUMMARY

The catastrophic risk of developing CS continues to be a concern in the management of critical cardiac care. The use of ML predictive models have the potential to provide the accurate and necessary feedback for the early detection and proper management of CS. This systematic review summarizes the AUC-ROCs accuracy for the early prediction of CS.

Artificial Intelligence and Liability in Medicine: Balancing Safety and Innovation

Policy Points With increasing integration of artificial intelligence and machine learning in medicine, there are concerns that algorithm inaccuracy could lead to patient injury and medical liability. While prior work has focused on medical malpractice, the artificial intelligence ecosystem consists of multiple stakeholders beyond clinicians. Current liability frameworks are inadequate to encourage both safe clinical implementation and disruptive innovation of artificial intelligence. Several policy options could ensure a more balanced liability system, including altering the standard of care, insurance, indemnification, special/no-fault adjudication systems, and regulation. Such liability frameworks could facilitate safe and expedient implementation of artificial intelligence and machine learning in clinical care.

Deep Learning for Identification of Acute Illness and Facial Cues of Illness

Background: The inclusion of facial and bodily cues (clinical gestalt) in machine learning (ML) models improves the assessment of patients' health status, as shown in genetic syndromes and acute coronary syndrome. It is unknown if the inclusion of clinical gestalt improves ML-based classification of acutely ill patients. As in previous research in ML analysis of medical images, simulated or augmented data may be used to assess the usability of clinical gestalt. Objective: To assess whether a deep learning algorithm trained on a dataset of simulated and augmented facial photographs reflecting acutely ill patients can distinguish between healthy and LPS-infused, acutely ill individuals. Methods: Photographs from twenty-six volunteers whose facial features were manipulated to resemble a state of acute illness were used to extract features of illness and generate a synthetic dataset of acutely ill photographs, using a neural transfer convolutional neural network (NT-CNN) for data augmentation. Then, four distinct CNNs were trained on different parts of the facial photographs and concatenated into one final, stacked CNN which classified individuals as healthy or acutely ill. Finally, the stacked CNN was validated in an external dataset of volunteers injected with lipopolysaccharide (LPS). Results: In the external validation set, the four individual feature models distinguished acutely ill patients with sensitivities ranging from 10.5% (95% CI, 1.3-33.1% for the skin model) to 89.4% (66.9-98.7%, for the nose model). Specificity ranged from 42.1% (20.3-66.5%) for the nose model and 94.7% (73.9-99.9%) for skin. The stacked model combining all four facial features achieved an area under the receiver characteristic operating curve (AUROC) of 0.67 (0.62-0.71) and distinguished acutely ill patients with a sensitivity of 100% (82.35-100.00%) and specificity of 42.11% (20.25-66.50%). Conclusion: A deep learning algorithm trained on a synthetic, augmented dataset of facial photographs distinguished between healthy and simulated acutely ill individuals, demonstrating that synthetically generated data can be used to develop algorithms for health conditions in which large datasets are difficult to obtain. These results support the potential of facial feature analysis algorithms to support the diagnosis of acute illness.

Effects of comprehensive nursing on negative emotion and prognosis of patients with sepsis

OBJECTIVE

To explore the effect of comprehensive nursing on negative emotion and prognosis of patients with sepsis.

METHODS

As a prospective study, 104 patients with sepsis were randomized into the observation group (n=52) and the control group (n=52). The patients in the control group underwent routine nursing, whereas the patients in the observation group underwent comprehensive nursing care in addition to routine nursing. The level of negative emotions, patients' prognosis, quality of life (QOL), Acute Physiology and Chronic Health Evaluation (APACHE) ll score, clinical indicators, and patient satisfaction with nursing were compared between the two groups.

RESULTS

Compared with the control group, the observation group had lower Self-Rating Anxiety Scale score, Self-Rating Depression Scale score, and APACHE ll score (all P<0.001). The scores of physical functioning, general health perceptions, social role functioning, emotional role functioning, and mental health of the observation group were all higher than those of the control group (all P<0.01). The duration of mechanical ventilation, hospitalization expenses, and the length of stay in the Intensive Care Unit (ICU) in the observation group were lower than those in the control group (all P<0.01). Moreover, the observation group had a lower total incidence of shock, multiple organ dysfunction syndrome, and death and higher patient satisfaction with the nursing care than the control group (all P<0.05).

CONCLUSION

Comprehensive nursing care can alleviate anxiety and depression in patients with sepsis and can improve the prognosis and QOL of patients. Also, it can shorten the length of stay in the ICU, lower treatment costs, and improve patient satisfaction; all of which can be recommended for clinical application.

The Surviving Sepsis Campaign: research priorities for the administration, epidemiology, scoring and identification of sepsis

Objective

To identify priorities for administrative, epidemiologic and diagnostic research in sepsis.

Design

As a follow-up to a previous consensus statement about sepsis research, members of the Surviving Sepsis Campaign Research Committee, representing the European Society of Intensive Care Medicine and the Society of Critical Care Medicine addressed six questions regarding care delivery, epidemiology, organ dysfunction, screening, identification of septic shock, and information that can predict outcomes in sepsis.

Methods

Six questions from the Scoring/Identification and Administration sections of the original Research Priorities publication were explored in greater detail to better examine the knowledge gaps and rationales for questions that were previously identified through a consensus process.

Results

The document provides a framework for priorities in research to address the following questions: (1) What is the optimal model of delivering sepsis care?; (2) What is the epidemiology of sepsis susceptibility and response to treatment?; (3) What information identifies organ dysfunction?; (4) How can we screen for sepsis in various settings?; (5) How do we identify septic shock?; and (6) What in-hospital clinical information is associated with important outcomes in patients with sepsis?

Conclusions

There is substantial knowledge of sepsis epidemiology and ways to identify and treat sepsis patients, but many gaps remain. Areas of uncertainty identified in this manuscript can help prioritize initiatives to improve an understanding of individual patient and demographic heterogeneity with sepsis and septic shock, biomarkers and accurate patient identification, organ dysfunction, and ways to improve sepsis care.

A Locally Optimized Data-Driven Tool to Predict Sepsis-Associated Vasopressor Use in the ICU

OBJECTIVES

To train a model to predict vasopressor use in ICU patients with sepsis and optimize external performance across hospital systems using domain adaptation, a transfer learning approach.

DESIGN

Observational cohort study.

SETTING

Two academic medical centers from January 2014 to June 2017.

PATIENTS

Data were analyzed from 14,512 patients (9,423 at the development site and 5,089 at the validation site) who were admitted to an ICU and met Center for Medicare and Medicaid Services definition of severe sepsis either before or during the ICU stay. Patients were excluded if they never developed sepsis, if the ICU length of stay was less than 8 hours or more than 20 days or if they developed shock up to the first 4 hours of ICU admission.

MEASUREMENTS AND MAIN RESULTS

Forty retrospectively collected features from the electronic medical records of adult ICU patients at the development site (four hospitals) were used as inputs for a neural network Weibull-Cox survival model to derive a prediction tool for future need of vasopressors. Domain adaptation updated parameters to optimize model performance in the validation site (two hospitals), a different healthcare system over 2,000 miles away. The cohorts at both sites were randomly split into training and testing sets (80% and 20%, respectively). When applied to the test set in the development site, the model predicted vasopressor use 4-24 hours in advance with an area under the receiver operator characteristic curve, specificity, and positive predictive value ranging from 0.80 to 0.81, 56.2% to 61.8%, and 5.6% to 12.1%, respectively. Domain adaptation improved performance of the model to predict vasopressor use within 4 hours at the validation site (area under the receiver operator characteristic curve 0.81 [CI, 0.80-0.81] from 0.77 [CI, 0.76-0.77], p < 0.01; specificity 59.7% [CI, 58.9-62.5%] from 49.9% [CI, 49.5-50.7%], p < 0.01; positive predictive value 8.9% [CI, 8.5-9.4%] from 7.3 [7.1-7.4%], p < 0.01).

CONCLUSIONS

Domain adaptation improved performance of a model predicting sepsis-associated vasopressor use during external validation.

Machine Learning for Early Warning of Septic Shock in Children With Hematological Malignancies Accompanied by Fever or Neutropenia: A Single Center Retrospective Study

OBJECTIVES

The purpose of this article was to establish and validate clinically applicable septic shock early warning model (SSEW model) that can identify septic shock in hospitalized children with onco-hematological malignancies accompanied with fever or neutropenia.

METHODS

Data from EMRs were collected from hospitalized pediatric patients with hematological and oncological disease at Shanghai Children's Medical Center. Medical records of patients (>30 days and <19 years old) with fever (≥38°C) or absolute neutrophil count (ANC) below 1.0 × 109/L hospitalized with hematological or oncological disease between January 1, 2017 and August 1, 2019 were considered. Patients in whom septic shock was diagnosed during the observation period formed the septic shock group, whereas non-septic-shock group was the control group. In the septic shock group, the time points at 4, 8, 12, and 24 hours prior to septic shock were taken as observation points, and corresponding observation points were obtained in the control group after matching. We employed machine learning artificial intelligence (AI) to filter features and used XGBoost algorithm to build SSEW model. Area under the ROC curve (AU-ROC) was used to compare the effectiveness among the SSEW Model, logistic regression model, and pediatric sequential organ failure score (pSOFA) for early warning of septic shock.

MAIN RESULTS

A total of 64 observation periods in the septic shock group and 2191 in the control group were included. AU-ROC of the SSEW model had higher predictive value for septic shock compared with the pSOFA score (0.93 vs. 0.76, Z = -2.73, P = 0.006). Further analysis showed that the AU-ROC of the SSEW model was superior to the pSOFA score at the observation points 4, 8, 12, and 24 h before septic shock. At the 24 h observation point, the SSEW model incorporated 14 module root features and 23 derived features.

CONCLUSION

The SSEW model for hematological or oncological pediatric patients could help clinicians to predict the risk of septic shock in patients with fever or neutropenia 24 h in advance. Further prospective studies on clinical application scenarios are needed to determine the clinical utility of this AI model.

Validation of a Machine Learning Model for Early Shock Detection

OBJECTIVES

The objectives of this study were to test in real time a Trauma Triage, Treatment, and Training Decision Support (4TDS) machine learning (ML) model of shock detection in a prospective silent trial, and to evaluate specificity, sensitivity, and other estimates of diagnostic performance compared to the gold standard of electronic medical records (EMRs) review.

DESIGN

We performed a single-center diagnostic performance study.

PATIENTS AND SETTING

A prospective cohort consisted of consecutive patients aged 18 years and older who were admitted from May 1 through September 30, 2020 to six Mayo Clinic intensive care units (ICUs) and five progressive care units.

MEASUREMENTS AND MAIN RESULTS

During the study time, 5,384 out of 6,630 hospital admissions were eligible. During the same period, the 4TDS shock model sent 825 alerts and 632 were eligible. Among 632 hospital admissions with alerts, 287 were screened positive and 345 were negative. Among 4,752 hospital admissions without alerts, 78 were screened positive and 4,674 were negative. The area under the receiver operating characteristics curve for the 4TDS shock model was 0.86 (95% CI 0.85-0.87%). The 4TDS shock model demonstrated a sensitivity of 78.6% (95% CI 74.1-82.7%) and a specificity of 93.1% (95% CI 92.4-93.8%). The model showed a positive predictive value of 45.4% (95% CI 42.6-48.3%) and a negative predictive value of 98.4% (95% CI 98-98.6%).

CONCLUSIONS

We successfully validated an ML model to detect circulatory shock in a prospective observational study. The model used only vital signs and showed moderate performance compared to the gold standard of clinician EMR review when applied to an ICU patient cohort.

Early Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review

Background: Sepsis is among the leading causes of death in intensive care units (ICUs) worldwide and its recognition, particularly in the early stages of the disease, remains a medical challenge. The advent of an affluence of available digital health data has created a setting in which machine learning can be used for digital biomarker discovery, with the ultimate goal to advance the early recognition of sepsis. Objective: To systematically review and evaluate studies employing machine learning for the prediction of sepsis in the ICU. Data Sources: Using Embase, Google Scholar, PubMed/Medline, Scopus, and Web of Science, we systematically searched the existing literature for machine learning-driven sepsis onset prediction for patients in the ICU. Study Eligibility Criteria: All peer-reviewed articles using machine learning for the prediction of sepsis onset in adult ICU patients were included. Studies focusing on patient populations outside the ICU were excluded. Study Appraisal and Synthesis Methods: A systematic review was performed according to the PRISMA guidelines. Moreover, a quality assessment of all eligible studies was performed. Results: Out of 974 identified articles, 22 and 21 met the criteria to be included in the systematic review and quality assessment, respectively. A multitude of machine learning algorithms were applied to refine the early prediction of sepsis. The quality of the studies ranged from "poor" (satisfying ≤ 40% of the quality criteria) to "very good" (satisfying ≥ 90% of the quality criteria). The majority of the studies (n = 19, 86.4%) employed an offline training scenario combined with a horizon evaluation, while two studies implemented an online scenario (n = 2, 9.1%). The massive inter-study heterogeneity in terms of model development, sepsis definition, prediction time windows, and outcomes precluded a meta-analysis. Last, only two studies provided publicly accessible source code and data sources fostering reproducibility. Limitations: Articles were only eligible for inclusion when employing machine learning algorithms for the prediction of sepsis onset in the ICU. This restriction led to the exclusion of studies focusing on the prediction of septic shock, sepsis-related mortality, and patient populations outside the ICU. Conclusions and Key Findings: A growing number of studies employs machine learning to optimize the early prediction of sepsis through digital biomarker discovery. This review, however, highlights several shortcomings of the current approaches, including low comparability and reproducibility. Finally, we gather recommendations how these challenges can be addressed before deploying these models in prospective analyses. Systematic Review Registration Number: CRD42020200133.

Graph Convolutional Networks-Based Noisy Data Imputation in Electronic Health Record

OBJECTIVES

A deep learning-based early warning system is proposed to predict sepsis prior to its onset.

DESIGN

A novel algorithm was devised to detect sepsis 6 hours prior to its onset based on electronic medical records.

SETTING

Retrospective cohorts from three separate hospitals are used in this study. Sepsis onset was defined based on Sepsis-3. Algorithms are evaluated based on the score function used in the Physionet Challenge 2019.

PATIENTS

Over 60,000 ICU patients with 40 clinical variables (vital signs, laboratory results) for each hour of a patient's ICU stay were used.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The proposed algorithm predicted the onset of sepsis in the preceding n hours (where n = 4, 6, 8, or 12). Furthermore, the proposed method compared how many sepsis patients can be predicted in a short time with other methods. To interpret a given result in a clinical perspective, the relationship between input variables and the probability of the proposed method were presented. The proposed method achieved superior results (area under the receiver operating characteristic curve, area under the precision-recall curve, and score) and predicted more sepsis patients in advance. In official phase, the proposed method showed the utility score of -0.101, area under the receiver operating characteristic curve 0.782, area under the precision-recall curve 0.041, accuracy 0.786, and F-measure 0.046.

CONCLUSIONS

Using Physionet Challenge 2019, the proposed method can accurately and early predict the onset of sepsis. The proposed method can be a practical early warning system in the environment of real hospitals.

A Time-Phased Machine Learning Model for Real-Time Prediction of Sepsis in Critical Care

OBJECTIVES

As a life-threatening condition, sepsis is one of the major public health issues worldwide. Early prediction can improve sepsis outcomes with appropriate interventions. With the PhysioNet/Computing in Cardiology Challenge 2019, we aimed to develop and validate a machine learning algorithm with high prediction performance and clinical interpretability for prediction of sepsis onset during critical care in real-time.

DESIGN

Retrospective observational cohort study.

SETTING

The dataset was collected from three ICUs in three different U.S. hospitals. Two of them were publicly available for model development (offline) and one was used for testing (online).

PATIENTS

Forty-thousand three-hundred thirty-six ICU patients from the two model development databases and 24,819 from the test database. There are up to 40 hourly-recorded clinical variables for each ICU stay. The Sepsis-3 criteria were used to confirm sepsis onset.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Three-hundred twelve features were constructed hourly as the input of our proposed Time-phAsed machine learning model for Sepsis Prediction. Time-phAsed machine learning model for Sepsis Prediction first estimates the likelihood of sepsis onset for each hour of an ICU stay in the following 6 hours, and then makes a binary prediction with three time-phased cutoff values. On the internal validation set, the utility score (official challenge measurement) achieved by Time-phAsed machine learning model for Sepsis Prediction was 0.430. On the test set, the utility score reached was 0.354. Furthermore, Time-phAsed machine learning model for Sepsis Prediction provides an intuitive way to illustrate the impact of the input features on the outcome prediction, which makes it clinically interpretable.

CONCLUSIONS

The proposed Time-phAsed machine learning model for Sepsis Prediction model is accurate and interpretable for real-time prediction of sepsis onset in critical care, which holds great potential for further evaluation in prospective studies.

Kinematics approach with neural networks for early detection of sepsis (KANNEDS)

BACKGROUND

Sepsis is a severe illness that affects millions of people worldwide, and its early detection is critical for effective treatment outcomes. In recent years, researchers have used models to classify positive patients or identify the probability for sepsis using vital signs and other time-series variables as input.

METHODS

In our study, we analyzed patients' conditions by their kinematics position, velocity, and acceleration, in a six-dimensional space defined by six vital signs. The patient is affected by the disease after a period if the position gets "near" to a calculated sepsis position in space. We imputed these kinematics features as explanatory variables of long short-term memory (LSTM), convolutional neural network (CNN) and linear neural network (LNN) and compared the prediction accuracies with only the vital signs as input. The dataset used contained information of approximately 4800 patients, each with 48 hourly registers.

RESULTS

We demonstrated that the kinematics features models had an improved performance compared with vital signs models. The kinematics features model of LSTM achieved the best accuracy, 0.803, which was nine points higher than the vital signs model. Although with lesser accuracies, the kinematics features models of the CNN and LNN showed better performances than vital signs models.

CONCLUSION

Applying our novel approach for early detection of sepsis using neural networks will prove to be an invaluable, more accurate method than considering only simple vital signs as input variables. We expect that other researchers with similar objectives can use the model presented in this innovative approach to improve their results.

Rethinking PICO in the Machine Learning Era: ML-PICO

BACKGROUND

Machine learning (ML) has captured the attention of many clinicians who may not have formal training in this area but are otherwise increasingly exposed to ML literature that may be relevant to their clinical specialties. ML papers that follow an outcomes-based research format can be assessed using clinical research appraisal frameworks such as PICO (Population, Intervention, Comparison, Outcome). However, the PICO frameworks strain when applied to ML papers that create new ML models, which are akin to diagnostic tests. There is a need for a new framework to help assess such papers.

OBJECTIVE

We propose a new framework to help clinicians systematically read and evaluate medical ML papers whose aim is to create a new ML model: ML-PICO (Machine Learning, Population, Identification, Crosscheck, Outcomes). We describe how the ML-PICO framework can be applied toward appraising literature describing ML models for health care.

CONCLUSION

The relevance of ML to practitioners of clinical medicine is steadily increasing with a growing body of literature. Therefore, it is increasingly important for clinicians to be familiar with how to assess and best utilize these tools. In this paper we have described a practical framework on how to read ML papers that create a new ML model (or diagnostic test): ML-PICO. We hope that this can be used by clinicians to better evaluate the quality and utility of ML papers.

More Generalizable Models For Sepsis Detection Under Covariate Shift

Sepsis is a major cause of mortality in the intensive care units (ICUs). Early intervention of sepsis can improve clinical outcomes for sepsis patients^1,2,3. Machine learning models have been developed for clinical recognition of sepsis^4,5,6. A common assumption of supervised machine learning models is that the covariates in the testing data follow the same distributions as those in the training data. When this assumption is violated (e.g., there is covariate shift), models that performed well for training data could perform badly for testing data. Covariate shift happens when the relationships between covariates and the outcome stay the same, but the marginal distributions of the covariates differ among training and testing data. Covariate shift could make clinical risk prediction model nongeneralizable. In this study, we applied covariate shift corrections onto common machine learning models and have observed that these corrections can help the models be more generalizable under the occurrence of covariate shift when detecting the onset of sepsis.

MGP-AttTCN: An interpretable machine learning model for the prediction of sepsis

With a mortality rate of 5.4 million lives worldwide every year and a healthcare cost of more than 16 billion dollars in the USA alone, sepsis is one of the leading causes of hospital mortality and an increasing concern in the ageing western world. Recently, medical and technological advances have helped re-define the illness criteria of this disease, which is otherwise poorly understood by the medical society. Together with the rise of widely accessible Electronic Health Records, the advances in data mining and complex nonlinear algorithms are a promising avenue for the early detection of sepsis. This work contributes to the research effort in the field of automated sepsis detection with an open-access labelling of the medical MIMIC-III data set. Moreover, we propose MGP-AttTCN: a joint multitask Gaussian Process and attention-based deep learning model to early predict the occurrence of sepsis in an interpretable manner. We show that our model outperforms the current state-of-the-art and present evidence that different labelling heuristics lead to discrepancies in task difficulty. For instance, when predicting sepsis five hours prior to onset on our new realistic labels, our proposed model achieves an area under the ROC curve of 0.660 and an area under the PR curve of 0.483, whereas the (less interpretable) previous state-of-the-art model (MGP-TCN) achieves 0.635 AUROC and 0.460 AUPR and the popular commercial InSight model achieves 0.490 AUROC and 0.359 AUPR.

Clinical factors associated with rapid treatment of sepsis

PURPOSE

To understand what clinical presenting features of sepsis patients are historically associated with rapid treatment involving antibiotics and fluids, as appropriate.

DESIGN

This was a retrospective, observational cohort study using a machine-learning model with an embedded feature selection mechanism (gradient boosting machine).

METHODS

For adult patients (age ≥ 18 years) who were admitted through Emergency Department (ED) meeting clinical criteria of severe sepsis from 11/2007 to 05/2018 at an urban tertiary academic medical center, we developed gradient boosting models (GBMs) using a total of 760 original and derived variables, including demographic variables, laboratory values, vital signs, infection diagnosis present on admission, and historical comorbidities. We identified the most impactful factors having strong association with rapid treatment, and further applied the Shapley Additive exPlanation (SHAP) values to examine the marginal effects for each factor.

RESULTS

For the subgroups with or without fluid bolus treatment component, the models achieved high accuracy of area-under-receiver-operating-curve of 0.91 [95% CI, 0.86-0.95] and 0.84 [95% CI, 0.81-0.86], and sensitivity of 0.81[95% CI, 0.72-0.87] and 0.91 [95% CI, 0.81-0.97], respectively. We identified the 20 most impactful factors associated with rapid treatment for each subgroup. In the non-hypotensive subgroup, initial physiological values were the most impactful to the model, while in the fluid bolus subgroup, value minima and maxima tended to be the most impactful.

CONCLUSION

These machine learning methods identified factors associated with rapid treatment of severe sepsis patients from a large volume of high-dimensional clinical data. The results provide insight into differences in the rapid provision of treatment among patients with sepsis.

Predicting presumed serious infection among hospitalized children on central venous lines with machine learning

BACKGROUND

Presumed serious infection (PSI) is defined as a blood culture drawn and new antibiotic course of at least 4 days among pediatric patients with Central Venous Lines (CVLs). Early PSI prediction and use of medical interventions can prevent adverse outcomes and improve the quality of care.

METHODS

Clinical features including demographics, laboratory results, vital signs, characteristics of the CVLs and medications used were extracted retrospectively from electronic medical records. Data were aggregated across all hospitals within a single pediatric health system and used to train machine learning models (XGBoost and ElasticNet) to predict the occurrence of PSI 8 h prior to clinical suspicion. Prediction for PSI was benchmarked against PRISM-III.

RESULTS

Our model achieved an area under the receiver operating characteristic curve of 0.84 (95% CI = [0.82, 0.85]), sensitivity of 0.73 [0.69, 0.74], and positive predictive value (PPV) of 0.36 [0.34, 0.36]. The PRISM-III conversely achieved a lower sensitivity of 0.19 [0.16, 0.22] and PPV of 0.30 [0.26, 0.34] at a cut-off of ≥ 10. The features with the most impact on the PSI prediction were maximum diastolic blood pressure prior to PSI prediction (mean SHAP = 3.4), height (mean SHAP = 3.2), and maximum temperature prior to PSI prediction (mean SHAP = 2.6).

CONCLUSION

A machine learning model using common features in the electronic medical records can predict the onset of serious infections in children with central venous lines at least 8 h prior to when a clinical team drew a blood culture.

Development and validation of an early warning tool for sepsis and decompensation in children during emergency department triage

This study was designed to develop and validate an early warning system for sepsis based on a predictive model of critical decompensation. Data from the electronic medical records for 537,837 visits to a pediatric Emergency Department (ED) from March 2013 to December 2019 were collected. A multiclass stochastic gradient boosting model was built to identify early warning signs associated with death, severe sepsis, non-severe sepsis, and bacteremia. Model features included triage vital signs, previous diagnoses, medications, and healthcare utilizations within 6 months of the index ED visit. There were 483 patients who had severe sepsis and/or died, 1102 had non-severe sepsis, 1103 had positive bacteremia tests, and the remaining had none of the events. The most important predictors were age, heart rate, length of stay of previous hospitalizations, temperature, systolic blood pressure, and prior sepsis. The one-versus-all area under the receiver operator characteristic curve (AUROC) were 0.979 (0.967, 0.991), 0.990 (0.985, 0.995), 0.976 (0.972, 0.981), and 0.968 (0.962, 0.974) for death, severe sepsis, non-severe sepsis, and bacteremia without sepsis respectively. The multi-class macro average AUROC and area under the precision recall curve were 0.977 and 0.316 respectively. The study findings were used to develop an automated early warning decision tool for sepsis. Implementation of this model in pediatric EDs will allow sepsis-related critical decompensation to be predicted accurately after a few seconds of triage.

Physiological machine learning models for prediction of sepsis in hospitalized adults: An integrative review

BACKGROUND

Diagnosing sepsis remains challenging. Data compiled from continuous monitoring and electronic health records allow for new opportunities to compute predictions based on machine learning techniques. There has been a lack of consensus identifying best practices for model development and validation towards early identification of sepsis.

OBJECTIVE

To evaluate the modeling approach and statistical methodology of machine learning prediction models for sepsis in the adult hospital population.

METHODS

PubMed, CINAHL, and Cochrane databases were searched with the Preferred Reporting Items for Systematic Reviews guided protocol development. We evaluated studies that developed or validated physiologic sepsis prediction models or implemented a model in the hospital environment.

RESULTS

Fourteen studies met the inclusion criteria, and the AUROC of the prediction models ranged from 0.61 to 0.96. We found a variety of sepsis definitions, methods used for event adjudication, model parameters used, and modeling methods. Two studies tested models in clinical settings; the results suggested that patient outcomes were improved with implementation of machine learning models.

CONCLUSION

Nurses have a unique perspective to offer in the development and implementation of machine learning models detecting patients at risk for sepsis. More work is needed in developing model harmonization standards and testing in clinical settings.

Clinical significance of MDRO screening and infection risk factor analysis in the ICU

OBJECTIVE

This study aimed to investigate the clinical significance of multi-drug resistant organism (MDRO) screening and infection risk factor analysis in the intensive care unit (ICU).

METHOD

A total of 210 patients treated in the ICU of our hospital were enrolled as the study subjects, and were divided into the MDRO group (n=100 cases) and the non-MDRO group (n=110) according to the presence or absence of MDRO infection after examination of the pharyngeal swabs. The pathogens of MDRO infection and drug resistance were analyzed. The single-factor as well as multifactor logistic regression analysis of MDRO infections were carried out and the 30-d mortality rate, hospitalization time and treatment costs were compared between the two groups.

RESULTS

A total of 158 MDRO strains were detected in 100 patients with MDRO infection, of which G-84 accounted for 53.16% and G+ 74 accounted for 46.84%. The resistance analysis revealed that G-MDRO was sensitive to imipenem and G+ MDRO was sensitive to vancomycin, and no vancomycin-resistant MDROs were found. The logistic regression model and multifactorial analysis showed that mechanical ventilation, arterial and venous intubation, implementation of fiberoptic bronchoscopy, concurrent chronic lung disease and chronic cardiovascular disease were independent risk factors for the development of MDRO infection (P<0.05). The length of hospital stay, cost of treatment, and 30-d mortality rate in the MDRO group were significantly higher than those in the non-MDRO group (P<0.05).

CONCLUSION

ICU mechanical ventilation, arterial and intravenous intubation, fiberoptic bronchoscopy, concurrent chronic lung disease and chronic cardiovascular disease are the independent risk factors for MDRO infection.

Machine Learning and Syncope Management in the ED: The Future Is Coming

In recent years, machine learning (ML) has been promisingly applied in many fields of clinical medicine, both for diagnosis and prognosis prediction. Aims of this narrative review were to summarize the basic concepts of ML applied to clinical medicine and explore its main applications in the emergency department (ED) setting, with a particular focus on syncope management. Through an extensive literature search in PubMed and Embase, we found increasing evidence suggesting that the use of ML algorithms can improve ED triage, diagnosis, and risk stratification of many diseases. However, the lacks of external validation and reliable diagnostic standards currently limit their implementation in clinical practice. Syncope represents a challenging problem for the emergency physician both because its diagnosis is not supported by specific tests and the available prognostic tools proved to be inefficient. ML algorithms have the potential to overcome these limitations and, in the future, they could support the clinician in managing syncope patients more efficiently. However, at present only few studies have addressed this issue, albeit with encouraging results.

Machine learning in patient flow: a review

This work is a review of the ways in which machine learning has been used in order to plan, improve or aid the problem of moving patients through healthcare services. We decompose the patient flow problem into four subcategories: prediction of demand on a healthcare institution, prediction of the demand and resource required to transfer patients from the emergency department to the hospital, prediction of potential resource required for the treatment and movement of inpatients and prediction of length-of-stay and discharge timing. We argue that there are benefits to both approaches of considering the healthcare institution as a whole as well as the patient by patient case and that ideally a combination of these would be best for improving patient flow through hospitals. We also argue that it is essential for there to be a shared dataset that will allow researchers to benchmark their algorithms on and thereby allow future researchers to build on that which has already been done. We conclude that machine learning for the improvement of patient flow is still a young field with very few papers tailor-making machine learning methods for the problem being considered. Future works should consider the need to transfer algorithms trained on a dataset to multiple hospitals and allowing for dynamic algorithms which will allow real-time decision-making to help clinical staff on the shop floor.

Enhancement in Performance of Septic Shock Prediction Using National Early Warning Score, Initial Triage Information, and Machine Learning Analysis

BACKGROUND

Several studies reported that the National Early Warning Score (NEWS) has shown superiority over other screening tools in discriminating emergency department (ED) patients who are likely to progress to septic shock.

OBJECTIVES

To improve the performance of the NEWS for septic shock prediction by adding variables collected during ED triage, and to implement a machine-learning algorithm.

METHODS

The study population comprised adult ED patients with suspected infection. To detect septic shock within 24 h after ED arrival, the Sepsis-3 clinical criteria and nine variables were used: NEWS, age, gender, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, and oxygen saturation. The model was developed using logistic regression (LR), extreme gradient boosting (XGB), and artificial neural network (ANN) algorithms. The evaluations were performed using an area under the receiver operating characteristic curve (AUROC), Hosmer-Lemeshow test, and net reclassification index (NRI).

RESULTS

Overall, 41,687 patients were enrolled. The AUROC of the model with NEWS, age, gender, and the six vital signs (0.835-0.845) was better than that of the baseline model (0.804). The XGB model (AUROC 0.845) was the most accurate, compared with LR (0.844) and ANN (0.835). The LR and XGB models were well calibrated; however, the ANN showed poor calibration power. The LR and XGB models showed better reclassification than the baseline model with positive NRI.

CONCLUSION

The discrimination power of the model for screening septic shock using NEWS, age, gender, and the six vital signs collected at ED triage outperformed the baseline NEWS model.

Implementation of an Automated Sepsis Screening Tool in a Community Hospital Setting

BACKGROUND

Early identification of sepsis remains the greatest barrier to compliance with recommended evidence-based bundles.

PURPOSE

The purpose was to improve the early identification and treatment of sepsis by developing an automated screening tool.

METHODS

Six variables associated with sepsis were identified. Logistic regression was used to weigh the variables, and a predictive model was developed to help identify patients at risk. A retrospective review of 10 792 records of hospitalizations was conducted including 339 cases of sepsis to retrieve data for the model.

RESULTS

The final model resulted an area under the curve of 0.857 (95% CI, 0.850-0.863), suggesting that the screening tool may assist in the early identification of patients developing sepsis.

CONCLUSION

By using artificial intelligence capabilities, we were able to screen 100% of our inpatient population and deliver results directly to the caregiver without any manual intervention by nursing staff.

Registered Trials on Artificial Intelligence Conducted in Emergency Department and Intensive Care Unit: A Cross-Sectional Study on ClinicalTrials.gov

Objective: Clinical trials contribute to the development of clinical practice. However, little is known about the current status of trials on artificial intelligence (AI) conducted in emergency department and intensive care unit. The objective of the study was to provide a comprehensive analysis of registered trials in such field based on ClinicalTrials.gov.

Methods: Registered trials on AI conducted in emergency department and intensive care unit were searched on ClinicalTrials.gov up to 12th January 2021. The characteristics were analyzed using SPSS21.0 software.

Results: A total of 146 registered trials were identified, including 61 in emergency department and 85 in intensive care unit. They were registered from 2004 to 2021. Regarding locations, 58 were conducted in Europe, 58 in America, 9 in Asia, 4 in Australia, and 17 did not report locations. The enrollment of participants was from 0 to 18,000,000, with a median of 233. Universities were the primary sponsors, which accounted for 43.15%, followed by hospitals (35.62%), and industries/companies (9.59%). Regarding study designs, 85 trials were interventional trials, while 61 were observational trials. Of the 85 interventional trials, 15.29% were for diagnosis and 38.82% for treatment; of the 84 observational trials, 42 were prospective, 14 were retrospective, 2 were cross-sectional, 2 did not report clear information and 1 was unknown. Regarding the trials' results, 69 trials had been completed, while only 10 had available results on ClinicalTrials.gov.

Conclusions: Our study suggest that more AI trials are needed in emergency department and intensive care unit and sponsors are encouraged to report the results.

Clinician involvement in research on machine learning-based predictive clinical decision support for the hospital setting: A scoping review

OBJECTIVE

The study sought to describe the prevalence and nature of clinical expert involvement in the development, evaluation, and implementation of clinical decision support systems (CDSSs) that utilize machine learning to analyze electronic health record data to assist nurses and physicians in prognostic and treatment decision making (ie, predictive CDSSs) in the hospital.

MATERIALS AND METHODS

A systematic search of PubMed, CINAHL, and IEEE Xplore and hand-searching of relevant conference proceedings were conducted to identify eligible articles. Empirical studies of predictive CDSSs using electronic health record data for nurses or physicians in the hospital setting published in the last 5 years in peer-reviewed journals or conference proceedings were eligible for synthesis. Data from eligible studies regarding clinician involvement, stage in system design, predictive CDSS intention, and target clinician were charted and summarized.

RESULTS

Eighty studies met eligibility criteria. Clinical expert involvement was most prevalent at the beginning and late stages of system design. Most articles (95%) described developing and evaluating machine learning models, 28% of which described involving clinical experts, with nearly half functioning to verify the clinical correctness or relevance of the model (47%).

DISCUSSION

Involvement of clinical experts in predictive CDSS design should be explicitly reported in publications and evaluated for the potential to overcome predictive CDSS adoption challenges.

CONCLUSIONS

If present, clinical expert involvement is most prevalent when predictive CDSS specifications are made or when system implementations are evaluated. However, clinical experts are less prevalent in developmental stages to verify clinical correctness, select model features, preprocess data, or serve as a gold standard.

DeepAISE - An interpretable and recurrent neural survival model for early prediction of sepsis

Sepsis, a dysregulated immune system response to infection, is among the leading causes of morbidity, mortality, and cost overruns in the Intensive Care Unit (ICU). Early prediction of sepsis can improve situational awareness among clinicians and facilitate timely, protective interventions. While the application of predictive analytics in ICU patients has shown early promising results, much of the work has been encumbered by high false-alarm rates and lack of trust by the end-users due to the 'black box' nature of these models. Here, we present DeepAISE (Deep Artificial Intelligence Sepsis Expert), a recurrent neural survival model for the early prediction of sepsis. DeepAISE automatically learns predictive features related to higher-order interactions and temporal patterns among clinical risk factors that maximize the data likelihood of observed time to septic events. A comparative study of four baseline models on data from hospitalized patients at three different healthcare systems indicates that DeepAISE produces the most accurate predictions (AUCs between 0.87 and 0.90) at the lowest false alarm rates (FARs between 0.20 and 0.25) while simultaneously producing interpretable representations of the clinical time series and risk factors.

Assessment of the timeliness and robustness for predicting adult sepsis

Sepsis is a leading cause of death among inpatients at hospitals. However, with early detection, death rate can drop substantially. In this study, we present the top-performing algorithm for Sepsis II prediction in the DII National Data Science Challenge using the Cerner Health Facts data involving more than 100,000 adult patients. This large sample size allowed us to dissect the predictability by age-groups, race, genders, and care settings and up to 192 hr of sepsis onset. This large data collection also allowed us to conclude that the last six biometric records on average are informative to the prediction of sepsis. We identified biomarkers that are common across the treatment time and novel biomarkers that are uniquely presented for early prediction. The algorithms showed meaningful signals days ahead of sepsis onset, supporting the potential of reducing death rate by focusing on high-risk populations identified from heterogeneous data integration.

Sepsis-Associated Acute Kidney Injury

Sepsis-associated acute kidney injury (S-AKI) is a common and life-threatening complication in hospitalized and critically ill patients. It is characterized by rapid deterioration of renal function associated with sepsis. The pathophysiology of S-AKI remains incompletely understood, so most therapies remain reactive and nonspecific. Possible pathogenic mechanisms to explain S-AKI include microcirculatory dysfunction, a dysregulated inflammatory response, and cellular metabolic reprogramming. In addition, several biomarkers have been developed in an attempt to improve diagnostic sensitivity and specificity of S-AKI. This article discusses the current understanding of S-AKI, recent advances in pathophysiology and biomarker development, and current preventive and therapeutic approaches.

Early Detection of Sepsis With Machine Learning Techniques: A Brief Clinical Perspective

Sepsis is a major cause of death worldwide. Over the past years, prediction of clinically relevant events through machine learning models has gained particular attention. In the present perspective, we provide a brief, clinician-oriented vision on the following relevant aspects concerning the use of machine learning predictive models for the early detection of sepsis in the daily practice: (i) the controversy of sepsis definition and its influence on the development of prediction models; (ii) the choice and availability of input features; (iii) the measure of the model performance, the output, and their usefulness in the clinical practice. The increasing involvement of artificial intelligence and machine learning in health care cannot be disregarded, despite important pitfalls that should be always carefully taken into consideration. In the long run, a rigorous multidisciplinary approach to enrich our understanding in the application of machine learning techniques for the early recognition of sepsis may show potential to augment medical decision-making when facing this heterogeneous and complex syndrome.

An interpretable deep-learning model for early prediction of sepsis in the emergency department

Sepsis is a life-threatening condition with high mortality rates and expensive treatment costs. Early prediction of sepsis improves survival in septic patients. In this paper, we report our top-performing method in the 2019 DII National Data Science Challenge to predict onset of sepsis 4 h before its diagnosis on electronic health records of over 100,000 unique patients in emergency departments. A long short-term memory (LSTM)-based model with event embedding and time encoding is leveraged to model clinical time series and boost prediction performance. Attention mechanism and global max pooling techniques are utilized to enable interpretation for the deep-learning model. Our model achieved an average area under the curve of 0.892 and was selected as one of the winners of the challenge for both prediction accuracy and clinical interpretability. This study paves the way for future intelligent clinical decision support, helping to deliver early, life-saving care to the bedside of septic patients.

The 4-Hour Cairns Sepsis Model: A novel approach to predicting sepsis mortality at intensive care unit admission

BACKGROUND

Sepsis commonly causes intensive care unit (ICU) mortality, yet early identification of adults with sepsis at risk of dying in the ICU remains a challenge.

OBJECTIVE

The aim of the study was to derive a mortality prediction model (MPM) to assist ICU clinicians and researchers as a clinical decision support tool for adults with sepsis within 4 h of ICU admission.

METHODS

A cohort study was performed using 500 consecutive admissions between 2014 and 2018 to an Australian tertiary ICU, who were aged ≥18 years and had sepsis. A total of 106 independent variables were assessed against ICU episode-of-care mortality. Multivariable backward stepwise logistic regression derived an MPM, which was assessed on discrimination, calibration, fit, sensitivity, specificity, and predictive values and bootstrapped.

RESULTS

The average cohort age was 58 years, the Acute Physiology and Chronic Health Evaluation III-j severity score was 72, and the case fatality rate was 12%. The 4-Hour Cairns Sepsis Model (CSM-4) consists of age, history of renal disease, number of vasopressors, Glasgow Coma Scale, lactate, bicarbonate, aspartate aminotransferase, lactate dehydrogenase, albumin, and magnesium with an area under the receiver operating characteristic curve of 0.90 (95% confidence interval = 0.84-0.95, p < 0.00001), a Nagelkerke R² of 0.51, specificity of 0.94, a negative predictive value of 0.98, and almost identical odds ratios during bootstrapping. The CSM-4 outperformed existing MPMs tested on our data set. The CSM-4 also performed similar to existing MPMs in their derivation papers whilst using fewer, routinely collected, and inexpensive variables.

CONCLUSIONS

The CSM-4 is a newly derived MPM for adults with sepsis at ICU admission. It displays excellent discrimination, calibration, fit, specificity, negative predictive value, and bootstrapping values whilst being easy to use and inexpensive. External validation is required.