Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU

Use of machine learning to analyse routinely collected intensive care unit data: a systematic review

BACKGROUND

Intensive care units (ICUs) face financial, bed management, and staffing constraints. Detailed data covering all aspects of patients' journeys into and through intensive care are now collected and stored in electronic health records: machine learning has been used to analyse such data in order to provide decision support to clinicians.

METHODS

Systematic review of the applications of machine learning to routinely collected ICU data. Web of Science and MEDLINE databases were searched to identify candidate articles: those on image processing were excluded. The study aim, the type of machine learning used, the size of dataset analysed, whether and how the model was validated, and measures of predictive accuracy were extracted.

RESULTS

Of 2450 papers identified, 258 fulfilled eligibility criteria. The most common study aims were predicting complications (77 papers [29.8% of studies]), predicting mortality (70 [27.1%]), improving prognostic models (43 [16.7%]), and classifying sub-populations (29 [11.2%]). Median sample size was 488 (IQR 108-4099): 41 studies analysed data on > 10,000 patients. Analyses focused on 169 (65.5%) papers that used machine learning to predict complications, mortality, length of stay, or improvement of health. Predictions were validated in 161 (95.2%) of these studies: the area under the ROC curve (AUC) was reported by 97 (60.2%) but only 10 (6.2%) validated predictions using independent data. The median AUC was 0.83 in studies of 1000-10,000 patients, rising to 0.94 in studies of > 100,000 patients. The most common machine learning methods were neural networks (72 studies [42.6%]), support vector machines (40 [23.7%]), and classification/decision trees (34 [20.1%]). Since 2015 (125 studies [48.4%]), the most common methods were support vector machines (37 studies [29.6%]) and random forests (29 [23.2%]).

CONCLUSIONS

The rate of publication of studies using machine learning to analyse routinely collected ICU data is increasing rapidly. The sample sizes used in many published studies are too small to exploit the potential of these methods. Methodological and reporting guidelines are needed, particularly with regard to the choice of method and validation of predictions, to increase confidence in reported findings and aid in translating findings towards routine use in clinical practice.

Predicting sepsis with a recurrent neural network using the MIMIC III database

OBJECTIVE

Predicting sepsis onset with a recurrent neural network and performance comparison with InSight - a previously proposed algorithm for the prediction of sepsis onset.

METHODOLOGY

A retrospective analysis of adult patients admitted to the intensive care unit (from the MIMIC III database) who did not fall under the definition of sepsis at the time of admission. The area under the receiver operating characteristic (AUROC) measures the performance of the prediction task. We examine the sequence length given to the machine learning algorithms for different points in time before sepsis onset concerning the prediction performance. Additionally, the impact of sepsis onset's definition is investigated. We evaluate the model with a relatively large and thus more representative patient population compared to related works in the field.

RESULTS

For a prediction 3 h prior to sepsis onset, our network achieves an AUROC of 0.81 (95% CI: 0.78-0.84). The InSight algorithm achieves an AUROC of 0.72 (95% CI: 0.69-0.75). For a fixed sensitivity of 90% our network reaches a specificity of 47.0% (95% CI: 43.1%-50.8%) compared to 31.1% (95% CI: 24.8%-37.5%) for InSight. In addition, we compare the performance for 6 and 12 h prediction time for both approaches.

CONCLUSION

Our findings demonstrate that a recurrent neural network is superior to InSight considering the prediction performance. Most probably, the improvement results from the network's ability of revealing time dependencies. We show that the length of the look back has a significant impact on the performance of the classifier. We also demonstrate that for the correct detection of sepsis onset for a retrospective analysis, further research is necessary.

Analysis and prediction of unplanned intensive care unit readmission using recurrent neural networks with long short-term memory

BACKGROUND

Unplanned readmission of a hospitalized patient is an indicator of patients' exposure to risk and an avoidable waste of medical resources. In addition to hospital readmission, intensive care unit (ICU) readmission brings further financial risk, along with morbidity and mortality risks. Identification of high-risk patients who are likely to be readmitted can provide significant benefits for both patients and medical providers. The emergence of machine learning solutions to detect hidden patterns in complex, multi-dimensional datasets provides unparalleled opportunities for developing an efficient discharge decision-making support system for physicians and ICU specialists.

METHODS AND FINDINGS

We used supervised machine learning approaches for ICU readmission prediction. We used machine learning methods on comprehensive, longitudinal clinical data from the MIMIC-III to predict the ICU readmission of patients within 30 days of their discharge. We incorporate multiple types of features including chart events, demographic, and ICD-9 embeddings. We have utilized recent machine learning techniques such as Recurrent Neural Networks (RNN) with Long Short-Term Memory (LSTM), by this we have been able to incorporate the multivariate features of EHRs and capture sudden fluctuations in chart event features (e.g. glucose and heart rate). We show that our LSTM-based solution can better capture high volatility and unstable status in ICU patients, an important factor in ICU readmission. Our machine learning models identify ICU readmissions at a higher sensitivity rate of 0.742 (95% CI, 0.718-0.766) and an improved Area Under the Curve of 0.791 (95% CI, 0.782-0.800) compared with traditional methods. We perform in-depth deep learning performance analysis, as well as the analysis of each feature contribution to the predictive model.

CONCLUSION

Our manuscript highlights the ability of machine learning models to improve our ICU decision-making accuracy and is a real-world example of precision medicine in hospitals. These data-driven solutions hold the potential for substantial clinical impact by augmenting clinical decision-making for physicians and ICU specialists. We anticipate that machine learning models will improve patient counseling, hospital administration, allocation of healthcare resources and ultimately individualized clinical care.

CADDIE2-evaluation of a clinical decision-support system for early detection of systemic inflammatory response syndrome in paediatric intensive care: study protocol for a diagnostic study

INTRODUCTION

Systemic inflammatory response syndrome (SIRS) is one of the most critical indicators determining the clinical outcome of paediatric intensive care patients. Clinical decision support systems (CDSS) can be designed to support clinicians in detection and treatment. However, the use of such systems is highly discussed as they are often associated with accuracy problems and 'alert fatigue'. We designed a CDSS for detection of paediatric SIRS and hypothesise that a high diagnostic accuracy together with an adequate alerting will accelerate the use. Our study will (1) determine the diagnostic accuracy of the CDSS compared with gold standard decisions created by two blinded, experienced paediatricians, and (2) compare the system's diagnostic accuracy with that of routine clinical care decisions compared with the same gold standard.

METHODS AND ANALYSIS

CADDIE2 is a prospective diagnostic accuracy study taking place at the Department of Pediatric Cardiology and Intensive Care Medicine at the Hannover Medical School; it represents the second step towards our vision of cross-institutional and data-driven decision-support for intensive care environments (CADDIE). The study comprises (1) recruitment of up to 300 patients (start date 1 August 2018), (2) creation of gold standard decisions (start date 1 May 2019), (3) routine SIRS assessments by physicians (starts with recruitment), (4) SIRS assessments by a CDSS (start date 1 May 2019), and (5) statistical analysis with a modified approach for determining sensitivity and specificity and comparing the accuracy results of the different diagnostic approaches (planned start date 1 July 2019).

ETHICS AND DISSEMINATION

Ethics approval was obtained at the study centre (Ethics Committee of Hannover Medical School). Results of the main study will be communicated via publication in a peer-reviewed journal.

TRIAL REGISTRATION NUMBER

ClinicalTrials.gov NCT03661450; Pre-results.

Applications of Machine Learning Approaches in Emergency Medicine; a Review Article

Using artificial intelligence and machine learning techniques in different medical fields, especially emergency medicine is rapidly growing. In this paper, studies conducted in the recent years on using artificial intelligence in emergency medicine have been collected and assessed. These studies belonged to three categories: prediction and detection of disease; prediction of need for admission, discharge and also mortality; and machine learning based triage systems. In each of these categories, the most important studies have been chosen and accuracy and results of the algorithms have been briefly evaluated by mentioning machine learning techniques and used datasets.

Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs

OBJECTIVE

Sepsis remains a costly and prevalent syndrome in hospitals; however, machine learning systems can increase timely sepsis detection using electronic health records. This study validates a gradient boosted ensemble machine learning tool for sepsis detection and prediction, and compares its performance to existing methods.

MATERIALS AND METHODS

Retrospective data was drawn from databases at the University of California, San Francisco (UCSF) Medical Center and the Beth Israel Deaconess Medical Center (BIDMC). Adult patient encounters without sepsis on admission, and with at least one recording of each of six vital signs (SpO₂, heart rate, respiratory rate, temperature, systolic and diastolic blood pressure) were included. We compared the performance of the machine learning algorithm (MLA) to that of commonly used scoring systems. Area under the receiver operating characteristic (AUROC) curve was our primary measure of accuracy. MLA performance was measured at sepsis onset, and at 24 and 48 h prior to sepsis onset.

RESULTS

The MLA achieved an AUROC of 0.88, 0.84, and 0.83 for sepsis onset and 24 and 48 h prior to onset, respectively. These values were superior to those of SIRS (0.66), MEWS (0.61), SOFA (0.72), and qSOFA (0.60) at time of onset. When trained on UCSF data and tested on BIDMC data, sepsis onset AUROC was 0.89.

DISCUSSION AND CONCLUSION

The MLA predicts sepsis up to 48 h in advance and identifies sepsis onset more accurately than commonly used tools, maintaining high performance for sepsis detection when trained and tested on separate datasets.

Data-driven discovery of a novel sepsis pre-shock state predicts impending septic shock in the ICU

Septic shock is a life-threatening condition in which timely treatment substantially reduces mortality. Reliable identification of patients with sepsis who are at elevated risk of developing septic shock therefore has the potential to save lives by opening an early window of intervention. We hypothesize the existence of a novel clinical state of sepsis referred to as the "pre-shock" state, and that patients with sepsis who enter this state are highly likely to develop septic shock at some future time. We apply three different machine learning techniques to the electronic health record data of 15,930 patients in the MIMIC-III database to test this hypothesis. This novel paradigm yields improved performance in identifying patients with sepsis who will progress to septic shock, as defined by Sepsis- 3 criteria, with the best method achieving a 0.93 area under the receiver operating curve, 88% sensitivity, 84% specificity, and median early warning time of 7 hours. Additionally, we introduce the notion of patient-specific positive predictive value, assigning confidence to individual predictions, and achieving values as high as 91%. This study demonstrates that early prediction of impending septic shock, and thus early intervention, is possible many hours in advance.

Prediction of sepsis patients using machine learning approach: A meta-analysis

STUDY OBJECTIVE

Sepsis is a common and major health crisis in hospitals globally. An innovative and feasible tool for predicting sepsis remains elusive. However, early and accurate prediction of sepsis could help physicians with proper treatments and minimize the diagnostic uncertainty. Machine learning models could help to identify potential clinical variables and provide higher performance than existing traditional low-performance models. We therefore performed a meta-analysis of observational studies to quantify the performance of a machine learning model to predict sepsis.

METHODS

A comprehensive literature search was conducted through the electronic database (e.g. PubMed, Scopus, Google Scholar, EMBASE, etc.) between January 1, 2000, and March 1, 2018. All the studies published in English and reporting the sepsis prediction using machine learning algorithms were considered in this study. Two authors independently extracted valuable information from the included studies. Inclusion and exclusion of studies were based on the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines.

RESULTS

A total of 7 out of 135 studies met all of our inclusion criteria. For machine learning models, the pooled area under receiving operating curve (SAUROC) for predicting sepsis onset 3 to 4 h before, was 0.89 (95%CI: 0.86-0.92); sensitivity 0.81 (95%CI:0.80-0.81), and specificity 0.72 (95%CI:0.72-0.72) whereas the pooled SAUROC for SIRS, MEWS, and SOFA was 0.70, 0.50, and 0.78. Additionally, diagnostic odd ratio for machine learning, SIRS, MEWS, and SOFA was 15.17 (95%CI: 9.51-24.20), 3.23 (95%CI: 1.52-6.87), 31.99 (95% CI: 1.54-666.74), and 3.75(95%CI: 2.06-6.83).

CONCLUSION

Our study findings suggest that the machine learning approach had a better performance than the existing sepsis scoring systems in predicting sepsis.

Cytokines and Chemokines in Pediatric Appendicitis: A Multiplex Analysis of Inflammatory Protein Mediators

OBJECTIVES

We aimed to demonstrate the potential of precision medicine to describe the inflammatory landscape present in children with suspected appendicitis. Our primary objective was to determine levels of seven inflammatory protein mediators previously associated with intra-abdominal inflammation (C-reactive protein-CRP, procalcitonin-PCT, interleukin-6 (IL), IL-8, IL-10, monocyte chemoattractant protein-1-MCP-1, and serum amyloid A-SAA) in a cohort of children with suspected appendicitis. Subsequently, using a multiplex proteomics approach, we examined an expansive array of novel candidate cytokine and chemokines within this population.

METHODS

We performed a secondary analysis of targeted proteomics data from Alberta Sepsis Network studies. Plasma mediator levels, analyzed by Luminex multiplex assays, were evaluated in children aged 5-17 years with nonappendicitis abdominal pain (NAAP), acute appendicitis (AA), and nonappendicitis sepsis (NAS). We used multivariate regression analysis to evaluate the seven target proteins, followed by decision tree and heat mapping analyses for all proteins evaluated.

RESULTS

185 children were included: 83 with NAAP, 79 AA, and 23 NAS. Plasma levels of IL-6, CRP, MCP-1, PCT, and SAA were significantly different in children with AA compared to those with NAAP (p < 0.001). Expansive proteomic analysis demonstrated 6 patterns in inflammatory mediator profiles based on severity of illness. A decision tree incorporating the proteins CRP, ferritin, SAA, regulated on activation normal T-cell expressed and secreted (RANTES), monokine induced by gamma interferon (MIG), and PCT demonstrated excellent specificity (0.920) and negative predictive value (0.882) for children with appendicitis.

CONCLUSIONS

Multiplex proteomic analyses described the inflammatory landscape of children presenting to the ED with suspected appendicitis. We have demonstrated the feasibility of this approach to identify potential novel candidate cytokines/chemokine patterns associated with a specific illness (appendicitis) amongst those with a broad ED presentation (abdominal pain). This approach can be modelled for future research initiatives in pediatric emergency medicine.

Machine-Learning-Based Laboratory Developed Test for the Diagnosis of Sepsis in High-Risk Patients

Sepsis, a dysregulated host response to infection, is a major health burden in terms of both mortality and cost. The difficulties clinicians face in diagnosing sepsis, alongside the insufficiencies of diagnostic biomarkers, motivate the present study. This work develops a machine-learning-based sepsis diagnostic for a high-risk patient group, using a geographically and institutionally diverse collection of nearly 500,000 patient health records. Using only a minimal set of clinical variables, our diagnostics outperform common severity scoring systems and sepsis biomarkers and benefit from being available immediately upon ordering.

Improving Prediction Performance Using Hierarchical Analysis of Real-Time Data: A Sepsis Case Study

This paper presents a novel method for hierarchical analysis of machine learning algorithms to improve predictions of at risk patients, thus further enabling prompt therapy. Specifically, we develop a multi-layer machine learning approach to analyze continuous, high-frequency data. We illustrate the capabilities of this approach for early identification of patients at risk of sepsis, a potentially life-threatening complication of an infection, using high-frequency (minute-by-minute) physiological data collected from bedside monitors. In our analysis of a cohort of 586 patients, the model obtained from analyzing the output of a previously developed sepsis prediction model resulted in improved outcomes. Specifically, the original model failed to predict 11.76 ± 4.26% of sepsis patients earlier than Systemic Inflammatory Response Syndrome (SIRS) criteria, commonly used to identify patients at risk for rapid physiological deterioration resulting from sepsis. In contrast, the multi-layer model only failed to predict 3.21 ± 3.11% of sepsis patients earlier than SIRS. In addition, sepsis patients were predicted on average 204.87 ± 7.90 minutes earlier than SIRS criteria using the multi-layer model, which can potentially help reduce mortality and morbidity if implemented in the ICU.

Sepsis associated acute kidney injury

Sepsis is defined as organ dysfunction resulting from the host's deleterious response to infection. One of the most common organs affected is the kidneys, resulting in sepsis associated acute kidney injury (SA-AKI) that contributes to the morbidity and mortality of sepsis. A growing body of knowledge has illuminated the clinical risk factors, pathobiology, response to treatment, and elements of renal recovery that have advanced our ability to prevent, detect, and treat SA-AKI. Despite these advances, SA-AKI remains an important concern and clinical burden, and further study is needed to reduce the acute and chronic consequences. This review summarizes the relevant evidence, with a focus on the risk factors, early recognition and diagnosis, treatment, and long term consequences of SA-AKI. In addition to literature pertaining to SA-AKI specifically, pertinent sepsis and acute kidney injury literature relevant to SA-AKI was included.

A guide to deep learning in healthcare

Here we present deep-learning techniques for healthcare, centering our discussion on deep learning in computer vision, natural language processing, reinforcement learning, and generalized methods. We describe how these computational techniques can impact a few key areas of medicine and explore how to build end-to-end systems. Our discussion of computer vision focuses largely on medical imaging, and we describe the application of natural language processing to domains such as electronic health record data. Similarly, reinforcement learning is discussed in the context of robotic-assisted surgery, and generalized deep-learning methods for genomics are reviewed.

Red-flag sepsis and SOFA identifies different patient population at risk of sepsis-related deaths on the general ward

Controversy exists regarding the best diagnostic and screening tool for sepsis outside the intensive care unit (ICU). Sequential organ failure assessment (SOFA) score has been shown to be superior to systemic inflammatory response syndrome (SIRS) criteria, however, the performance of "Red Flag sepsis criteria" has not been tested formally.The aim of the study was to investigate the ability of Red Flag sepsis criteria to identify the patients at high risk of sepsis-related death in comparison to SOFA based sepsis criteria. We also investigated the comparison of Red Flag sepsis to quick SOFA (qSOFA), SIRS, and national early warning score (NEWS) scores and factors influencing patient mortality.Patients were recruited into a 24-hour point-prevalence study on the general wards and emergency departments across all Welsh acute hospitals. Inclusion criteria were: clinical suspicion of infection and NEWS 3 or above in-line with established escalation criteria in Wales. Data on Red Flag sepsis and SOFA criteria was collected together with qSOFA and SIRS scores and 90-day mortality.459 patients were recruited over a 24-hour period. 246 were positive for Red Flag sepsis, mortality 33.7% (83/246); 241 for SOFA based sepsis criteria, mortality 39.4% (95/241); 54 for qSOFA, mortality 57.4% (31/54), and 268 for SIRS, mortality 33.6% (90/268). 55 patients were not picked up by any criteria. We found that older age was associated with death with OR (95% CI) of 1.03 (1.02-1.04); higher frailty score 1.24 (1.11-1.40); DNA-CPR order 1.74 (1.14-2.65); ceiling of care 1.55 (1.02-2.33); and SOFA score of 2 and above 1.69 (1.16-2.47).The different clinical tools captured different subsets of the at-risk population, with similar sensitivity. SOFA score 2 or above was independently associated with increased risk of death at 90 days. The sequalae of infection-related organ dysfunction cannot be reliably captured based on routine clinical and physiological parameters alone.

Data collection and research with MargheritaTre

OBJECTIVE

MargheritaTre is an electronic health record developed by the Italian Group for the Evaluation of Interventions in Intensive Care Medicine designed to support clinical practice in intensive care units (ICUs) and ensure high-quality data for research purposes.

APPROACH

MargheritaTre was developed in collaboration with clinical experts, researchers, and IT specialists. It is currently installed in 40 ICUs and its database contains complete records of more than 65,000 patients. To facilitate data analysis, information is mostly stored in structured or partially structured form.

MAIN RESULTS

Data collected with MargheritaTre allow one to conduct research studies on complex clinical problems from manifold perspectives and with different levels of detail, such as epidemiological studies, analyses of the process of care and physiopathological investigations, at both single-organ and organism level. In this paper we describe some of the first projects based on this electronic health record to illustrate its potential for research.

SIGNIFICANCE

The MargheritaTre database is a huge and rapidly growing mine of data that will be exploited by our laboratory and shared with other groups to address complex and innovative research and clinical questions. The ultimate aim of these projects is the improvement of the quality of care and patient outcomes, through the development of expert systems integrated in the electronic health record to support clinical practice.

Impact of predictive analytics based on continuous cardiorespiratory monitoring in a surgical and trauma intensive care unit

Predictive analytics monitoring, the use of patient data to provide continuous risk estimation of deterioration, is a promising new application of big data analytical techniques to the care of individual patients. We tested the hypothesis that continuous display of novel electronic risk visualization of respiratory and cardiovascular events would impact intensive care unit (ICU) patient outcomes. In an adult tertiary care surgical trauma ICU, we displayed risk estimation visualizations on a large monitor, but in the medical ICU in the same institution we did not. The risk estimates were based solely on analysis of continuous cardiorespiratory monitoring. We examined 4275 individual patient records within a 7 month time period preceding and following data display. We determined cases of septic shock, emergency intubation, hemorrhage, and death to compare rates per patient care pre-and post-implementation. Following implementation, the incidence of septic shock fell by half (p < 0.01 in a multivariate model that included age and APACHE) in the surgical trauma ICU, where the data were continuously on display, but by only 10% (p = NS) in the control Medical ICU. There were no significant changes in the other outcomes. Display of a predictive analytics monitor based on continuous cardiorespiratory monitoring was followed by a reduction in the rate of septic shock, even when controlling for age and APACHE score.

Artificial intelligence and machine learning in emergency medicine

Interest in artificial intelligence (AI) research has grown rapidly over the past few years, in part thanks to the numerous successes of modern machine learning techniques such as deep learning, the availability of large datasets and improvements in computing power. AI is proving to be increasingly applicable to healthcare and there is a growing list of tasks where algorithms have matched or surpassed physician performance. Despite the successes there remain significant concerns and challenges surrounding algorithm opacity, trust and patient data security. Notwithstanding these challenges, AI technologies will likely become increasingly integrated into emergency medicine in the coming years. This perspective presents an overview of current AI research relevant to emergency medicine.

Prediction of Acute Kidney Injury With a Machine Learning Algorithm Using Electronic Health Record Data

Background

A major problem in treating acute kidney injury (AKI) is that clinical criteria for recognition are markers of established kidney damage or impaired function; treatment before such damage manifests is desirable. Clinicians could intervene during what may be a crucial stage for preventing permanent kidney injury if patients with incipient AKI and those at high risk of developing AKI could be identified.

Objective

In this study, we evaluate a machine learning algorithm for early detection and prediction of AKI.

Design

We used a machine learning technique, boosted ensembles of decision trees, to train an AKI prediction tool on retrospective data taken from more than 300 000 inpatient encounters.

Setting

Data were collected from inpatient wards at Stanford Medical Center and intensive care unit patients at Beth Israel Deaconess Medical Center.

Patients

Patients older than the age of 18 whose hospital stays lasted between 5 and 1000 hours and who had at least one documented measurement of heart rate, respiratory rate, temperature, serum creatinine (SCr), and Glasgow Coma Scale (GCS).

Measurements

We tested the algorithm's ability to detect AKI at onset and to predict AKI 12, 24, 48, and 72 hours before onset.

Methods

We tested AKI detection and prediction using the National Health Service (NHS) England AKI Algorithm as a gold standard. We additionally tested the algorithm's ability to detect AKI as defined by the Kidney Disease: Improving Global Outcomes (KDIGO) guidelines. We compared the algorithm's 3-fold cross-validation performance to the Sequential Organ Failure Assessment (SOFA) score for AKI identification in terms of area under the receiver operating characteristic (AUROC).

Results

The algorithm demonstrated high AUROC for detecting and predicting NHS-defined AKI at all tested time points. The algorithm achieves AUROC of 0.872 (95% confidence interval [CI], 0.867-0.878) for AKI detection at time of onset. For prediction 12 hours before onset, the algorithm achieves an AUROC of 0.800 (95% CI, 0.792-0.809). For 24-hour predictions, the algorithm achieves AUROC of 0.795 (95% CI, 0.785-0.804). For 48-hour and 72-hour predictions, the algorithm achieves AUROC values of 0.761 (95% CI, 0.753-0.768) and 0.728 (95% CI, 0.719-0.737), respectively.

Limitations

Because of the retrospective nature of this study, we cannot draw any conclusions about the impact the algorithm's predictions will have on patient outcomes in a clinical setting.

Conclusions

The results of these experiments suggest that a machine learning-based AKI prediction tool may offer important prognostic capabilities for determining which patients are likely to suffer AKI, potentially allowing clinicians to intervene before kidney damage manifests.