Improving Prediction Performance Using Hierarchical Analysis of Real-Time Data: A Sepsis Case Study

Machine Learning-Based Early Prediction of Sepsis Using Electronic Health Records: A Systematic Review

BACKGROUND

Sepsis, a life-threatening infection-induced inflammatory condition, has significant global health impacts. Timely detection is crucial for improving patient outcomes as sepsis can rapidly progress to severe forms. The application of machine learning (ML) and deep learning (DL) to predict sepsis using electronic health records (EHRs) has gained considerable attention for timely intervention.

METHODS

PubMed, IEEE Xplore, Google Scholar, and Scopus were searched for relevant studies. All studies that used ML/DL to detect or early-predict the onset of sepsis in the adult population using EHRs were considered. Data were extracted and analyzed from all studies that met the criteria and were also evaluated for their quality.

RESULTS

This systematic review examined 1942 articles, selecting 42 studies while adhering to strict criteria. The chosen studies were predominantly retrospective (n = 38) and spanned diverse geographic settings, with a focus on the United States. Different datasets, sepsis definitions, and prevalence rates were employed, necessitating data augmentation. Heterogeneous parameter utilization, diverse model distribution, and varying quality assessments were observed. Longitudinal data enabled early sepsis prediction, and quality criteria fulfillment varied, with inconsistent funding-article quality correlation.

CONCLUSIONS

This systematic review underscores the significance of ML/DL methods for sepsis detection and early prediction through EHR data.

Application Prospect of the SOFA Score and Related Modification Research Progress in Sepsis

In 2016, the SOFA score was proposed as the main evaluation system for diagnosis in the definition of sepsis 3.0, and the SOFA score has become a new research focus in sepsis. Some people are skeptical about diagnosing sepsis using the SOFA score. Experts and scholars from different regions have proposed different, modified versions of SOFA score to make up for the related problems with the use of the SOFA score in the diagnosis of sepsis. While synthesizing the different improved versions of SOFA proposed by experts and scholars in various regions, this paper also summarizes the relevant definitions of sepsis put forward in recent years to build a clear, improved application framework of SOFA score. In addition, the comparison between machine learning and SOFA scores related to sepsis is described and discussed in the article. Taken together, by summarizing the application of the improved SOFA score proposed in recent years in the related definition of sepsis, we believe that the SOFA score is still an effective means of diagnosing sepsis, but in the process of the continuous refinement and development of sepsis in the future, the SOFA score needs to be further refined and improved to provide more accurate coping strategies for different patient populations or application directions regarding sepsis. Against the big data background, machine learning has immeasurable value and significance, but its future applications should add more humanistic references and assistance.

Prediction of Multiple Clinical Complications in Cancer Patients to Ensure Hospital Preparedness and Improved Cancer Care

Reliable and rapid medical diagnosis is the cornerstone for improving the survival rate and quality of life of cancer patients. The problem of clinical decision-making pertaining to the management of patients with hematologic cancer is multifaceted and intricate due to the risk of therapy-induced myelosuppression, multiple infections, and febrile neutropenia (FN). Myelosuppression due to treatment increases the risk of sepsis and mortality in hematological cancer patients with febrile neutropenia. A high prevalence of multidrug-resistant organisms is also noted in such patients, which implies that these patients are left with limited or no-treatment options amidst severe health complications. Hence, early screening of patients for such organisms in their bodies is vital to enable hospital preparedness, curtail the spread to other weak patients in hospitals, and limit community outbreaks. Even though predictive models for sepsis and mortality exist, no model has been suggested for the prediction of multidrug-resistant organisms in hematological cancer patients with febrile neutropenia. Hence, for predicting three critical clinical complications, such as sepsis, the presence of multidrug-resistant organisms, and mortality, from the data available from medical records, we used 1166 febrile neutropenia episodes reported in 513 patients. The XGboost algorithm is suggested from 10-fold cross-validation on 6 candidate models. Other highlights are (1) a novel set of easily available features for the prediction of the aforementioned clinical complications and (2) the use of data augmentation methods and model-scoring-based hyperparameter tuning to address the problem of class disproportionality, a common challenge in medical datasets and often the reason behind poor event prediction rate of various predictive models reported so far. The proposed model depicts improved recall and AUC (area under the curve) for sepsis (recall = 98%, AUC = 0.85), multidrug-resistant organism (recall = 96%, AUC = 0.91), and mortality (recall = 86%, AUC = 0.88) prediction. Our results encourage the need to popularize artificial intelligence-based devices to support clinical decision-making.

Dynamic prediction of life-threatening events for patients in intensive care unit

Background

Early prediction of patients’ deterioration is helpful in early intervention for patients at greater risk of deterioration in Intensive Care Unit (ICU). This study aims to apply machine learning approaches to heterogeneous clinical data for predicting life-threatening events of patients in ICU.

Methods

We collected clinical data from a total of 3151 patients admitted to the Medical Intensive Care Unit of Peking Union Medical College Hospital in China from January 1st, 2014, to October 1st, 2019. After excluding the patients who were under 18 years old or stayed less than 24 h at the ICU, a total of 2170 patients were enrolled in this study. Multiple machine learning approaches were utilized to predict life-threatening events (i.e., death) in seven 24-h windows (day 1 to day 7) and their performance was compared.

Results

Light Gradient Boosting Machine showed the best performance. We found that life-threatening events during the short-term windows can be better predicted than those in the medium-term windows. For example, death in 24 h can be predicted with an Area Under Curve of 0.905. Features like infusion pump related fluid input were highly related to life-threatening events. Furthermore, the prediction power of static features such as age and cardio-pulmonary function increased with the extended prediction window.

Conclusion

This study demonstrates that the integration of machine learning approaches and large-scale high-quality clinical data in ICU could accurately predict life-threatening events for ICU patients for early intervention.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-02026-x.

Prior Distribution Estimation of Monitored Information in the Intensive Care Unit with the Hidden Markov Model and Decision Tree Methods

In the intensive care unit, the monitored variables collected from sensors may have different behaviors among patients with different clinical basic information. Giving prior information of the monitored variables based on their specific basic information as soon as the patient is admitted will support the clinicians with better decisions during the surgery. Instead of black box models, the explainable hidden Markov model is proposed, which can estimate the possible distribution parameters of the monitored variables under different clinical basic information. A Student's t-test or correlation test is conducted further to test whether the parameters have a significant relationship with the basic variables. The specific relationship is explored by using a conditional inference tree, which is an explainable model giving deciding rules. Instead of point estimation, interval forecast is chosen as the performance metrics including coverage rate and relative interval width, which provide more reliable results. By applying the methods to an intensive care unit data set with more than 20 thousand patients, the model has good performance with an area under the ROC Curve value of 0.75, which means the hidden states can generally be correctly labelled. The significant test shows that only a few combinations of the basic and monitored variables are not significant under the 0.01 significant level. The tree model based on different quantile intervals provides different coverage and width combination choices. A coverage rate around 0.8 is suggested, which has a relative interval width of 0.77.

Sepsis Performance Improvement Programs: From Evidence Toward Clinical Implementation

This article is one of ten reviews selected from the Annual Update in Intensive Care and Emergency Medicine 2022. Other selected articles can be found online at https://www.biomedcentral.com/collections/annualupdate2022. Further information about the Annual Update in Intensive Care and Emergency Medicine is available from https://link.springer.com/bookseries/8901.

Accurate diagnosis of sepsis using a neural network: Pilot study using routine clinical variables

BACKGROUND AND OBJECTIVES

Sepsis is a severe infection that increases mortality risk and is one if the main causes of death in intensive care units. Accurate detection is key to successful interventions, but diagnosis of sepsis is complicated because the initial signs and symptoms are not specific. Biomarkers that have been proposed have low specificity and sensitivity, are expensive, and not available in every hospital. In this study, we propose the use of artificial intelligence in the form of a neural network to diagnose sepsis using only common laboratory tests and vital signs that are routine and widely available.

METHODS

A retrospective, cross sectional cohort of 113 patients from an intensive care unit, each with 48 routinely evaluated vital signs and biochemical parameters was used to train, validate and test a neural network with 48 inputs, 10 neurons in a single hidden layer and one output. The sensitivity and specificity of the neural network as a point sampled diagnostic test was calculated.

RESULTS

All but one case were correctly diagnosed by the neural network, with 91% sensitivity and 100% specificity in the validation data set, and 100% sensitivity and specificity in the test data set.

CONCLUSIONS

The designed neural network system can identify patients with sepsis, with minimal resources using standard laboratory tests widely available in most health care facilities. This should reduce the burden on the medical staff of a difficult diagnosis and should improve outcomes for patients with sepsis.

Deep Learning Model to Predict Serious Infection Among Children With Central Venous Lines

Objective: Predict the onset of presumed serious infection, defined as a positive blood culture drawn and new antibiotic course of at least 4 days (PSI*), among pediatric patients with Central Venous Lines (CVLs). Design: Retrospective cohort study. Setting: Single academic children's hospital. Patients: All hospital encounters from January 2013 to December 2018, excluding the ones without a CVL or with a length-of-stay shorter than 24 h. Measurements and Main Results: Clinical features including demographics, laboratory results, vital signs, characteristics of the CVLs and medications used were extracted retrospectively from electronic medical records. Data were aggregated across all hospitals within a single pediatric health system and used to train a deep learning model to predict the occurrence of PSI* during the next 48 h of hospitalization. The proposed model prediction was compared to prediction of PSI* by a marker of illness severity (PELOD-2). The baseline prevalence of line infections was 0.34% over all segmented 48-h time windows. Events were identified among cases using onset time. All data from admission till the onset was used for cases and among controls we used all data from admission till discharge. The benchmarks were aggregated over all 48 h time windows [N=748,380 associated with 27,137 patient encounters]. The model achieved an area under the receiver operating characteristic curve of 0.993 (95% CI = [0.990, 0.996]), the enriched positive predictive value (PPV) was 23 times greater than the base prevalence. Conversely, prediction by PELOD-2 achieved a lower PPV of 1.5% [0.9%, 2.1%] which was 5 times the baseline prevalence. Conclusion: A deep learning model that employs common clinical features in the electronic health record can help predict the onset of CLABSI in hospitalized children with central venous line 48 hours prior to the time of specimen collection.

Early Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review

Background: Sepsis is among the leading causes of death in intensive care units (ICUs) worldwide and its recognition, particularly in the early stages of the disease, remains a medical challenge. The advent of an affluence of available digital health data has created a setting in which machine learning can be used for digital biomarker discovery, with the ultimate goal to advance the early recognition of sepsis. Objective: To systematically review and evaluate studies employing machine learning for the prediction of sepsis in the ICU. Data Sources: Using Embase, Google Scholar, PubMed/Medline, Scopus, and Web of Science, we systematically searched the existing literature for machine learning-driven sepsis onset prediction for patients in the ICU. Study Eligibility Criteria: All peer-reviewed articles using machine learning for the prediction of sepsis onset in adult ICU patients were included. Studies focusing on patient populations outside the ICU were excluded. Study Appraisal and Synthesis Methods: A systematic review was performed according to the PRISMA guidelines. Moreover, a quality assessment of all eligible studies was performed. Results: Out of 974 identified articles, 22 and 21 met the criteria to be included in the systematic review and quality assessment, respectively. A multitude of machine learning algorithms were applied to refine the early prediction of sepsis. The quality of the studies ranged from "poor" (satisfying ≤ 40% of the quality criteria) to "very good" (satisfying ≥ 90% of the quality criteria). The majority of the studies (n = 19, 86.4%) employed an offline training scenario combined with a horizon evaluation, while two studies implemented an online scenario (n = 2, 9.1%). The massive inter-study heterogeneity in terms of model development, sepsis definition, prediction time windows, and outcomes precluded a meta-analysis. Last, only two studies provided publicly accessible source code and data sources fostering reproducibility. Limitations: Articles were only eligible for inclusion when employing machine learning algorithms for the prediction of sepsis onset in the ICU. This restriction led to the exclusion of studies focusing on the prediction of septic shock, sepsis-related mortality, and patient populations outside the ICU. Conclusions and Key Findings: A growing number of studies employs machine learning to optimize the early prediction of sepsis through digital biomarker discovery. This review, however, highlights several shortcomings of the current approaches, including low comparability and reproducibility. Finally, we gather recommendations how these challenges can be addressed before deploying these models in prospective analyses. Systematic Review Registration Number: CRD42020200133.

Outcomes prediction in longitudinal data: Study designs evaluation, use case in ICU acquired sepsis

Outcomes' prediction in Electronic Health Records (EHR) and specifically in Critical Care is increasingly attracting more exploration and research. In this study, we used clinical data from the Intensive Care Unit (ICU), focusing on ICU acquired sepsis. Looking at the current literature, several evaluation approaches are reported, inspired by epidemiological designs, in which some do not always reflect real-life application's conditions. This problem seems relevant generally to outcomes' prediction in longitudinal EHR data, or generally longitudinal data, while in this study we focused on ICU data. Unlike in most previous studies that investigated all sepsis admissions, we focused specifically on ICU-Acquired Sepsis. Due to the sparse nature of the longitudinal data, we employed the use of Temporal Abstraction and Time Interval-Related Patterns discovery, which are further used as classification features. Two experiments were designed using three different outcomes prediction study designs from the literature, implementing various levels of real-life conditions to evaluate the prediction models. The first experiment focused on predicting whether a patient would suffer from ICU-acquired sepsis and when during her admission, given a sliding observation time window, and the comparison of the three study designs behavior. The second experiment focused only on predicting whether the patient will suffer from ICU-acquired sepsis, based on data taken relatively to his admission start time. Our results show that using Temporal Discretization for Classification (TD4C) led to better performance than using the Equal-Width Discretization, Knowledge-Based, or SAX. Also, using two states abstraction was better than three or four. Using the default Binary TIRP representation method performed better than Mean Duration, Horizontal Support, and horizontally normalized horizontal support. Using XGBoost as a classifier performed better than Logistic Regression, Neural Net, or Random Forest. Additionally, it is demonstrated why the use of case-crossover-control is most appropriate for real life application conditions evaluation, unlike other incomplete designs that may even result in "better performance".

HeMA: A hierarchically enriched machine learning approach for managing false alarms in real time: A sepsis prediction case study

Early detection of sepsis can be life-saving. Machine learning models have shown great promise in early sepsis prediction when applied to patient physiological data in real-time. However, these existing models often under-perform in terms of positive predictive value, an important metric in clinical settings. This is especially the case when the models are applied to data with less than 50% sepsis prevalence, reflective of the incidence rate of sepsis on the floor or in the ICU. In this study, we develop HeMA, a hierarchically enriched machine learning approach for managing false alarms in real time, and conduct a case study for early sepsis prediction. Specifically, we develop a two-stage framework, where a first stage machine learning model is paired with statistical tests, particularly Kolmogorov-Smirnov tests, in the second stage, to predict whether a patient would develop sepsis. Compared with machine learning models alone, the framework results in an increase in specificity and positive predictive value, without compromising F1 score. In particular, the framework shows improved performance when applied to data with 50% and 25% sepsis prevalence, collected from a large hospital system in the US, resulting in up to 18% and 7% increase in specificity and positive predictive value, respectively. Despite the significant improvements observed, and although F1 score is not negatively affected, because of the up to 6% decrease in sensitivity, further improvements and pilot studies may be necessary before deploying the framework in a clinical setting. Finally, external validation conducted using a publicly available dataset produces similar results, validating that the proposed framework is generalizable.

Early Detection of Sepsis With Machine Learning Techniques: A Brief Clinical Perspective

Sepsis is a major cause of death worldwide. Over the past years, prediction of clinically relevant events through machine learning models has gained particular attention. In the present perspective, we provide a brief, clinician-oriented vision on the following relevant aspects concerning the use of machine learning predictive models for the early detection of sepsis in the daily practice: (i) the controversy of sepsis definition and its influence on the development of prediction models; (ii) the choice and availability of input features; (iii) the measure of the model performance, the output, and their usefulness in the clinical practice. The increasing involvement of artificial intelligence and machine learning in health care cannot be disregarded, despite important pitfalls that should be always carefully taken into consideration. In the long run, a rigorous multidisciplinary approach to enrich our understanding in the application of machine learning techniques for the early recognition of sepsis may show potential to augment medical decision-making when facing this heterogeneous and complex syndrome.

Unpacking Prevalence and Dichotomy in Quick Sequential Organ Failure Assessment and Systemic Inflammatory Response Syndrome Parameters: Observational Data-Driven Approach Backed by Sepsis Pathophysiology

BACKGROUND

Considering morbidity, mortality, and annual treatment costs, the dramatic rise in the incidence of sepsis and septic shock among intensive care unit (ICU) admissions in US hospitals is an increasing concern. Recent changes in the sepsis definition (sepsis-3), based on the quick Sequential Organ Failure Assessment (qSOFA), have motivated the international medical informatics research community to investigate score recalculation and information retrieval, and to study the intersection between sepsis-3 and the previous definition (sepsis-2) based on systemic inflammatory response syndrome (SIRS) parameters.

OBJECTIVE

The objective of this study was three-fold. First, we aimed to unpack the most prevalent criterion for sepsis (for both sepsis-3 and sepsis-2 predictors). Second, we intended to determine the most prevalent sepsis scenario in the ICU among 4 possible scenarios for qSOFA and 11 possible scenarios for SIRS. Third, we investigated the multicollinearity or dichotomy among qSOFA and SIRS predictors.

METHODS

This observational study was conducted according to the most recent update of Medical Information Mart for Intensive Care (MIMIC-III, Version 1.4), the critical care database developed by MIT. The qSOFA (sepsis-3) and SIRS (sepsis-2) parameters were analyzed for patients admitted to critical care units from 2001 to 2012 in Beth Israel Deaconess Medical Center (Boston, MA, USA) to determine the prevalence and underlying relation between these parameters among patients undergoing sepsis screening. We adopted a multiblind Delphi method to seek a rationale for decisions in several stages of the research design regarding handling missing data and outlier values, statistical imputations and biases, and generalizability of the study.

RESULTS

Altered mental status in the Glasgow Coma Scale (59.28%, 38,854/65,545 observations) was the most prevalent sepsis-3 (qSOFA) criterion and the white blood cell count (53.12%, 17,163/32,311 observations) was the most prevalent sepsis-2 (SIRS) criterion confronted in the ICU. In addition, the two-factored sepsis criterion of high respiratory rate (≥22 breaths/minute) and altered mental status (28.19%, among four possible qSOFA scenarios besides no sepsis) was the most prevalent sepsis-3 (qSOFA) scenario, and the three-factored sepsis criterion of tachypnea, high heart rate, and high white blood cell count (12.32%, among 11 possible scenarios besides no sepsis) was the most prevalent sepsis-2 (SIRS) scenario in the ICU. Moreover, the absolute Pearson correlation coefficients were not significant, thereby nullifying the likelihood of any linear correlation among the critical parameters and assuring the lack of multicollinearity between the parameters. Although this further bolsters evidence for their dichotomy, the absence of multicollinearity cannot guarantee that two random variables are statistically independent.

CONCLUSIONS

Quantifying the prevalence of the qSOFA criteria of sepsis-3 in comparison with the SIRS criteria of sepsis-2, and understanding the underlying dichotomy among these parameters provides significant inferences for sepsis treatment initiatives in the ICU and informing hospital resource allocation. These data-driven results further offer design implications for multiparameter intelligent sepsis prediction in the ICU.

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy

PURPOSE

Early clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis.

METHODS

A systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance.

RESULTS

After screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68-0.99 in the ICU, to 0.96-0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance.

CONCLUSION

This systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside.

Machine learning in infection management using routine electronic health records: tools, techniques, and reporting of future technologies

BACKGROUND

Machine learning (ML) is increasingly being used in many areas of health care. Its use in infection management is catching up as identified in a recent review in this journal. We present here a complementary review to this work.

OBJECTIVES

To support clinicians and researchers in navigating through the methodological aspects of ML approaches in the field of infection management.

SOURCES

A Medline search was performed with the keywords artificial intelligence, machine learning, infection∗, and infectious disease∗ for the years 2014-2019. Studies using routinely available electronic hospital record data from an inpatient setting with a focus on bacterial and fungal infections were included.

CONTENT

Fifty-two studies were included and divided into six groups based on their focus. These studies covered detection/prediction of sepsis (n = 19), hospital-acquired infections (n = 11), surgical site infections and other postoperative infections (n = 11), microbiological test results (n = 4), infections in general (n = 2), musculoskeletal infections (n = 2), and other topics (urinary tract infections, deep fungal infections, antimicrobial prescriptions; n = 1 each). In total, 35 different ML techniques were used. Logistic regression was applied in 18 studies followed by random forest, support vector machines, and artificial neural networks in 18, 12, and seven studies, respectively. Overall, the studies were very heterogeneous in their approach and their reporting. Detailed information on data handling and software code was often missing. Validation on new datasets and/or in other institutions was rarely done. Clinical studies on the impact of ML in infection management were lacking.

IMPLICATIONS

Promising approaches for ML use in infectious diseases were identified. But building trust in these new technologies will require improved reporting. Explainability and interpretability of the models used were rarely addressed and should be further explored. Independent model validation and clinical studies evaluating the added value of ML approaches are needed.

Clinical applications of artificial intelligence in sepsis: A narrative review

Many studies have been published on a variety of clinical applications of artificial intelligence (AI) for sepsis, while there is no overview of the literature. The aim of this review is to give an overview of the literature and thereby identify knowledge gaps and prioritize areas with high priority for further research. A literature search was conducted in PubMed from inception to February 2019. Search terms related to AI were combined with terms regarding sepsis. Articles were included when they reported an area under the receiver operator characteristics curve (AUROC) as outcome measure. Fifteen articles on diagnosis of sepsis with AI models were included. The best performing model reached an AUROC of 0.97. There were also seven articles on prognosis, predicting mortality over time with an AUROC of up to 0.895. Finally, there were three articles on assistance of treatment of sepsis, where the use of AI was associated with the lowest mortality rates. Of the articles, twenty-two were judged to be at high risk of bias or had major concerns regarding applicability. This was mostly because predictor variables in these models, such as blood pressure, were also part of the definition of sepsis, which led to overestimation of the performance. We conclude that AI models have great potential for improving early identification of patients who may benefit from administration of antibiotics. Current AI prediction models to diagnose sepsis are at major risks of bias when the diagnosis criteria are part of the predictor variables in the model. Furthermore, generalizability of these models is poor due to overfitting and a lack of standardized protocols for the construction and validation of the models. Until these problems have been resolved, a large gap remains between the creation of an AI algorithm and its implementation in clinical practice.