Cost and mortality impact of an algorithm-driven sepsis prediction system

The value of artificial intelligence for the treatment of mechanically ventilated intensive care unit patients: An early health technology assessment

PURPOSE

The health and economic consequences of artificial intelligence (AI) systems for mechanically ventilated intensive care unit patients often remain unstudied. Early health technology assessments (HTA) can examine the potential impact of AI systems by using available data and simulations. Therefore, we developed a generic health-economic model suitable for early HTA of AI systems for mechanically ventilated patients.

MATERIALS AND METHODS

Our generic health-economic model simulates mechanically ventilated patients from their hospitalisation until their death. The model simulates two scenarios, care as usual and care with the AI system, and compares these scenarios to estimate their cost-effectiveness.

RESULTS

The generic health-economic model we developed is suitable for estimating the cost-effectiveness of various AI systems. By varying input parameters and assumptions, the model can examine the cost-effectiveness of AI systems across a wide range of different clinical settings.

CONCLUSIONS

Using the proposed generic health-economic model, investors and innovators can easily assess whether implementing a certain AI system is likely to be cost-effective before an exact clinical impact is determined. The results of the early HTA can aid investors and innovators in deployment of AI systems by supporting development decisions, informing value-based pricing, clinical trial design, and selection of target patient groups.

Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework

OBJECTIVE

To retrieve and appraise studies of deployed artificial intelligence (AI)-based sepsis prediction algorithms using systematic methods, identify implementation barriers, enablers, and key decisions and then map these to a novel end-to-end clinical AI implementation framework.

MATERIALS AND METHODS

Systematically review studies of clinically applied AI-based sepsis prediction algorithms in regard to methodological quality, deployment and evaluation methods, and outcomes. Identify contextual factors that influence implementation and map these factors to the SALIENT implementation framework.

RESULTS

The review identified 30 articles of algorithms applied in adult hospital settings, with 5 studies reporting significantly decreased mortality post-implementation. Eight groups of algorithms were identified, each sharing a common algorithm. We identified 14 barriers, 26 enablers, and 22 decision points which were able to be mapped to the 5 stages of the SALIENT implementation framework.

DISCUSSION

Empirical studies of deployed sepsis prediction algorithms demonstrate their potential for improving care and reducing mortality but reveal persisting gaps in existing implementation guidance. In the examined publications, key decision points reflecting real-word implementation experience could be mapped to the SALIENT framework and, as these decision points appear to be AI-task agnostic, this framework may also be applicable to non-sepsis algorithms. The mapping clarified where and when barriers, enablers, and key decisions arise within the end-to-end AI implementation process.

CONCLUSIONS

A systematic review of real-world implementation studies of sepsis prediction algorithms was used to validate an end-to-end staged implementation framework that has the ability to account for key factors that warrant attention in ensuring successful deployment, and which extends on previous AI implementation frameworks.

Early prediction of central line associated bloodstream infection using machine learning

BACKGROUND

Central line-associated bloodstream infections (CLABSIs) are associated with significant morbidity, mortality, and increased healthcare costs. Despite the high prevalence of CLABSIs in the U.S., there are currently no tools to stratify a patient's risk of developing an infection as the result of central line placement. To this end, we have developed and validated a machine learning algorithm (MLA) that can predict a patient's likelihood of developing CLABSI using only electronic health record data in order to provide clinical decision support.

METHODS

We created three machine learning models to retrospectively analyze electronic health record data from 27,619 patient encounters. The models were trained and validated using an 80:20 split for the train and test data. Patients designated as having a central line procedure based on International Statistical Classification of Diseases and Related Health Problems 10 codes were included.

RESULTS

XGBoost was the highest performing MLA out of the three models, obtaining an AUROC of 0.762 for CLABSI risk prediction at 48 hours after the recorded time for central line placement.

CONCLUSIONS

Our results demonstrate that MLAs may be effective clinical decision support tools for assessment of CLABSI risk and should be explored further for this purpose.

Prediction of diabetic kidney disease with machine learning algorithms, upon the initial diagnosis of type 2 diabetes mellitus

INTRODUCTION

Diabetic kidney disease (DKD) accounts for the majority of increased risk of mortality for patients with diabetes, and eventually manifests in approximately half of those patients diagnosed with type 2 diabetes mellitus (T2DM). Although increased screening frequency can avoid delayed diagnoses, this is not uniformly implemented. The purpose of this study was to develop and retrospectively validate a machine learning algorithm (MLA) that predicts stages of DKD within 5 years upon diagnosis of T2DM.

RESEARCH DESIGN AND METHODS

Two MLAs were trained to predict stages of DKD severity, and compared with the Centers for Disease Control and Prevention (CDC) risk score to evaluate performance. The models were validated on a hold-out test set as well as an external dataset sourced from separate facilities.

RESULTS

The MLAs outperformed the CDC risk score in both the hold-out test and external datasets. Our algorithms achieved an area under the receiver operating characteristic curve (AUROC) of 0.75 on the hold-out set for prediction of any-stage DKD and an AUROC of over 0.82 for more severe endpoints, compared with the CDC risk score with an AUROC <0.70 on all test sets and endpoints.

CONCLUSION

This retrospective study shows that an MLA can provide timely predictions of DKD among patients with recently diagnosed T2DM.

Machine Learning Model to Identify Sepsis Patients in the Emergency Department: Algorithm Development and Validation

Accurate stratification of sepsis can effectively guide the triage of patient care and shared decision making in the emergency department (ED). However, previous research on sepsis identification models focused mainly on ICU patients, and discrepancies in model performance between the development and external validation datasets are rarely evaluated. The aim of our study was to develop and externally validate a machine learning model to stratify sepsis patients in the ED. We retrospectively collected clinical data from two geographically separate institutes that provided a different level of care at different time periods. The Sepsis-3 criteria were used as the reference standard in both datasets for identifying true sepsis cases. An eXtreme Gradient Boosting (XGBoost) algorithm was developed to stratify sepsis patients and the performance of the model was compared with traditional clinical sepsis tools; quick Sequential Organ Failure Assessment (qSOFA) and Systemic Inflammatory Response Syndrome (SIRS). There were 8296 patients (1752 (21%) being septic) in the development and 1744 patients (506 (29%) being septic) in the external validation datasets. The mortality of septic patients in the development and validation datasets was 13.5% and 17%, respectively. In the internal validation, XGBoost achieved an area under the receiver operating characteristic curve (AUROC) of 0.86, exceeding SIRS (0.68) and qSOFA (0.56). The performance of XGBoost deteriorated in the external validation (the AUROC of XGBoost, SIRS and qSOFA was 0.75, 0.57 and 0.66, respectively). Heterogeneity in patient characteristics, such as sepsis prevalence, severity, age, comorbidity and infection focus, could reduce model performance. Our model showed good discriminative capabilities for the identification of sepsis patients and outperformed the existing sepsis identification tools. Implementation of the ML model in the ED can facilitate timely sepsis identification and treatment. However, dataset discrepancies should be carefully evaluated before implementing the ML approach in clinical practice. This finding reinforces the necessity for future studies to perform external validation to ensure the generalisability of any developed ML approaches.

MGP-AttTCN: An interpretable machine learning model for the prediction of sepsis

With a mortality rate of 5.4 million lives worldwide every year and a healthcare cost of more than 16 billion dollars in the USA alone, sepsis is one of the leading causes of hospital mortality and an increasing concern in the ageing western world. Recently, medical and technological advances have helped re-define the illness criteria of this disease, which is otherwise poorly understood by the medical society. Together with the rise of widely accessible Electronic Health Records, the advances in data mining and complex nonlinear algorithms are a promising avenue for the early detection of sepsis. This work contributes to the research effort in the field of automated sepsis detection with an open-access labelling of the medical MIMIC-III data set. Moreover, we propose MGP-AttTCN: a joint multitask Gaussian Process and attention-based deep learning model to early predict the occurrence of sepsis in an interpretable manner. We show that our model outperforms the current state-of-the-art and present evidence that different labelling heuristics lead to discrepancies in task difficulty. For instance, when predicting sepsis five hours prior to onset on our new realistic labels, our proposed model achieves an area under the ROC curve of 0.660 and an area under the PR curve of 0.483, whereas the (less interpretable) previous state-of-the-art model (MGP-TCN) achieves 0.635 AUROC and 0.460 AUPR and the popular commercial InSight model achieves 0.490 AUROC and 0.359 AUPR.

Economic evaluations of big data analytics for clinical decision-making: a scoping review

OBJECTIVE

Much has been invested in big data analytics to improve health and reduce costs. However, it is unknown whether these investments have achieved the desired goals. We performed a scoping review to determine the health and economic impact of big data analytics for clinical decision-making.

MATERIALS AND METHODS

We searched Medline, Embase, Web of Science and the National Health Services Economic Evaluations Database for relevant articles. We included peer-reviewed papers that report the health economic impact of analytics that assist clinical decision-making. We extracted the economic methods and estimated impact and also assessed the quality of the methods used. In addition, we estimated how many studies assessed "big data analytics" based on a broad definition of this term.

RESULTS

The search yielded 12 133 papers but only 71 studies fulfilled all eligibility criteria. Only a few papers were full economic evaluations; many were performed during development. Papers frequently reported savings for healthcare payers but only 20% also included costs of analytics. Twenty studies examined "big data analytics" and only 7 reported both cost-savings and better outcomes.

DISCUSSION

The promised potential of big data is not yet reflected in the literature, partly since only a few full and properly performed economic evaluations have been published. This and the lack of a clear definition of "big data" limit policy makers and healthcare professionals from determining which big data initiatives are worth implementing.

Clinician involvement in research on machine learning-based predictive clinical decision support for the hospital setting: A scoping review

OBJECTIVE

The study sought to describe the prevalence and nature of clinical expert involvement in the development, evaluation, and implementation of clinical decision support systems (CDSSs) that utilize machine learning to analyze electronic health record data to assist nurses and physicians in prognostic and treatment decision making (ie, predictive CDSSs) in the hospital.

MATERIALS AND METHODS

A systematic search of PubMed, CINAHL, and IEEE Xplore and hand-searching of relevant conference proceedings were conducted to identify eligible articles. Empirical studies of predictive CDSSs using electronic health record data for nurses or physicians in the hospital setting published in the last 5 years in peer-reviewed journals or conference proceedings were eligible for synthesis. Data from eligible studies regarding clinician involvement, stage in system design, predictive CDSS intention, and target clinician were charted and summarized.

RESULTS

Eighty studies met eligibility criteria. Clinical expert involvement was most prevalent at the beginning and late stages of system design. Most articles (95%) described developing and evaluating machine learning models, 28% of which described involving clinical experts, with nearly half functioning to verify the clinical correctness or relevance of the model (47%).

DISCUSSION

Involvement of clinical experts in predictive CDSS design should be explicitly reported in publications and evaluated for the potential to overcome predictive CDSS adoption challenges.

CONCLUSIONS

If present, clinical expert involvement is most prevalent when predictive CDSS specifications are made or when system implementations are evaluated. However, clinical experts are less prevalent in developmental stages to verify clinical correctness, select model features, preprocess data, or serve as a gold standard.

Mortality prediction model for the triage of COVID-19, pneumonia, and mechanically ventilated ICU patients: A retrospective study

Rationale

Prediction of patients at risk for mortality can help triage patients and assist in resource allocation.

Objectives

Develop and evaluate a machine learning-based algorithm which accurately predicts mortality in COVID-19, pneumonia, and mechanically ventilated patients.

Methods

Retrospective study of 53,001 total ICU patients, including 9166 patients with pneumonia and 25,895 mechanically ventilated patients, performed on the MIMIC dataset. An additional retrospective analysis was performed on a community hospital dataset containing 114 patients positive for SARS-COV-2 by PCR test. The outcome of interest was in-hospital patient mortality.

Results

When trained and tested on the MIMIC dataset, the XGBoost predictor obtained area under the receiver operating characteristic (AUROC) values of 0.82, 0.81, 0.77, and 0.75 for mortality prediction on mechanically ventilated patients at 12-, 24-, 48-, and 72- hour windows, respectively, and AUROCs of 0.87, 0.78, 0.77, and 0.734 for mortality prediction on pneumonia patients at 12-, 24-, 48-, and 72- hour windows, respectively. The predictor outperformed the qSOFA, MEWS and CURB-65 risk scores at all prediction windows. When tested on the community hospital dataset, the predictor obtained AUROCs of 0.91, 0.90, 0.86, and 0.87 for mortality prediction on COVID-19 patients at 12-, 24-, 48-, and 72- hour windows, respectively, outperforming the qSOFA, MEWS and CURB-65 risk scores at all prediction windows.

Conclusions

This machine learning-based algorithm is a useful predictive tool for anticipating patient mortality at clinically useful timepoints, and is capable of accurate mortality prediction for mechanically ventilated patients as well as those diagnosed with pneumonia and COVID-19.

•

Mortality predictions have not previously been evaluated for COVID-19 patients.

•

Machine learning may be a useful predictive tool for anticipating patient mortality.

•

Prediction can be estimated at clinically useful windows up to 72 h in advance.

A new analytical framework for missing data imputation and classification with uncertainty: Missing data imputation and heart failure readmission prediction

Background

The wide adoption of electronic health records (EHR) system has provided vast opportunities to advance health care services. However, the prevalence of missing values in EHR system poses a great challenge on data analysis to support clinical decision-making. The objective of this study is to develop a new methodological framework that can address the missing data challenge and provide a reliable tool to predict the hospital readmission among Heart Failure patients.

Methods

We used Gaussian Process Latent Variable Model (GPLVM) to impute the missing values. Specifically, a lower dimensional embedding was learned from a small complete dataset and then used to impute the missing values in the incomplete dataset. The GPLVM-based missing data imputation can provide both the mean estimate and the uncertainty associated with the mean estimate. To incorporate the uncertainty in prediction, a constrained support vector machine (cSVM) was developed to obtain robust predictions. We first sampled multiple datasets from the distributions of input uncertainty and trained a support vector machine for each dataset. Then an optimal classifier was identified by selecting the support vectors that maximize the separation margin of a newly sampled dataset and minimize the similarity with the pre-trained support vectors.

Results

The proposed model was derived and validated using Physionet MIMIC-III clinical database. The GPLVM imputation provided normalized mean absolute errors of 0.11 and 0.12 respectively when 20% and 30% of instances contained missing values, and the confidence bounds of the estimations captures 97% of the true values. The cSVM model provided an average Area Under Curve of 0.68, which improves the prediction accuracy by 7% as compared to some existing classifiers.

Conclusions

The proposed method provides accurate imputation of missing values and has a better prediction performance as compared to existing models that can only deal with deterministic inputs.

Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission: a prospective multicentre clinical outcomes evaluation of real-world patient data from US hospitals

BACKGROUND

Severe sepsis and septic shock are among the leading causes of death in the USA. While early prediction of severe sepsis can reduce adverse patient outcomes, sepsis remains one of the most expensive conditions to diagnose and treat.

OBJECTIVE

The purpose of this study was to evaluate the effect of a machine learning algorithm for severe sepsis prediction on in-hospital mortality, hospital length of stay and 30-day readmission.

DESIGN

Prospective clinical outcomes evaluation.

SETTING

Evaluation was performed on a multiyear, multicentre clinical data set of real-world data containing 75 147 patient encounters from nine hospitals across the continental USA, ranging from community hospitals to large academic medical centres.

PARTICIPANTS

Analyses were performed for 17 758 adult patients who met two or more systemic inflammatory response syndrome criteria at any point during their stay ('sepsis-related' patients).

INTERVENTIONS

Machine learning algorithm for severe sepsis prediction.

OUTCOME MEASURES

In-hospital mortality, length of stay and 30-day readmission rates.

RESULTS

Hospitals saw an average 39.5% reduction of in-hospital mortality, a 32.3% reduction in hospital length of stay and a 22.7% reduction in 30-day readmission rate for sepsis-related patient stays when using the machine learning algorithm in clinical outcomes analysis.

CONCLUSIONS

Reductions of in-hospital mortality, hospital length of stay and 30-day readmissions were observed in real-world clinical use of the machine learning-based algorithm. The predictive algorithm may be successfully used to improve sepsis-related outcomes in live clinical settings.

TRIAL REGISTRATION NUMBER

NCT03960203.

Clinical Significance of MicroRNAs in Patients with Sepsis: Protocol for a Systematic Review and Meta-Analysis

Sepsis is a dysregulated immune response that leads to organ dysfunction and has high mortality rates despite recent therapeutic advancements. Accurate diagnosis and risk stratification are important for effective sepsis treatment; however, no decisive diagnostic or prognostic biomarkers are currently available. To understand whether microRNA (miRNA) might be useful biomarkers of sepsis, we aim to assess the diagnostic and prognostic accuracy of three miRNAs (122, 150, and 223) in sepsis patients via a meta-analysis of relevant published data. We will search electronic bibliographic databases (MEDLINE, EMBASE, and the Cochrane Central Register of Controlled Trials) for pertinent retrospective and prospective studies in October 2019. Two reviewers will evaluate the collected titles, abstracts, and full articles, and extract the data. We will assess the included studies using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. If feasible, we will use bivariate random effects and hierarchical summary receiver operating characteristic (ROC) models to estimate summary ROCs, pooled sensitivity and specificity values, and the corresponding 95% confidence intervals. We will evaluate heterogeneity via clinical and methodological subgroup and sensitivity analyses. This systematic review will clarify the diagnostic and prognostic accuracy of select miRNAs in sepsis. It may also identify knowledge gaps in sepsis’ diagnosis and prognosis.

Predicted Economic Benefits of a Novel Biomarker for Earlier Sepsis Identification and Treatment: A Counterfactual Analysis

To estimate the potential clinical and health economic value of earlier sepsis identification in the emergency department using a novel diagnostic marker, monocyte distribution width.

Design

The analysis was conducted in two phases: 1) an analysis of the pivotal registration trial evidence to estimate the potential benefit of monocyte distribution width for early sepsis identification and (2) a cost-consequence analysis to estimate the potential economic and clinical benefits that could have resulted from earlier administration of antibiotics for those patients.

Setting

Sepsis identified in the emergency department which led to inpatient hospitalizations.

Patients

Adult sepsis patients admitted through the emergency department.

Interventions

None. This was a model simulation of clinical and economic outcomes of monocyte distribution width based on results from a noninterventional, multicenter clinical trial.

Measurements and Main Results

Among the 385 patients with sepsis, a total of 349 were eligible for inclusion. Sixty-seven percent of patients were predicted to benefit from monocyte distribution width results, resulting in an estimated mean reduction in time to antibiotics administration from 3.98 hours using standard of care to 2.07 hours using monocyte distribution width + standard of care. Based on this simulated reduction in time to antibiotics, monocyte distribution width + standard of care could have resulted in a less than or equal to 14.2% reduction (27.9% vs 32.5%) in mortality, a mean reduction of 1.48 days (10.0 vs 11.5 d) in length of stay, and $3,460 ($23,466 vs $26,926) savings per hospitalization. At the hospital level, based on an established national mean of 206 sepsis hospitalizations per hospital per year, earlier identification with monocyte distribution width is predicted to result in a total of $712,783 in annual cost savings per hospital.

Conclusions

Improved early identification of sepsis using monocyte distribution width along with current standard of care is estimated to improve both clinical and economic outcomes of sepsis patients presenting in the emergency department. Further research is warranted to confirm these model projections.

Analysis and prediction of unplanned intensive care unit readmission using recurrent neural networks with long short-term memory

BACKGROUND

Unplanned readmission of a hospitalized patient is an indicator of patients' exposure to risk and an avoidable waste of medical resources. In addition to hospital readmission, intensive care unit (ICU) readmission brings further financial risk, along with morbidity and mortality risks. Identification of high-risk patients who are likely to be readmitted can provide significant benefits for both patients and medical providers. The emergence of machine learning solutions to detect hidden patterns in complex, multi-dimensional datasets provides unparalleled opportunities for developing an efficient discharge decision-making support system for physicians and ICU specialists.

METHODS AND FINDINGS

We used supervised machine learning approaches for ICU readmission prediction. We used machine learning methods on comprehensive, longitudinal clinical data from the MIMIC-III to predict the ICU readmission of patients within 30 days of their discharge. We incorporate multiple types of features including chart events, demographic, and ICD-9 embeddings. We have utilized recent machine learning techniques such as Recurrent Neural Networks (RNN) with Long Short-Term Memory (LSTM), by this we have been able to incorporate the multivariate features of EHRs and capture sudden fluctuations in chart event features (e.g. glucose and heart rate). We show that our LSTM-based solution can better capture high volatility and unstable status in ICU patients, an important factor in ICU readmission. Our machine learning models identify ICU readmissions at a higher sensitivity rate of 0.742 (95% CI, 0.718-0.766) and an improved Area Under the Curve of 0.791 (95% CI, 0.782-0.800) compared with traditional methods. We perform in-depth deep learning performance analysis, as well as the analysis of each feature contribution to the predictive model.

CONCLUSION

Our manuscript highlights the ability of machine learning models to improve our ICU decision-making accuracy and is a real-world example of precision medicine in hospitals. These data-driven solutions hold the potential for substantial clinical impact by augmenting clinical decision-making for physicians and ICU specialists. We anticipate that machine learning models will improve patient counseling, hospital administration, allocation of healthcare resources and ultimately individualized clinical care.

Machine-Learning-Based Laboratory Developed Test for the Diagnosis of Sepsis in High-Risk Patients

Sepsis, a dysregulated host response to infection, is a major health burden in terms of both mortality and cost. The difficulties clinicians face in diagnosing sepsis, alongside the insufficiencies of diagnostic biomarkers, motivate the present study. This work develops a machine-learning-based sepsis diagnostic for a high-risk patient group, using a geographically and institutionally diverse collection of nearly 500,000 patient health records. Using only a minimal set of clinical variables, our diagnostics outperform common severity scoring systems and sepsis biomarkers and benefit from being available immediately upon ordering.

Emerging Technologies for Molecular Diagnosis of Sepsis

Rapid and accurate profiling of infection-causing pathogens remains a significant challenge in modern health care. Despite advances in molecular diagnostic techniques, blood culture analysis remains the gold standard for diagnosing sepsis. However, this method is too slow and cumbersome to significantly influence the initial management of patients. The swift initiation of precise and targeted antibiotic therapies depends on the ability of a sepsis diagnostic test to capture clinically relevant organisms along with antimicrobial resistance within 1 to 3 h. The administration of appropriate, narrow-spectrum antibiotics demands that such a test be extremely sensitive with a high negative predictive value. In addition, it should utilize small sample volumes and detect polymicrobial infections and contaminants. All of this must be accomplished with a platform that is easily integrated into the clinical workflow. In this review, we outline the limitations of routine blood culture testing and discuss how emerging sepsis technologies are converging on the characteristics of the ideal sepsis diagnostic test. We include seven molecular technologies that have been validated on clinical blood specimens or mock samples using human blood. In addition, we discuss advances in machine learning technologies that use electronic medical record data to provide contextual evaluation support for clinical decision-making.