Discharge recommendation based on a novel technique of homeostatic analysis

Prediction of respiratory decompensation in Covid-19 patients using machine learning: The READY trial

Background

Currently, physicians are limited in their ability to provide an accurate prognosis for COVID-19 positive patients. Existing scoring systems have been ineffective for identifying patient decompensation. Machine learning (ML) may offer an alternative strategy. A prospectively validated method to predict the need for ventilation in COVID-19 patients is essential to help triage patients, allocate resources, and prevent emergency intubations and their associated risks.

Methods

In a multicenter clinical trial, we evaluated the performance of a machine learning algorithm for prediction of invasive mechanical ventilation of COVID-19 patients within 24 h of an initial encounter. We enrolled patients with a COVID-19 diagnosis who were admitted to five United States health systems between March 24 and May 4, 2020.

Results

197 patients were enrolled in the REspirAtory Decompensation and model for the triage of covid-19 patients: a prospective studY (READY) clinical trial. The algorithm had a higher diagnostic odds ratio (DOR, 12.58) for predicting ventilation than a comparator early warning system, the Modified Early Warning Score (MEWS). The algorithm also achieved significantly higher sensitivity (0.90) than MEWS, which achieved a sensitivity of 0.78, while maintaining a higher specificity (p < 0.05).

Conclusions

In the first clinical trial of a machine learning algorithm for ventilation needs among COVID-19 patients, the algorithm demonstrated accurate prediction of the need for mechanical ventilation within 24 h. This algorithm may help care teams effectively triage patients and allocate resources. Further, the algorithm is capable of accurately identifying 16% more patients than a widely used scoring system while minimizing false positive results.

•

Validation of prediction algorithm for ventilation requirements in COVID-19 patients.

•

Algorithm achieved significantly higher sensitivity than the common scoring system MEWS.

•

Algorithm detected 16% more patients who will require invasive ventilation than MEWS.

•

Advance warning of ventilation needs can help improve COVID-19 patient outcomes.

The development an artificial intelligence algorithm for early sepsis diagnosis in the intensive care unit

BACKGROUND

Severe sepsis and septic shock are still the leading causes of death in Intensive Care Units (ICUs), and timely diagnosis is crucial for treatment outcomes. The progression of electronic medical records (EMR) offers the possibility of storing a large quantity of clinical data that can facilitate the development of artificial intelligence (AI) in medicine. However, several difficulties, such as poor structure and heterogenicity of the raw EMR data, are encountered when introducing AI with ICU data. Labor-intensive work, including manual data entry, personal medical records sorting, and laboratory results interpretation may hinder the progress of AI. In this article, we introduce the developing of an AI algorithm designed for sepsis diagnosis using pre-selected features; and compare the performance of the AI algorithm with SOFA score based diagnostic method.

MATERIALS AND METHODS

This is a prospective open-label cohort study. A specialized EMR, named TED_ICU, was implemented for continuous data recording. One hundred six clinical features relevant to sepsis diagnosis were selected prospectively. A labeling work to allocate SEPSIS or NON_SEPSIS status for each ICU patient was performed by the in-charge intensivist according to SEPSIS-3 criteria, along with the automatic recording of selected features every day by TED_ICU. Afterward, we use de-identified data to develop the AI algorithm. Several machine learning methods were evaluated using 5-fold cross-validation, and XGBoost, a decision-tree based algorithm was adopted for our AI algorithm development due to best performance.

RESULTS

The study was conducted between August 2018 and December 2018 for the first stage of analysis. We collected 1588 instances, including 444 SEPSIS and 1144 NON-SEPSIS, from 434 patients. The 434 patients included 259 (59.6%) male patients and 175 female patients. The mean age was 67.6-year-old, and the mean APACHE II score was 13.8. The SEPSIS cohort had a higher SOFA score and increased use of organ support treatment. The AI algorithm was developed with a shuffle method using 80% of the instances for training and 20% for testing. The established AI algorithm achieved the following: accuracy = 82% ± 1%; sensitivity = 65% ± 5%; specificity = 88% ± 2%; precision = 67% ± 3%; and F1 = 0.66 ± 0.02. The area under the receiver operating characteristic curve (AUROC) was approximately 0.89. The SOFA score was used on the same 1588 instances for sepsis diagnosis, and the result was inferior to our AI algorithm (AUROC = 0.596).

CONCLUSION

Using real-time data, collected by EMR, from the ICU daily practice, our AI algorithm established with pre-selected features and XGBoost can provide a timely diagnosis of sepsis with an accuracy greater than 80%. AI algorithm also outperforms the SOFA score in sepsis diagnosis and exhibits practicality as clinicians can deploy appropriate treatment earlier. The early and precise response of this AI algorithm will result in cost reduction, outcome improvement, and benefit for healthcare systems, medical staff, and patients as well.

Incorporating repeated measurements into prediction models in the critical care setting: a framework, systematic review and meta-analysis

BACKGROUND

The incorporation of repeated measurements into multivariable prediction research may greatly enhance predictive performance. However, the methodological possibilities vary widely and a structured overview of the possible and utilized approaches lacks. Therefore, we [1] propose a structured framework for these approaches, [2] determine what methods are currently used to incorporate repeated measurements in prediction research in the critical care setting and, where possible, [3] assess the added discriminative value of incorporating repeated measurements.

METHODS

The proposed framework consists of three domains: the observation window (static or dynamic), the processing of the raw data (raw data modelling, feature extraction and reduction) and the type of modelling. A systematic review was performed to identify studies which incorporate repeated measurements to predict (e.g. mortality) in the critical care setting. The within-study difference in c-statistics between models with versus without repeated measurements were obtained and pooled in a meta-analysis.

RESULTS

From the 2618 studies found, 29 studies incorporated multiple repeated measurements. The annual number of studies with repeated measurements increased from 2.8/year (2000-2005) to 16.0/year (2016-2018). The majority of studies that incorporated repeated measurements for prediction research used a dynamic observation window, and extracted features directly from the data. Differences in c statistics ranged from - 0.048 to 0.217 in favour of models that utilize repeated measurements.

CONCLUSIONS

Repeated measurements are increasingly common to predict events in the critical care domain, but their incorporation is lagging. A framework of possible approaches could aid researchers to optimize future prediction models.

Predictive modeling in urgent care: a comparative study of machine learning approaches

OBJECTIVE

The growing availability of rich clinical data such as patients' electronic health records provide great opportunities to address a broad range of real-world questions in medicine. At the same time, artificial intelligence and machine learning (ML)-based approaches have shown great premise on extracting insights from those data and helping with various clinical problems. The goal of this study is to conduct a systematic comparative study of different ML algorithms for several predictive modeling problems in urgent care.

DESIGN

We assess the performance of 4 benchmark prediction tasks (eg mortality and prediction, differential diagnostics, and disease marker discovery) using medical histories, physiological time-series, and demographics data from the Medical Information Mart for Intensive Care (MIMIC-III) database.

MEASUREMENTS

For each given task, performance was estimated using standard measures including the area under the receiver operating characteristic (AUC) curve, F-1 score, sensitivity, and specificity. Microaveraged AUC was used for multiclass classification models.

RESULTS AND DISCUSSION

Our results suggest that recurrent neural networks show the most promise in mortality prediction where temporal patterns in physiologic features alone can capture in-hospital mortality risk (AUC > 0.90). Temporal models did not provide additional benefit compared to deep models in differential diagnostics. When comparing the training-testing behaviors of readmission and mortality models, we illustrate that readmission risk may be independent of patient stability at discharge. We also introduce a multiclass prediction scheme for length of stay which preserves sensitivity and AUC with outliers of increasing duration despite decrease in sample size.

Emerging Technologies for Molecular Diagnosis of Sepsis

Rapid and accurate profiling of infection-causing pathogens remains a significant challenge in modern health care. Despite advances in molecular diagnostic techniques, blood culture analysis remains the gold standard for diagnosing sepsis. However, this method is too slow and cumbersome to significantly influence the initial management of patients. The swift initiation of precise and targeted antibiotic therapies depends on the ability of a sepsis diagnostic test to capture clinically relevant organisms along with antimicrobial resistance within 1 to 3 h. The administration of appropriate, narrow-spectrum antibiotics demands that such a test be extremely sensitive with a high negative predictive value. In addition, it should utilize small sample volumes and detect polymicrobial infections and contaminants. All of this must be accomplished with a platform that is easily integrated into the clinical workflow. In this review, we outline the limitations of routine blood culture testing and discuss how emerging sepsis technologies are converging on the characteristics of the ideal sepsis diagnostic test. We include seven molecular technologies that have been validated on clinical blood specimens or mock samples using human blood. In addition, we discuss advances in machine learning technologies that use electronic medical record data to provide contextual evaluation support for clinical decision-making.

Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU

OBJECTIVES

We validate a machine learning-based sepsis-prediction algorithm (InSight) for the detection and prediction of three sepsis-related gold standards, using only six vital signs. We evaluate robustness to missing data, customisation to site-specific data using transfer learning and generalisability to new settings.

DESIGN

A machine-learning algorithm with gradient tree boosting. Features for prediction were created from combinations of six vital sign measurements and their changes over time.

SETTING

A mixed-ward retrospective dataset from the University of California, San Francisco (UCSF) Medical Center (San Francisco, California, USA) as the primary source, an intensive care unit dataset from the Beth Israel Deaconess Medical Center (Boston, Massachusetts, USA) as a transfer-learning source and four additional institutions' datasets to evaluate generalisability.

PARTICIPANTS

684 443 total encounters, with 90 353 encounters from June 2011 to March 2016 at UCSF.

INTERVENTIONS

None.

PRIMARY AND SECONDARY OUTCOME MEASURES

Area under the receiver operating characteristic (AUROC) curve for detection and prediction of sepsis, severe sepsis and septic shock.

RESULTS

For detection of sepsis and severe sepsis, InSight achieves an AUROC curve of 0.92 (95% CI 0.90 to 0.93) and 0.87 (95% CI 0.86 to 0.88), respectively. Four hours before onset, InSight predicts septic shock with an AUROC of 0.96 (95% CI 0.94 to 0.98) and severe sepsis with an AUROC of 0.85 (95% CI 0.79 to 0.91).

CONCLUSIONS

InSight outperforms existing sepsis scoring systems in identifying and predicting sepsis, severe sepsis and septic shock. This is the first sepsis screening system to exceed an AUROC of 0.90 using only vital sign inputs. InSight is robust to missing data, can be customised to novel hospital data using a small fraction of site data and retains strong discrimination across all institutions.

Prediction of early unplanned intensive care unit readmission in a UK tertiary care hospital: a cross-sectional machine learning approach

OBJECTIVES

Unplanned readmissions to the intensive care unit (ICU) are highly undesirable, increasing variance in care, making resource planning difficult and potentially increasing length of stay and mortality in some settings. Identifying patients who are likely to suffer unplanned ICU readmission could reduce the frequency of this adverse event.

SETTING

A single academic, tertiary care hospital in the UK.

PARTICIPANTS

A set of 3326 ICU episodes collected between October 2014 and August 2016. All records were of patients who visited an ICU at some point during their stay. We excluded patients who were ≤16 years of age; visited ICUs other than the general and neurosciences ICU; were missing crucial electronic patient record measurements; or had indeterminate ICU discharge outcomes or very early or extremely late discharge times. After exclusion, 2018 outcome-labelled episodes remained.

PRIMARY AND SECONDARY OUTCOME MEASURES

Area under the receiver operating characteristic curve (AUROC) for prediction of unplanned ICU readmission or in-hospital death within 48 hours of first ICU discharge.

RESULTS

In 10-fold cross-validation, an ensemble predictor was trained on data from both the target hospital and the Medical Information Mart for Intensive Care (MIMIC-III) database and tested on the target hospital's data. This predictor discriminated between patients with the unplanned ICU readmission or death outcome and those without this outcome, attaining mean AUROC of 0.7095 (SE 0.0260), superior to the purpose-built Stability and Workload Index for Transfer (SWIFT) score (AUROC=0.6082, SE 0.0249; p=0.014, pairwise t-test).

CONCLUSIONS

Despite the inherent difficulties, we demonstrate that a novel machine learning algorithm based on transfer learning could achieve good discrimination, over and above that of the treating clinicians or the value added by the SWIFT score. Accurate prediction of unplanned readmission could be used to target resources more efficiently.

Using Transfer Learning for Improved Mortality Prediction in a Data-Scarce Hospital Setting

Algorithm–based clinical decision support (CDS) systems associate patient-derived health data with outcomes of interest, such as in-hospital mortality. However, the quality of such associations often depends on the availability of site-specific training data. Without sufficient quantities of data, the underlying statistical apparatus cannot differentiate useful patterns from noise and, as a result, may underperform. This initial training data burden limits the widespread, out-of-the-box, use of machine learning–based risk scoring systems. In this study, we implement a statistical transfer learning technique, which uses a large “source” data set to drastically reduce the amount of data needed to perform well on a “target” site for which training data are scarce. We test this transfer technique with AutoTriage, a mortality prediction algorithm, on patient charts from the Beth Israel Deaconess Medical Center (the source) and a population of 48 249 adult inpatients from University of California San Francisco Medical Center (the target institution). We find that the amount of training data required to surpass 0.80 area under the receiver operating characteristic (AUROC) on the target set decreases from more than 4000 patients to fewer than 220. This performance is superior to the Modified Early Warning Score (AUROC: 0.76) and corresponds to a decrease in clinical data collection time from approximately 6 months to less than 10 days. Our results highlight the usefulness of transfer learning in the specialization of CDS systems to new hospital sites, without requiring expensive and time-consuming data collection efforts.

Cost and mortality impact of an algorithm-driven sepsis prediction system

AIMS

To compute the financial and mortality impact of InSight, an algorithm-driven biomarker, which forecasts the onset of sepsis with minimal use of electronic health record data.

METHODS

This study compares InSight with existing sepsis screening tools and computes the differential life and cost savings associated with its use in the inpatient setting. To do so, mortality reduction is obtained from an increase in the number of sepsis cases correctly identified by InSight. Early sepsis detection by InSight is also associated with a reduction in length-of-stay, from which cost savings are directly computed.

RESULTS

InSight identifies more true positive cases of severe sepsis, with fewer false alarms, than comparable methods. For an individual ICU with 50 beds, for example, it is determined that InSight annually saves 75 additional lives and reduces sepsis-related costs by $560,000.

LIMITATIONS

InSight performance results are derived from analysis of a single-center cohort. Mortality reduction results rely on a simplified use case, which fixes prediction times at 0, 1, and 2 h before sepsis onset, likely leading to under-estimates of lives saved. The corresponding cost reduction numbers are based on national averages for daily patient length-of-stay cost.

CONCLUSIONS

InSight has the potential to reduce sepsis-related deaths and to lead to substantial cost savings for healthcare facilities.

Using electronic health record collected clinical variables to predict medical intensive care unit mortality

BACKGROUND

Clinical decision support systems are used to help predict patient stability and mortality in the Intensive Care Unit (ICU). Accurate patient information can assist clinicians with patient management and in allocating finite resources. However, systems currently in common use have limited predictive value in the clinical setting. The increasing availability of Electronic Health Records (EHR) provides an opportunity to use medical information for more accurate patient stability and mortality prediction in the ICU.

OBJECTIVE

Develop and evaluate an algorithm which more accurately predicts patient mortality in the ICU, using the correlations between widely available clinical variables from the EHR.

METHODS

We have developed an algorithm, AutoTriage, which uses eight common clinical variables from the EHR to assign patient mortality risk scores. Each clinical variable produces a subscore, and combinations of two or three discretized clinical variables also produce subscores. A combination of weighted subscores produces the overall score. We validated the performance of this algorithm in a retrospective study on the MIMIC III medical ICU dataset.

RESULTS

AutoTriage 12 h mortality prediction yields an Area Under Receiver Operating Characteristic value of 0.88 (95% confidence interval 0.86 to 0.88). At a sensitivity of 80%, AutoTriage maintains a specificity of 81% with a diagnostic odds ratio of 16.26.

CONCLUSIONS

Through the multidimensional analysis of the correlations between eight common clinical variables, AutoTriage provides an improvement in the specificity and sensitivity of patient mortality prediction over existing prediction methods.

Prediction of Sepsis in the Intensive Care Unit With Minimal Electronic Health Record Data: A Machine Learning Approach

Background

Sepsis is one of the leading causes of mortality in hospitalized patients. Despite this fact, a reliable means of predicting sepsis onset remains elusive. Early and accurate sepsis onset predictions could allow more aggressive and targeted therapy while maintaining antimicrobial stewardship. Existing detection methods suffer from low performance and often require time-consuming laboratory test results.

Objective

To study and validate a sepsis prediction method, InSight, for the new Sepsis-3 definitions in retrospective data, make predictions using a minimal set of variables from within the electronic health record data, compare the performance of this approach with existing scoring systems, and investigate the effects of data sparsity on InSight performance.

Methods

We apply InSight, a machine learning classification system that uses multivariable combinations of easily obtained patient data (vitals, peripheral capillary oxygen saturation, Glasgow Coma Score, and age), to predict sepsis using the retrospective Multiparameter Intelligent Monitoring in Intensive Care (MIMIC)-III dataset, restricted to intensive care unit (ICU) patients aged 15 years or more. Following the Sepsis-3 definitions of the sepsis syndrome, we compare the classification performance of InSight versus quick sequential organ failure assessment (qSOFA), modified early warning score (MEWS), systemic inflammatory response syndrome (SIRS), simplified acute physiology score (SAPS) II, and sequential organ failure assessment (SOFA) to determine whether or not patients will become septic at a fixed period of time before onset. We also test the robustness of the InSight system to random deletion of individual input observations.

Results

In a test dataset with 11.3% sepsis prevalence, InSight produced superior classification performance compared with the alternative scores as measured by area under the receiver operating characteristic curves (AUROC) and area under precision-recall curves (APR). In detection of sepsis onset, InSight attains AUROC = 0.880 (SD 0.006) at onset time and APR = 0.595 (SD 0.016), both of which are superior to the performance attained by SIRS (AUROC: 0.609; APR: 0.160), qSOFA (AUROC: 0.772; APR: 0.277), and MEWS (AUROC: 0.803; APR: 0.327) computed concurrently, as well as SAPS II (AUROC: 0.700; APR: 0.225) and SOFA (AUROC: 0.725; APR: 0.284) computed at admission (P<.001 for all comparisons). Similar results are observed for 1-4 hours preceding sepsis onset. In experiments where approximately 60% of input data are deleted at random, InSight attains an AUROC of 0.781 (SD 0.013) and APR of 0.401 (SD 0.015) at sepsis onset time. Even with 60% of data missing, InSight remains superior to the corresponding SIRS scores (AUROC and APR, P<.001), qSOFA scores (P=.0095; P<.001) and superior to SOFA and SAPS II computed at admission (AUROC and APR, P<.001), where all of these comparison scores (except InSight) are computed without data deletion.

Conclusions

Despite using little more than vitals, InSight is an effective tool for predicting sepsis onset and performs well even with randomly missing data.

High-performance detection and early prediction of septic shock for alcohol-use disorder patients

Background

The presence of Alcohol Use Disorder (AUD) complicates the medical conditions of patients and increases the difficulty of detecting and predicting the onset of septic shock for patients in the ICU.

Methods

We have developed a high-performance sepsis prediction algorithm, InSight, which outperforms existing methods for AUD patient populations. InSight analyses a combination of singlets, doublets, and triplets of clinical measurements over time to generate a septic shock risk score. AUD patients obtained from the MIMIC III database were used in this retrospective study to train InSight and compare performance with the Modified Early Warning Score (MEWS), the Simplified Acute Physiology Score (SAPS II), and the Systemic Inflammatory Response Syndrome (SIRS) for septic shock prediction and detection.

Results

From 4-fold cross validation, InSight performs particularly well on diagnostic odds ratio and demonstrates a relatively high Area Under the Receiver Operating Characteristic (AUROC) metric. Four hours prior to onset, InSight had an average AUROC of 0.815, and at the time of onset, InSight had an average AUROC value of 0.965. When applied to patient populations where AUD may complicate prediction methods of sepsis, InSight outperforms existing diagnostic tools.

Conclusions

Analysis of the higher order correlations and trends between relevant clinical measurements using the InSight algorithm leads to more accurate detection and prediction of septic shock, even in cases where diagnosis may be confounded by AUD.

•

At 93% sensitivity, InSight reduces false alarms by >80% over other detection tools.

•

InSight's diagnostic odds ratio is >30X those of MEWS, SAPS II, SIRS for detection.

•

InSight outperforms comparable methods for septic shock prediction hours before onset.