Evaluation of ICU Risk Models Adapted for Use as Continuous Markers of Severity of Illness Throughout the ICU Stay

Machine Learning for Benchmarking Critical Care Outcomes

OBJECTIVES

Enhancing critical care efficacy involves evaluating and improving system functioning. Benchmarking, a retrospective comparison of results against standards, aids risk-adjusted assessment and helps healthcare providers identify areas for improvement based on observed and predicted outcomes. The last two decades have seen the development of several models using machine learning (ML) for clinical outcome prediction. ML is a field of artificial intelligence focused on creating algorithms that enable computers to learn from and make predictions or decisions based on data. This narrative review centers on key discoveries and outcomes to aid clinicians and researchers in selecting the optimal methodology for critical care benchmarking using ML.

METHODS

We used PubMed to search the literature from 2003 to 2023 regarding predictive models utilizing ML for mortality (592 articles), length of stay (143 articles), or mechanical ventilation (195 articles). We supplemented the PubMed search with Google Scholar, making sure relevant articles were included. Given the narrative style, papers in the cohort were manually curated for a comprehensive reader perspective.

RESULTS

Our report presents comparative results for benchmarked outcomes and emphasizes advancements in feature types, preprocessing, model selection, and validation. It showcases instances where ML effectively tackled critical care outcome-prediction challenges, including nonlinear relationships, class imbalances, missing data, and documentation variability, leading to enhanced results.

CONCLUSIONS

Although ML has provided novel tools to improve the benchmarking of critical care outcomes, areas that require further research include class imbalance, fairness, improved calibration, generalizability, and long-term validation of published models.

Prediction of in-hospital mortality among intensive care unit patients using modified daily Laboratory-based Acute Physiology Scores, version 2 (LAPS2)

Background

Mortality prediction for intensive care unit (ICU) patients frequently relies on single acuity measures based on ICU admission physiology without accounting for subsequent clinical changes.

Objectives

Evaluate novel models incorporating modified admission and daily, time-updating Laboratory-based Acute Physiology Scores, version 2 (LAPS2) to predict in-hospital mortality among ICU patients.

Research design

Retrospective cohort study.

Subjects

All ICU patients in five hospitals from October 2017 through September 2019.

Measures

We used logistic regression, penalized logistic regression, and random forest models to predict in-hospital mortality within 30 days of ICU admission using admission LAPS2 alone in patient-level and patient-day-level models, or admission and daily LAPS2 at the patient-day level. Multivariable models included patient and admission characteristics. We performed internal-external validation using four hospitals for training and the fifth for validation, repeating analyses for each hospital as the validation set. We assessed performance using scaled Brier scores (SBS), c-statistics, and calibration plots.

Results

The cohort included 13,993 patients and 120,101 ICU days. The patient-level model including the modified admission LAPS2 without daily LAPS2 had an SBS of 0.175 (95% CI 0.148-0.201) and c-statistic of 0.824 (95% CI 0.808-0.840). Patient-day-level models including daily LAPS2 consistently outperformed models with modified admission LAPS2 alone. Among patients with <50% predicted mortality, daily models were better calibrated than models with modified admission LAPS2 alone.

Conclusions

Models incorporating daily, time-updating LAPS2 to predict mortality among an ICU population perform as well or better than models incorporating modified admission LAPS2 alone.

Signatures of illness in children requiring unplanned intubation in the pediatric intensive care unit: A retrospective cohort machine-learning study

Acute respiratory failure requiring the initiation of invasive mechanical ventilation remains commonplace in the pediatric intensive care unit (PICU). Early recognition of patients at risk for respiratory failure may provide clinicians with the opportunity to intervene and potentially improve outcomes. Through the development of a random forest model to identify patients at risk for requiring unplanned intubation, we tested the hypothesis that subtle signatures of illness are present in physiological and biochemical time series of PICU patients in the early stages of respiratory decompensation. We included 116 unplanned intubation events as recorded in the National Emergency Airway Registry for Children in 92 PICU admissions over a 29-month period at our institution. We observed that children have a physiologic signature of illness preceding unplanned intubation in the PICU. Generally, it comprises younger age, and abnormalities in electrolyte, hematologic and vital sign parameters. Additionally, given the heterogeneity of the PICU patient population, we found differences in the presentation among the major patient groups - medical, cardiac surgical, and non-cardiac surgical. At four hours prior to the event, our random forest model demonstrated an area under the receiver operating characteristic curve of 0.766 (0.738 for medical, 0.755 for cardiac surgical, and 0.797 for non-cardiac surgical patients). The multivariable statistical models that captured the physiological and biochemical dynamics leading up to the event of urgent unplanned intubation in a PICU can be repurposed for bedside risk prediction.

A Machine Learning Based Discharge Prediction of Cardiovascular Diseases Patients in Intensive Care Units

This paper targets a major challenge of how to effectively allocate medical resources in intensive care units (ICUs). We trained multiple regression models using the Medical Information Mart for Intensive Care III (MIMIC III) database recorded in the period between 2001 and 2012. The training and validation dataset included pneumonia, sepsis, congestive heart failure, hypotension, chest pain, coronary artery disease, fever, respiratory failure, acute coronary syndrome, shortness of breath, seizure and transient ischemic attack, and aortic stenosis patients’ recorded data. Then we tested the models on the unseen data of patients diagnosed with coronary artery disease, congestive heart failure or acute coronary syndrome. We included the admission characteristics, clinical prescriptions, physiological measurements, and discharge characteristics of those patients. We assessed the models’ performance using mean residuals and running times as metrics. We ran multiple experiments to study the data partition’s impact on the learning phase. The total running time of our best-evaluated model is 123,450.9 mS. The best model gives an average accuracy of 98%, highlighting the location of discharge, initial diagnosis, location of admission, drug therapy, length of stay and internal transfers as the most influencing patterns to decide a patient’s readiness for discharge.

E-CatBoost: An efficient machine learning framework for predicting ICU mortality using the eICU Collaborative Research Database

Improving the Intensive Care Unit (ICU) management network and building cost-effective and well-managed healthcare systems are high priorities for healthcare units. Creating accurate and explainable mortality prediction models helps identify the most critical risk factors in the patients' survival/death status and early detect the most in-need patients. This study proposes a highly accurate and efficient machine learning model for predicting ICU mortality status upon discharge using the information available during the first 24 hours of admission. The most important features in mortality prediction are identified, and the effects of changing each feature on the prediction are studied. We used supervised machine learning models and illness severity scoring systems to benchmark the mortality prediction. We also implemented a combination of SHAP, LIME, partial dependence, and individual conditional expectation plots to explain the predictions made by the best-performing model (CatBoost). We proposed E-CatBoost, an optimized and efficient patient mortality prediction model, which can accurately predict the patients' discharge status using only ten input features. We used eICU-CRD v2.0 to train and validate the models; the dataset contains information on over 200,000 ICU admissions. The patients were divided into twelve disease groups, and models were fitted and tuned for each group. The models' predictive performance was evaluated using the area under a receiver operating curve (AUROC). The AUROC scores were 0.86 [std:0.02] to 0.92 [std:0.02] for CatBoost and 0.83 [std:0.02] to 0.91 [std:0.03] for E-CatBoost models across the defined disease groups; if measured over the entire patient population, their AUROC scores were 7 to 18 and 2 to 12 percent higher than the baseline models, respectively. Based on SHAP explanations, we found age, heart rate, respiratory rate, blood urine nitrogen, and creatinine level as the most critical cross-disease features in mortality predictions.

The criticality Index-mortality: A dynamic machine learning prediction algorithm for mortality prediction in children cared for in an ICU

BACKGROUND

The Criticality Index-Mortality uses physiology, therapy, and intensity of care to compute mortality risk for pediatric ICU patients. If the frequency of mortality risk computations were increased to every 3 h with model performance that could improve the assessment of severity of illness, it could be utilized to monitor patients for significant mortality risk change.

OBJECTIVES

To assess the performance of a dynamic method of updating mortality risk every 3 h using the Criticality Index-Mortality methodology and identify variables that are significant contributors to mortality risk predictions.

POPULATION

There were 8,399 pediatric ICU admissions with 312 (3.7%) deaths from January 1, 2018 to February 29, 2020. We randomly selected 75% of patients for training, 13% for validation, and 12% for testing.

MODEL

A neural network was trained to predict hospital survival or death during or following an ICU admission. Variables included age, gender, laboratory tests, vital signs, medications categories, and mechanical ventilation variables. The neural network was calibrated to mortality risk using nonparametric logistic regression.

RESULTS

Discrimination assessed across all time periods found an AUROC of 0.851 (0.841-0.862) and an AUPRC was 0.443 (0.417-0.467). When assessed for performance every 3 h, the AUROCs had a minimum value of 0.778 (0.689-0.867) and a maximum value of 0.885 (0.841,0.862); the AUPRCs had a minimum value 0.148 (0.058-0.328) and a maximum value of 0.499 (0.229-0.769). The calibration plot had an intercept of 0.011, a slope of 0.956, and the R² was 0.814. Comparison of observed vs. expected proportion of deaths revealed that 95.8% of the 543 risk intervals were not statistically significantly different. Construct validity assessed by death and survivor risk trajectories analyzed by mortality risk quartiles and 7 high and low risk diseases confirmed a priori clinical expectations about the trajectories of death and survivors.

CONCLUSIONS

The Criticality Index-Mortality computing mortality risk every 3 h for pediatric ICU patients has model performance that could enhance the clinical assessment of severity of illness. The overall Criticality Index-Mortality framework was effectively applied to develop an institutionally specific, and clinically relevant model for dynamic risk assessment of pediatric ICU patients.

A relationship between the incremental values of area under the ROC curve and of area under the precision-recall curve

BACKGROUND

Incremental value (IncV) evaluates the performance change between an existing risk model and a new model. Different IncV metrics do not always agree with each other. For example, compared with a prescribed-dose model, an ovarian-dose model for predicting acute ovarian failure has a slightly lower area under the receiver operating characteristic curve (AUC) but increases the area under the precision-recall curve (AP) by 48%. This phenomenon of disagreement is not uncommon, and can create confusion when assessing whether the added information improves the model prediction accuracy.

METHODS

In this article, we examine the analytical connections and differences between the AUC IncV (ΔAUC) and AP IncV (ΔAP). We also compare the true values of these two IncV metrics in a numerical study. Additionally, as both are semi-proper scoring rules, we compare them with a strictly proper scoring rule: the IncV of the scaled Brier score (ΔsBrS) in the numerical study.

RESULTS

We demonstrate that ΔAUC and ΔAP are both weighted averages of the changes (from the existing model to the new one) in separating the risk score distributions between events and non-events. However, ΔAP assigns heavier weights to the changes in higher-risk regions, whereas ΔAUC weights the changes equally. Due to this difference, the two IncV metrics can disagree, and the numerical study shows that their disagreement becomes more pronounced as the event rate decreases. In the numerical study, we also find that ΔAP has a wide range, from negative to positive, but the range of ΔAUC is much smaller. In addition, ΔAP and ΔsBrS are highly consistent, but ΔAUC is negatively correlated with ΔsBrS and ΔAP when the event rate is low.

CONCLUSIONS

ΔAUC treats the wins and losses of a new risk model equally across different risk regions. When neither the existing or new model is the true model, this equality could attenuate a superior performance of the new model for a sub-region. In contrast, ΔAP accentuates the change in the prediction accuracy for higher-risk regions.

Continuous Prediction of Mortality in the PICU: A Recurrent Neural Network Model in a Single-Center Dataset

OBJECTIVES

Develop, as a proof of concept, a recurrent neural network model using electronic medical records data capable of continuously assessing an individual child's risk of mortality throughout their ICU stay as a proxy measure of severity of illness.

DESIGN

Retrospective cohort study.

SETTING

PICU in a tertiary care academic children's hospital.

PATIENTS/SUBJECTS

Twelve thousand five hundred sixteen episodes (9,070 children) admitted to the PICU between January 2010 and February 2019, partitioned into training (50%), validation (25%), and test (25%) sets.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

On 2,475 test set episodes lasting greater than or equal to 24 hours in the PICU, the area under the receiver operating characteristic curve of the recurrent neural network's 12th hour predictions was 0.94 (CI, 0.93-0.95), higher than those of Pediatric Index of Mortality 2 (0.88; CI, [0.85-0.91]; p < 0.02), Pediatric Risk of Mortality III (12th hr) (0.89; CI, [0.86-0.92]; p < 0.05), and Pediatric Logistic Organ Dysfunction day 1 (0.85; [0.81-0.89]; p < 0.002). The recurrent neural network's discrimination increased with more acquired data and smaller lead time, achieving a 0.99 area under the receiver operating characteristic curve 24 hours prior to discharge. Despite not having diagnostic information, the recurrent neural network performed well across different primary diagnostic categories, generally achieving higher area under the receiver operating characteristic curve for these groups than the other three scores. On 692 test set episodes lasting greater than or equal to 5 days in the PICU, the recurrent neural network area under the receiver operating characteristic curves significantly outperformed their daily Pediatric Logistic Organ Dysfunction counterparts (p < 0.005).

CONCLUSIONS

The recurrent neural network model can process hundreds of input variables contained in a patient's electronic medical record and integrate them dynamically as measurements become available. Its high discrimination suggests the recurrent neural network's potential to provide an accurate, continuous, and real-time assessment of a child in the ICU.

Cellular and plasma proteomic determinants of COVID-19 and non-COVID-19 pulmonary diseases relative to healthy aging

We examine the cellular and soluble determinants of coronavirus disease 2019 (COVID-19) relative to aging by performing mass cytometry in parallel with clinical blood testing and plasma proteomic profiling of ~4,700 proteins from 71 individuals with pulmonary disease and 148 healthy donors (25-80 years old). Distinct cell populations were associated with age (GZMK⁺CD8⁺ T cells and CD25^low CD4⁺ T cells) and with COVID-19 (TBET^-EOMES^- CD4⁺ T cells, HLA-DR⁺CD38⁺ CD8⁺ T cells and CD27⁺CD38⁺ B cells). A unique population of TBET⁺EOMES⁺ CD4⁺ T cells was associated with individuals with COVID-19 who experienced moderate, rather than severe or lethal, disease. Disease severity correlated with blood creatinine and urea nitrogen levels. Proteomics revealed a major impact of age on the disease-associated plasma signatures and highlighted the divergent contribution of hepatocyte and muscle secretomes to COVID-19 plasma proteins. Aging plasma was enriched in matrisome proteins and heart/aorta smooth muscle cell-specific proteins. These findings reveal age-specific and disease-specific changes associated with COVID-19, and potential soluble mediators of the physiological impact of COVID-19.

Severity Trajectories of Pediatric Inpatients Using the Criticality Index

OBJECTIVES

To assess severity of illness trajectories described by the Criticality Index for survivors and deaths in five patient groups defined by the sequence of patient care in ICU and routine patient care locations.

DESIGN

The Criticality Index developed using a calibrated, deep neural network, measures severity of illness using physiology, therapies, and therapeutic intensity. Criticality Index values in sequential 6-hour time periods described severity trajectories.

SETTING

Hospitals with pediatric inpatient and ICU care.

PATIENTS

Pediatric patients never cared for in an ICU (n = 20,091), patients only cared for in the ICU (n = 2,096) and patients cared for in both ICU and non-ICU care locations (n = 17,023) from 2009 to 2016 Health Facts database (Cerner Corporation, Kansas City, MO).

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Criticality Index values were consistent with clinical experience. The median (25-75th percentile) ICU Criticality Index values (0.878 [0.696-0.966]) were more than 80-fold higher than the non-ICU values (0.010 [0.002-0.099]). Non-ICU Criticality Index values for patients transferred to the ICU were 40-fold higher than those never transferred to the ICU (0.164 vs 0.004). The median for ICU deaths was higher than ICU survivors (0.983 vs 0.875) (p < 0.001). The severity trajectories for the five groups met expectations based on clinical experience. Survivors had increasing Criticality Index values in non-ICU locations prior to ICU admission, decreasing Criticality Index values in the ICU, and decreasing Criticality Index values until hospital discharge. Deaths had higher Criticality Index values than survivors, steeper increases prior to the ICU, and worsening values in the ICU. Deaths had a variable course, especially those who died in non-ICU care locations, consistent with deaths associated with both active therapies and withdrawals/limitations of care.

CONCLUSIONS

Severity trajectories measured by the Criticality Index showed strong validity, reflecting the expected clinical course for five diverse patient groups.

Development of a scoring tool for predicting prolonged length of hospital stay in peritoneal dialysis patients through data mining

Background

The hospital admission rate is high in patients treated with peritoneal dialysis (PD), and the length of stay (LOS) in the hospital is a key indicator of medical resource allocation. This study aimed to develop a scoring tool for predicting prolonged LOS (pLOS) in PD patients by combining machine learning and traditional logistic regression (LR).

Methods

This study was based on patient data collected using the Hospital Quality Monitoring System (HQMS) in China. Three machine learning methods, classification and regression tree (CART), random forest (RF), and gradient boosting decision tree (GBDT), were used to develop models to predict pLOS, which is longer than the average LOS, in PD patients. The model with the best prediction performance was used to identify predictive factors contributing to the outcome. A multivariate LR model based on the identified predictors was then built to derive the score assigned to each predictor. Finally, a scoring tool was developed, and it was tested by stratifying PD patients into different pLOS risk groups.

Results

A total of 22,859 PD patients were included in our study, with 25.2% having pLOS. Among the three machine learning models, the RF model achieved the best prediction performance and thus was used to identify the 10 most predictive variables for building the scoring system. The multivariate LR model based on the identified predictors showed good discrimination power with an AUROC of 0.721 in the test dataset, and its coefficients were used as a basis for scoring tool development. On the basis of the developed scoring tool, PD patients were divided into three groups: low risk (≤5), median risk [5–10], and high risk (>10). The observed pLOS proportions in the low-risk, median-risk, and high-risk groups in the test dataset were 11.4%, 29.5%, and 54.7%, respectively.

Conclusions

This study developed a scoring tool to predict pLOS in PD patients. The scoring tool can effectively discriminate patients with different pLOS risks and be easily implemented in clinical practice. The pLOS scoring tool has a great potential to help physicians allocate medical resources optimally and achieve improved clinical outcomes.

Sepsis in the critically ill patient: current and emerging management strategies

Introduction: Sepsis, a dysregulated host response to infection, is a major cause of morbidity and mortality worldwide. Early identification and evidence-based treatment of sepsis are associated with improved outcomes.Areas covered: This narrative review was undertaken following a PubMed search for English language reports published before July 2020 using the terms 'sepsis,' 'septic shock,' 'fluids,' 'fluid therapy,' 'albumin,' 'corticosteroids,' 'vasopressor.' Emerging management strategies were identified following a search of the ClinicalTrails.gov database using the term 'sepsis.' Additional reports were identified by examining the reference lists of selected articles and based on personnel knowledge of the field of sepsis.Expert opinion: The core treatment of sepsis relies on source control, early antibiotics, and organ support. The main emerging strategies focus on immunomodulation, artificial intelligence, and on multi-omics approaches for a personalized therapy.

Linking Big Data and Prediction Strategies: Tools, Pitfalls, and Lessons Learned

OBJECTIVES

Modern critical care amasses unprecedented amounts of clinical data-so called "big data"-on a minute-by-minute basis. Innovative processing of these data has the potential to revolutionize clinical prognostics and decision support in the care of the critically ill but also forces clinicians to depend on new and complex tools of which they may have limited understanding and over which they have little control. This concise review aims to provide bedside clinicians with ways to think about common methods being used to extract information from clinical big datasets and to judge the quality and utility of that information.

DATA SOURCES

We searched the free-access search engines PubMed and Google Scholar using the MeSH terms "big data", "prediction", and "intensive care" with iterations of a range of additional potentially associated factors, along with published bibliographies, to find papers suggesting illustration of key points in the structuring and analysis of clinical "big data," with special focus on outcomes prediction and major clinical concerns in critical care.

STUDY SELECTION

Three reviewers independently screened preliminary citation lists.

DATA EXTRACTION

Summary data were tabulated for review.

DATA SYNTHESIS

To date, most relevant big data research has focused on development of and attempts to validate patient outcome scoring systems and has yet to fully make use of the potential for automation and novel uses of continuous data streams such as those available from clinical care monitoring devices.

CONCLUSIONS

Realizing the potential for big data to improve critical care patient outcomes will require unprecedented team building across disparate competencies. It will also require clinicians to develop statistical awareness and thinking as yet another critical judgment skill they bring to their patients' bedsides and to the array of evidence presented to them about their patients over the course of care.

Brief introduction of medical database and data mining technology in big data era

Data mining technology can search for potentially valuable knowledge from a large amount of data, mainly divided into data preparation and data mining, and expression and analysis of results. It is a mature information processing technology and applies database technology. Database technology is a software science that researches manages, and applies databases. The data in the database are processed and analyzed by studying the underlying theory and implementation methods of the structure, storage, design, management, and application of the database. We have introduced several databases and data mining techniques to help a wide range of clinical researchers better understand and apply database technology.

Machine Learning and Artificial Intelligence in Neurocritical Care: a Specialty-Wide Disruptive Transformation or a Strategy for Success

PURPOSE OF REVIEW

Neurocritical care combines the complexity of both medical and surgical disease states with the inherent limitations of assessing patients with neurologic injury. Artificial intelligence (AI) has garnered interest in the basic management of these complicated patients as data collection becomes increasingly automated.

RECENT FINDINGS

In this opinion article, we highlight the potential AI has in aiding the clinician in several aspects of neurocritical care, particularly in monitoring and managing intracranial pressure, seizures, hemodynamics, and ventilation. The model-based method and data-driven method are currently the two major AI methods for analyzing critical care data. Both are able to analyze the vast quantities of patient data that are accumulated in the neurocritical care unit. AI has the potential to reduce healthcare costs, minimize delays in patient management, and reduce medical errors. However, these systems are an aid to, not a replacement for, the clinician's judgment.

Development and Performance of Electronic Pediatric Risk of Mortality and Pediatric Logistic Organ Dysfunction-2 Automated Acuity Scores

OBJECTIVES

Develop and test the performance of electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV and electronic version of the Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 scores.

DESIGN

Retrospective, single-center cohort derived from structured electronic health record data.

SETTING

Large, quaternary PICU at a freestanding, university-affiliated children's hospital.

PATIENTS

All encounters with a PICU admission between January 1, 2009, and December 31, 2017, identified using electronic definitions of inpatient encounter.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The main outcome was predictive validity of each score for hospital mortality, assessed as model discrimination and calibration. Discrimination was examined with the area under the receiver operating characteristics curve and the area under the precision-recall curve. Calibration was assessed with the Hosmer-Lemeshow goodness of fit test and calculation of a standardized mortality ratio. Models were recalibrated with new regression coefficients in a training subset of 75% of encounters selected randomly from all years of the cohort and the calibrated models were tested in the remaining 25% of the cohort. Content validity was assessed by examining correlation between electronic versions of the scores and prospectively calculated data (electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV) and an alternative informatics approach (Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 score). The cohort included 21,335 encounters. Correlation coefficients indicated strong agreement between different methods of score calculation. Uncalibrated area under the receiver operating characteristics curves were 0.96 (95% CI, 0.95-0.97) for electronic version of the Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 score and 0.87 (95% CI, 0.85-0.89) for electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV for inpatient mortality. The uncalibrated electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV standardized mortality ratio was 0.63 (0.59-0.66), demonstrating strong agreement with previous, prospective evaluation at the study center. The uncalibrated electronic version of the Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 score standardized mortality ratio was 0.20 (0.18-0.21). All models required recalibrating (all Hosmer-Lemeshow goodness-of-fit, p < 0.001) and subsequently demonstrated acceptable goodness-of-fit when examined in a test subset (n = 5,334) of the cohort.

CONCLUSIONS

Electronically derived intensive care acuity scores demonstrate very good to excellent discrimination and can be calibrated to institutional outcomes. This approach can facilitate both performance improvement and research initiatives and may offer a scalable strategy for comparison of interinstitutional PICU outcomes.

Telemedicine in the ICU: clinical outcomes, economic aspects, and trainee education

PURPOSE OF REVIEW

The evidence base for telemedicine in the ICU (tele-ICU) is rapidly expanding. The last 2 years have seen important additions to our understanding of when, where, and how telemedicine in the ICU adds value.

RECENT FINDINGS

Recent publications and a recent meta-analysis confirm that tele-ICU improves core clinical outcomes for ICU patients. Recent evidence further demonstrates that comprehensive tele-ICU programs have the potential to quickly recuperate their implementation and operational costs and significantly increase case volumes and direct contribution margins particularly if additional logistics and care standardization functions are embedded to optimize ICU bed utilization and reduce complications. Even though the adoption of tele-ICU is increasing and the vast majority of today's medical graduates will regularly use some form of telemedicine and/or tele-ICU, telemedicine modules have not consistently found their way into educational curricula yet. Tele-ICU can be used very effectively to standardize supervision of medical trainees in bedside procedures or point-of-care ultrasound exams, especially during off-hours. Lastly, tele-ICUs routinely generate rich operational data, as well as risk-adjusted acuity and outcome data across the spectrum of critically ill patients, which can be utilized to support important clinical research and quality improvement projects.

SUMMARY

The value of tele-ICU to improve patient outcomes, optimize ICU bed utilization, increase financial performance and enhance educational opportunities for the next generation of providers has become more evident and differentiated in the last 2 years.

Describing organ dysfunction in the intensive care unit: a cohort study of 20,000 patients

BACKGROUND

Multiple organ dysfunction is a common cause of morbidity and mortality in intensive care units (ICUs). Original development of the Sequential Organ Failure Assessment (SOFA) score was not to predict outcome, but to describe temporal changes in organ dysfunction in critically ill patients. Organ dysfunction scoring may be a reasonable surrogate outcome in clinical trials but further exploration of the impact of case mix on the temporal sequence of organ dysfunction is required. Our aim was to compare temporal changes in SOFA scores between hospital survivors and non-survivors.

METHODS

We performed a population-based observational retrospective cohort study of critically ill patients admitted from January 1, 2004, to December 31, 2013, to 4 multisystem adult intensive care units (ICUs) in Calgary, Canada. The primary outcome was temporal changes in daily SOFA scores during the first 14 days of ICU admission. SOFA scores were modeled between hospital survivors and non-survivors using generalized estimating equations (GEE) and were also stratified by admission SOFA (≤ 11 versus > 11).

RESULTS

The cohort consisted of 20,007 patients with at least one SOFA score and was mostly male (58.2%) with a median age of 59 (interquartile range [IQR] 44-72). Median ICU length of stay was 3.5 (IQR 1.7-7.5) days. ICU and hospital mortality were 18.5% and 25.5%, respectively. Temporal change in SOFA scores varied by survival and admission SOFA score in a complicated relationship. Area under the receiver operating characteristic (ROC) curve using admission SOFA as a predictor of hospital mortality was 0.77. The hospital mortality rate was 5.6% for patients with an admission SOFA of 0-2 and 94.4% with an admission SOFA of 20-24. There was an approximately linear increase in hospital mortality for SOFA scores of 3-19 (range 8.7-84.7%).

CONCLUSIONS

Examining the clinical course of organ dysfunction in a large non-selective cohort of patients provides insight into the utility of SOFA. We have demonstrated that hospital outcome is associated with both admission SOFA and the temporal rate of change in SOFA after admission. It is necessary to further explore the impact of additional clinical factors on the clinical course of SOFA with large datasets.

Using the Shapes of Clinical Data Trajectories to Predict Mortality in ICUs

Supplemental Digital Content is available in the text.

1) To show how to exploit the information contained in the trajectories of time-varying patient clinical data for dynamic predictions of mortality in the ICU; and 2) to demonstrate the additional predictive value that can be achieved by incorporating this trajectory information.

Intensive Care Unit Telemedicine in the Era of Big Data, Artificial Intelligence, and Computer Clinical Decision Support Systems

This article examines the history of the telemedicine intensive care unit (tele-ICU), the current state of clinical decision support systems (CDSS) in the tele-ICU, applications of machine learning (ML) algorithms to critical care, and opportunities to integrate ML with tele-ICU CDSS. The enormous quantities of data generated by tele-ICU systems is a major driver in the development of the large, comprehensive, heterogeneous, and granular data sets necessary to develop generalizable ML CDSS algorithms, and deidentification of these data sets expands opportunities for ML CDSS research.

Towards a decision support tool for intensive care discharge: machine learning algorithm development using electronic healthcare data from MIMIC-III and Bristol, UK

OBJECTIVE

The primary objective is to develop an automated method for detecting patients that are ready for discharge from intensive care.

DESIGN

We used two datasets of routinely collected patient data to test and improve on a set of previously proposed discharge criteria.

SETTING

Bristol Royal Infirmary general intensive care unit (GICU).

PATIENTS

Two cohorts derived from historical datasets: 1870 intensive care patients from GICU in Bristol, and 7592 from Medical Information Mart for Intensive Care (MIMIC)-III.

RESULTS

In both cohorts few successfully discharged patients met all of the discharge criteria. Both a random forest and a logistic classifier, trained using multiple-source cross-validation, demonstrated improved performance over the original criteria and generalised well between the cohorts. The classifiers showed good agreement on which features were most predictive of readiness-for-discharge, and these were generally consistent with clinical experience. By weighting the discharge criteria according to feature importance from the logistic model we showed improved performance over the original criteria, while retaining good interpretability.

CONCLUSIONS

Our findings indicate the feasibility of the proposed approach to ready-for-discharge classification, which could complement other risk models of specific adverse outcomes in a future decision support system. Avenues for improvement to produce a clinically useful tool are identified.

DeepSOFA: A Continuous Acuity Score for Critically Ill Patients using Clinically Interpretable Deep Learning

Traditional methods for assessing illness severity and predicting in-hospital mortality among critically ill patients require time-consuming, error-prone calculations using static variable thresholds. These methods do not capitalize on the emerging availability of streaming electronic health record data or capture time-sensitive individual physiological patterns, a critical task in the intensive care unit. We propose a novel acuity score framework (DeepSOFA) that leverages temporal measurements and interpretable deep learning models to assess illness severity at any point during an ICU stay. We compare DeepSOFA with SOFA (Sequential Organ Failure Assessment) baseline models using the same model inputs and find that at any point during an ICU admission, DeepSOFA yields significantly more accurate predictions of in-hospital mortality. A DeepSOFA model developed in a public database and validated in a single institutional cohort had a mean AUC for the entire ICU stay of 0.90 (95% CI 0.90–0.91) compared with baseline SOFA models with mean AUC 0.79 (95% CI 0.79–0.80) and 0.85 (95% CI 0.85–0.86). Deep models are well-suited to identify ICU patients in need of life-saving interventions prior to the occurrence of an unexpected adverse event and inform shared decision-making processes among patients, providers, and families regarding goals of care and optimal resource utilization.

Validation of Prediction Models for Critical Care Outcomes Using Natural Language Processing of Electronic Health Record Data

Importance

Accurate prediction of outcomes among patients in intensive care units (ICUs) is important for clinical research and monitoring care quality. Most existing prediction models do not take full advantage of the electronic health record, using only the single worst value of laboratory tests and vital signs and largely ignoring information present in free-text notes. Whether capturing more of the available data and applying machine learning and natural language processing (NLP) can improve and automate the prediction of outcomes among patients in the ICU remains unknown.

Objectives

To evaluate the change in power for a mortality prediction model among patients in the ICU achieved by incorporating measures of clinical trajectory together with NLP of clinical text and to assess the generalizability of this approach.

Design, Setting, and Participants

This retrospective cohort study included 101 196 patients with a first-time admission to the ICU and a length of stay of at least 4 hours. Twenty ICUs at 2 academic medical centers (University of California, San Francisco [UCSF], and Beth Israel Deaconess Medical Center [BIDMC], Boston, Massachusetts) and 1 community hospital (Mills-Peninsula Medical Center [MPMC], Burlingame, California) contributed data from January 1, 2001, through June 1, 2017. Data were analyzed from July 1, 2017, through August 1, 2018.

Main Outcomes and Measures

In-hospital mortality and model discrimination as assessed by the area under the receiver operating characteristic curve (AUC) and model calibration as assessed by the modified Hosmer-Lemeshow statistic.

Results

Among 101 196 patients included in the analysis, 51.3% (n = 51 899) were male, with a mean (SD) age of 61.3 (17.1) years; their in-hospital mortality rate was 10.4% (n = 10 505). A baseline model using only the highest and lowest observed values for each laboratory test result or vital sign achieved a cross-validated AUC of 0.831 (95% CI, 0.830-0.832). In contrast, that model augmented with measures of clinical trajectory achieved an AUC of 0.899 (95% CI, 0.896-0.902; P < .001 for AUC difference). Further augmenting this model with NLP-derived terms associated with mortality further increased the AUC to 0.922 (95% CI, 0.916-0.924; P < .001). These NLP-derived terms were associated with improved model performance even when applied across sites (AUC difference for UCSF: 0.077 to 0.021; AUC difference for MPMC: 0.071 to 0.051; AUC difference for BIDMC: 0.035 to 0.043; P < .001) when augmenting with NLP at each site.

Conclusions and Relevance

Intensive care unit mortality prediction models incorporating measures of clinical trajectory and NLP-derived terms yielded excellent predictive performance and generalized well in this sample of hospitals. The role of these automated algorithms, particularly those using unstructured data from notes and other sources, in clinical research and quality improvement seems to merit additional investigation.