DeepSOFA: A Continuous Acuity Score for Critically Ill Patients using Clinically Interpretable Deep Learning

Real-time estimated Sequential Organ Failure Assessment (SOFA) score with intervals: improved risk monitoring with estimated uncertainty in health condition for patients in intensive care units

Purpose

Real-time risk monitoring is critical but challenging in intensive care units (ICUs) due to the lack of real-time updates for most clinical variables. Although real-time predictions have been integrated into various risk monitoring systems, existing systems do not address uncertainties in risk assessments. We developed a novel framework based on commonly used systems like the Sequential Organ Failure Assessment (SOFA) score by incorporating uncertainties to improve the effectiveness of real-time risk monitoring.

Methods

This study included 5351 patients admitted to the Cardiothoracic ICU in the National University Hospital in Singapore. We developed machine learning models to predict long lead-time variables and computed real-time SOFA scores using predictions. We calculated intervals to capture uncertainties in risk assessments and validated the association of the estimated real-time scores and intervals with mortality and readmission.

Results

Our model outperforms SOFA score in predicting 24-h mortality: Nagelkerke's R-squared (0.224 vs. 0.185, p < 0.001) and the area under the receiver operating characteristic curve (AUC) (0.870 vs. 0.843, p < 0.001), and significantly outperforms quick SOFA (Nagelkerke's R-squared = 0.125, AUC = 0.778). Our model also performs better in predicting 30-day readmission. We confirmed a positive net reclassification improvement (NRI) of our model over the SOFA score (0.184, p < 0.001). Similarly, we enhanced two additional scoring systems.

Conclusions

Incorporating uncertainties improved existing scores in real-time monitoring, which could be used to trigger on-demand laboratory tests, potentially improving early detection, reducing unnecessary testing, and thereby lowering healthcare expenditures, mortality, and readmission rates in clinical practice.

Supplementary Information

The online version contains supplementary material available at 10.1007/s13755-024-00331-5.

Assessment of the relationship between central venous pressure waveform and the severity of tricuspid valve regurgitation using data science

Purpose The relationship between the height of the V wave in the central venous pressure (CVP) waveform and the severity of tricuspid regurgitation (TR) is well known. Their diagnostic ability is unconfirmed. This study explored CVP waveform variations with TR. Methods All patients who underwent preoperative echocardiography and CVP waveform measurements before surgery at our institution were included. Indices were created to capture each feature of the CVP waveform. The median value for each case was obtained and statistically analyzed according to the severity of TR. A deep learning technique, Transformer, was used to handle the complex features of CVP waveforms. Results This study included 436 cases. The values for C wave - Y descent, X descent - Y descent, and V wave - Y descent differed significantly in the Jonckheere-Terpstra test (p = 0.0018, 0.027, and 0.077, respectively). The area under the receiver operating characteristic (ROC) curve (AUC) for X descent - Y descent in two groups, none to moderate TR and severe TR, was 0.83 (95% confidence interval (CI) [0.68, 0.98]). For Transformer, the accuracy of the validation dataset was 0.97. Conclusions The shape of the CVP waveform varied with the severity of TR in a large dataset.

Artificial Intelligence-Based Models for Prediction of Mortality in ICU Patients: A Scoping Review

BACKGROUND AND OBJECTIVE

Healthcare professionals may be able to anticipate more accurately a patient's timing of death and assess their possibility of recovery by implementing a real-time clinical decision support system. Using such a tool, the healthcare system can better understand a patient's condition and make more informed judgements about distributing limited resources. This scoping review aimed to analyze various death prediction AI (Artificial Intelligence) algorithms that have been used in ICU (Intensive Care Unit) patient populations.

METHODS

The search strategy of this study involved keyword combinations of outcome and patient setting such as mortality, survival, ICU, terminal care. These terms were used to perform database searches in MEDLINE, Embase, and PubMed up to July 2022. The variables, characteristics, and performance of the identified predictive models were summarized. The accuracy of the models was compared using their Area Under the Curve (AUC) values.

RESULTS

Databases search yielded an initial pool of 8271 articles. A two-step screening process was then applied: first, titles and abstracts were reviewed for relevance, reducing the pool to 429 articles. Next, a full-text review was conducted, further narrowing down the selection to 400 key studies. Out of 400 studies on different tools or models for prediction of mortality in ICUs, 16 papers focused on AI-based models which were ultimately included in this study that have deployed different AI-based and machine learning models to make a prediction about negative patient outcome. The accuracy and performance of the different models varied depending on the patient populations and medical conditions. It was found that AI models compared with traditional tools like SAP3 or APACHE IV score were more accurate in death prediction, with some models achieving an AUC of up to 92.9%. The overall mortality rate ranged from 5% to more than 60% in different studies.

CONCLUSION

We found that AI-based models exhibit varying performance across different patient populations. To enhance the accuracy of mortality prediction, we recommend customizing models for specific patient groups and medical contexts. By doing so, healthcare professionals may more effectively assess mortality risk and tailor treatments accordingly. Additionally, incorporating additional variables-such as genetic information-into new models can further improve their accuracy.

APRICOT-Mamba: Acuity Prediction in Intensive Care Unit (ICU): Development and Validation of a Stability, Transitions, and Life-Sustaining Therapies Prediction Model

On average, more than 5 million patients are admitted to intensive care units (ICUs) in the US, with mortality rates ranging from 10 to 29%. The acuity state of patients in the ICU can quickly change from stable to unstable, sometimes leading to life-threatening conditions. Early detection of deteriorating conditions can assist in more timely interventions and improved survival rates. While Artificial Intelligence (AI)-based models show potential for assessing acuity in a more granular and automated manner, they typically use mortality as a proxy of acuity in the ICU. Furthermore, these methods do not determine the acuity state of a patient (i.e., stable or unstable), the transition between acuity states, or the need for life-sustaining therapies. In this study, we propose APRICOT-M (Acuity Prediction in Intensive Care Unit-Mamba), a 1M-parameter state space-based neural network to predict acuity state, transitions, and the need for life-sustaining therapies in real-time among ICU patients. The model integrates ICU data in the preceding four hours (including vital signs, laboratory results, assessment scores, and medications) and patient characteristics (age, sex, race, and comorbidities) to predict the acuity outcomes in the next four hours. Our state space-based model can process sparse and irregularly sampled data without manual imputation, thus reducing the noise in input data and increasing inference speed. The model was trained on data from 107,473 patients (142,062 ICU admissions) from 55 hospitals between 2014-2017 and validated externally on data from 74,901 patients (101,356 ICU admissions) from 143 hospitals. Additionally, it was validated temporally on data from 12,927 patients (15,940 ICU admissions) from one hospital in 2018-2019 and prospectively on data from 215 patients (369 ICU admissions) from one hospital in 2021-2023. Three datasets were used for training and evaluation: the University of Florida Health (UFH) dataset, the electronic ICU Collaborative Research Database (eICU), and the Medical Information Mart for Intensive Care (MIMIC)-IV dataset. APRICOT-M significantly outperforms the baseline acuity assessment, Sequential Organ Failure Assessment (SOFA), for mortality prediction in both external (AUROC 0.95 CI: 0.94-0.95 compared to 0.78 CI: 0.78-0.79) and prospective (AUROC 0.99 CI: 0.97-1.00 compared to 0.80 CI: 0.65-0.92) cohorts, as well as for instability prediction (external AUROC 0.75 CI: 0.74-0.75 compared to 0.51 CI: 0.51-0.51, and prospective AUROC 0.69 CI: 0.64-0.74 compared to 0.53 CI: 0.50-0.57). This tool has the potential to help clinicians make timely interventions by predicting the transition between acuity states and decision-making on life-sustaining within the next four hours in the ICU.

Explainable text-tabular models for predicting mortality risk in companion animals

As interest in using machine learning models to support clinical decision-making increases, explainability is an unequivocal priority for clinicians, researchers and regulators to comprehend and trust their results. With many clinical datasets containing a range of modalities, from the free-text of clinician notes to structured tabular data entries, there is a need for frameworks capable of providing comprehensive explanation values across diverse modalities. Here, we present a multimodal masking framework to extend the reach of SHapley Additive exPlanations (SHAP) to text and tabular datasets to identify risk factors for companion animal mortality in first-opinion veterinary electronic health records (EHRs) from across the United Kingdom. The framework is designed to treat each modality consistently, ensuring uniform and consistent treatment of features and thereby fostering predictability in unimodal and multimodal contexts. We present five multimodality approaches, with the best-performing method utilising PetBERT, a language model pre-trained on a veterinary dataset. Utilising our framework, we shed light for the first time on the reasons each model makes its decision and identify the inclination of PetBERT towards a more pronounced engagement with free-text narratives compared to BERT-base's predominant emphasis on tabular data. The investigation also explores the important features on a more granular level, identifying distinct words and phrases that substantially influenced an animal's life status prediction. PetBERT showcased a heightened ability to grasp phrases associated with veterinary clinical nomenclature, signalling the productivity of additional pre-training of language models.

A Multimodal Approach for Detection and Assessment of Depression Using Text, Audio and Video

Depression is one of the most common mental disorders, and rates of depression in individuals increase each year. Traditional diagnostic methods are primarily based on professional judgment, which is prone to individual bias. Therefore, it is crucial to design an effective and robust diagnostic method for automated depression detection. Current artificial intelligence approaches are limited in their abilities to extract features from long sentences. In addition, current models are not as robust with large input dimensions. To solve these concerns, a multimodal fusion model comprised of text, audio, and video for both depression detection and assessment tasks was developed. In the text modality, pre-trained sentence embedding was utilized to extract semantic representation along with Bidirectional long short-term memory (BiLSTM) to predict depression. This study also used Principal component analysis (PCA) to reduce the dimensionality of the input feature space and Support vector machine (SVM) to predict depression based on audio modality. In the video modality, Extreme gradient boosting (XGBoost) was employed to conduct both feature selection and depression detection. The final predictions were given by outputs of the different modalities with an ensemble voting algorithm. Experiments on the Distress analysis interview corpus wizard-of-Oz (DAIC-WOZ) dataset showed a great improvement of performance, with a weighted F1 score of 0.85, a Root mean square error (RMSE) of 5.57, and a Mean absolute error (MAE) of 4.48. Our proposed model outperforms the baseline in both depression detection and assessment tasks, and was shown to perform better than other existing state-of-the-art depression detection methods.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43657-023-00152-8.

Wearable sensors in patient acuity assessment in critical care

Acuity assessments are vital for timely interventions and fair resource allocation in critical care settings. Conventional acuity scoring systems heavily depend on subjective patient assessments, leaving room for implicit bias and errors. These assessments are often manual, time-consuming, intermittent, and challenging to interpret accurately, especially for healthcare providers. This risk of bias and error is likely most pronounced in time-constrained and high-stakes environments, such as critical care settings. Furthermore, such scores do not incorporate other information, such as patients' mobility level, which can indicate recovery or deterioration in the intensive care unit (ICU), especially at a granular level. We hypothesized that wearable sensor data could assist in assessing patient acuity granularly, especially in conjunction with clinical data from electronic health records (EHR). In this prospective study, we evaluated the impact of integrating mobility data collected from wrist-worn accelerometers with clinical data obtained from EHR for estimating acuity. Accelerometry data were collected from 87 patients wearing accelerometers on their wrists in an academic hospital setting. The data was evaluated using five deep neural network models: VGG, ResNet, MobileNet, SqueezeNet, and a custom Transformer network. These models outperformed a rule-based clinical score (Sequential Organ Failure Assessment, SOFA) used as a baseline when predicting acuity state (for ground truth we labeled as unstable patients if they needed life-supporting therapies, and as stable otherwise), particularly regarding the precision, sensitivity, and F1 score. The results demonstrate that integrating accelerometer data with demographics and clinical variables improves predictive performance compared to traditional scoring systems in healthcare. Deep learning models consistently outperformed the SOFA score baseline across various scenarios, showing notable enhancements in metrics such as the area under the receiver operating characteristic (ROC) Curve (AUC), precision, sensitivity, specificity, and F1 score. The most comprehensive scenario, leveraging accelerometer, demographics, and clinical data, achieved the highest AUC of 0.73, compared to 0.53 when using SOFA score as the baseline, with significant improvements in precision (0.80 vs. 0.23), specificity (0.79 vs. 0.73), and F1 score (0.77 vs. 0.66). This study demonstrates a novel approach beyond the simplistic differentiation between stable and unstable conditions. By incorporating mobility and comprehensive patient information, we distinguish between these states in critically ill patients and capture essential nuances in physiology and functional status. Unlike rudimentary definitions, such as equating low blood pressure with instability, our methodology delves deeper, offering a more holistic understanding and potentially valuable insights for acuity assessment.

Identifying acute illness phenotypes via deep temporal interpolation and clustering network on physiologic signatures

Using clustering analysis for early vital signs, unique patient phenotypes with distinct pathophysiological signatures and clinical outcomes may be revealed and support early clinical decision-making. Phenotyping using early vital signs has proven challenging, as vital signs are typically sampled sporadically. We proposed a novel, deep temporal interpolation and clustering network to simultaneously extract latent representations from irregularly sampled vital signs and derive phenotypes. Four distinct clusters were identified. Phenotype A (18%) had the greatest prevalence of comorbid disease with increased prevalence of prolonged respiratory insufficiency, acute kidney injury, sepsis, and long-term (3-year) mortality. Phenotypes B (33%) and C (31%) had a diffuse pattern of mild organ dysfunction. Phenotype B's favorable short-term clinical outcomes were tempered by the second highest rate of long-term mortality. Phenotype C had favorable clinical outcomes. Phenotype D (17%) exhibited early and persistent hypotension, high incidence of early surgery, and substantial biomarker incidence of inflammation. Despite early and severe illness, phenotype D had the second lowest long-term mortality. After comparing the sequential organ failure assessment scores, the clustering results did not simply provide a recapitulation of previous acuity assessments. This tool may impact triage decisions and have significant implications for clinical decision-support under time constraints and uncertainty.

Artificial Intelligence in Surgical Research: Accomplishments and Future Directions

MINI-ABSTRACT

The study introduces various methods of performing conventional ML and their implementation in surgical areas, and the need to move beyond these traditional approaches given the advent of big data.

OBJECTIVE

Investigate current understanding and future directions of machine learning applications, such as risk stratification, clinical data analytics, and decision support, in surgical practice.

SUMMARY BACKGROUND DATA

The advent of the electronic health record, near unlimited computing, and open-source computational packages have created an environment for applying artificial intelligence, machine learning, and predictive analytic techniques to healthcare. The "hype" phase has passed, and algorithmic approaches are being developed for surgery patients through all stages of care, involving preoperative, intraoperative, and postoperative components. Surgeons must understand and critically evaluate the strengths and weaknesses of these methodologies.

METHODS

The current body of AI literature was reviewed, emphasizing on contemporary approaches important in the surgical realm.

RESULTS AND CONCLUSIONS

The unrealized impacts of AI on clinical surgery and its subspecialties are immense. As this technology continues to pervade surgical literature and clinical applications, knowledge of its inner workings and shortcomings is paramount in determining its appropriate implementation.

Real-time machine learning model to predict short-term mortality in critically ill patients: development and international validation

BACKGROUND

A real-time model for predicting short-term mortality in critically ill patients is needed to identify patients at imminent risk. However, the performance of the model needs to be validated in various clinical settings and ethnicities before its clinical application. In this study, we aim to develop an ensemble machine learning model using routinely measured clinical variables at a single academic institution in South Korea.

METHODS

We developed an ensemble model using deep learning and light gradient boosting machine models. Internal validation was performed using the last two years of the internal cohort dataset, collected from a single academic hospital in South Korea between 2007 and 2021. External validation was performed using the full Medical Information Mart for Intensive Care (MIMIC), eICU-Collaborative Research Database (eICU-CRD), and Amsterdam University Medical Center database (AmsterdamUMCdb) data. The area under the receiver operating characteristic curve (AUROC) was calculated and compared to that for the National Early Warning Score (NEWS).

RESULTS

The developed model (iMORS) demonstrated high predictive performance with an internal AUROC of 0.964 (95% confidence interval [CI] 0.963-0.965) and external AUROCs of 0.890 (95% CI 0.889-0.891) for MIMIC, 0.886 (95% CI 0.885-0.887) for eICU-CRD, and 0.870 (95% CI 0.868-0.873) for AmsterdamUMCdb. The model outperformed the NEWS with higher AUROCs in the internal and external validation (0.866 for the internal, 0.746 for MIMIC, 0.798 for eICU-CRD, and 0.819 for AmsterdamUMCdb; p < 0.001).

CONCLUSIONS

Our real-time machine learning model to predict short-term mortality in critically ill patients showed excellent performance in both internal and external validations. This model could be a useful decision-support tool in the intensive care units to assist clinicians.

Quantitative methods for optimizing patient outcomes in liver transplantation

Liver transplantation (LT) is a lifesaving yet complex intervention with considerable challenges impacting graft and patient outcomes. Despite best practices, 5-year graft survival is only 70%. Sophisticated quantitative techniques offer potential solutions by assimilating multifaceted data into insights exceeding human cognition. Optimizing donor-recipient matching and graft allocation presents additional intricacies, involving the integration of clinical and laboratory data to select the ideal donor and recipient pair. Allocation must balance physiological variables with geographical and logistical constraints and timing. Quantitative methods can integrate these complex factors to optimize graft utilization. Such methods can also aid in personalizing treatment regimens, drawing on both pretransplant and posttransplant data, possibly using continuous immunological monitoring to enable early detection of graft injury or infected states. Advanced analytics is thus poised to transform management in LT, maximizing graft and patient survival. In this review, we describe quantitative methods applied to organ transplantation, with a focus on LT. These include quantitative methods for (1) utilizing and allocating donor organs equitably and optimally, (2) improving surgical planning through preoperative imaging, (3) monitoring graft and immune status, (4) determining immunosuppressant doses, and (5) establishing and maintaining the health of graft and patient after LT.

Temporal-Spatial Correlation Attention Network for Clinical Data Analysis in Intensive Care Unit

Recent advancements in medical information technology have enabled electronic health records (EHRs) to store comprehensive clinical data which has ushered healthcare into the era of "big data". However, medical data are rather complicated, making problem-solving in healthcare be limited in scope and comprehensiveness. The rapid development of deep learning in recent years has opened up opportunities for leveraging big data in healthcare. In this article we introduce a temporal-spatial correlation attention network (TSCAN) to address various clinical characteristic prediction problems, including mortality prediction, length of stay prediction, physiologic decline detection, and phenotype classification. Leveraging the attention mechanism model's design, our approach efficiently identifies relevant items in clinical data and temporally correlated nodes based on specific tasks, resulting in improved prediction accuracy. Additionally, our method identifies crucial clinical indicators associated with significant outcomes, which can inform and enhance treatment options. Our experiments utilize data from the publicly accessible Medical Information Mart for Intensive Care (MIMIC-IV) database. Finally, our approach demonstrates notable performance improvements of 2.0% (metric) compared to other SOTA prediction methods. Specifically, we achieved an impressive 90.7% mortality rate prediction accuracy and 45.1% accuracy in length of stay prediction.

An overview of clinical machine learning applications in neurology

Machine learning techniques for clinical applications are evolving, and the potential impact this will have on clinical neurology is important to recognize. By providing a broad overview on this growing paradigm of clinical tools, this article aims to help healthcare professionals in neurology prepare to navigate both the opportunities and challenges brought on through continued advancements in machine learning. This narrative review first elaborates on how machine learning models are organized and implemented. Machine learning tools are then classified by clinical application, with examples of uses within neurology described in more detail. Finally, this article addresses limitations and considerations regarding clinical machine learning applications in neurology.

Children are small adults (when properly normalized): Transferrable/generalizable sepsis prediction

Background

Though governed by the same underlying biology, the differential physiology of children causes the temporal evolution from health to a septic/diseased state to follow trajectories that are distinct from adult cases. As pediatric sepsis data sets are less readily available than for adult sepsis, we aim to leverage this shared underlying biology by normalizing pediatric physiological data such that it would be directly comparable to adult data, and then develop machine-learning (ML) based classifiers to predict the onset of sepsis in the pediatric population. We then externally validated the classifiers in an independent adult dataset.

Methods

Vital signs and laboratory observables were obtained from the Pediatric Intensive Care (PIC) database. These data elements were normalized for age and placed on a continuous scale, termed the Continuous Age-Normalized SOFA (CAN-SOFA) score. The XGBoost algorithm was used to classify pediatric patients that are septic. We tested the trained model using adult data from the MIMIC-IV database.

Results

On the pediatric population, the sepsis classifier has an accuracy of 0.84 and an F1-Score of 0.867. On the adult population, the sepsis classifier has an accuracy of 0.80 and an F1-score of 0.88; when tested on the adult population, the model showed similar performance degradation ("data drift") as in the pediatric population.

Conclusions

In this work, we demonstrate that, using a straightforward age-normalization method, EHR's can be generalizable compared (at least in the context of sepsis) between the pediatric and adult populations.

Prognostic models of in-hospital mortality of intensive care patients using neural representation of unstructured text: A systematic review and critical appraisal

OBJECTIVE

To review and critically appraise published and preprint reports of prognostic models of in-hospital mortality of patients in the intensive-care unit (ICU) based on neural representations (embeddings) of clinical notes.

METHODS

PubMed and arXiv were searched up to August 1, 2022. At least two reviewers independently selected the studies that developed a prognostic model of in-hospital mortality of intensive-care patients using free-text represented as embeddings and extracted data using the CHARMS checklist. Risk of bias was assessed using PROBAST. Reporting on the model was assessed with the TRIPOD guideline. To assess the machine learning components that were used in the models, we present a new descriptive framework based on different techniques to represent text and provide predictions from text. The study protocol was registered in the PROSPERO database (CRD42022354602).

RESULTS

Eighteen studies out of 2,825 were included. All studies used the publicly-available MIMIC dataset. Context-independent word embeddings are widely used. Model discrimination was provided by all studies (AUROC 0.75-0.96), but measures of calibration were scarce. Seven studies used both structural clinical variables and notes. Model discrimination improved when adding clinical notes to variables. None of the models was externally validated and often a simple train/test split was used for internal validation. Our critical appraisal demonstrated a high risk of bias in all studies and concerns regarding their applicability in clinical practice.

CONCLUSION

All studies used a neural architecture for prediction and were based on one publicly available dataset. Clinical notes were reported to improve predictive performance when used in addition to only clinical variables. Most studies had methodological, reporting, and applicability issues. We recommend reporting both model discrimination and calibration, using additional data sources, and using more robust evaluation strategies, including prospective and external validation. Finally, sharing data and code is encouraged to improve study reproducibility.

Dynamic Delirium Prediction in the Intensive Care Unit using Machine Learning on Electronic Health Records

Delirium is a syndrome of acute brain failure which is prevalent amongst older adults in the Intensive Care Unit (ICU). Incidence of delirium can significantly worsen prognosis and increase mortality, therefore necessitating its rapid and continual assessment in the ICU. Currently, the common approach for delirium assessment is manual and sporadic. Hence, there exists a critical need for a robust and automated system for predicting delirium in the ICU. In this work, we develop a machine learning (ML) system for real-time prediction of delirium using Electronic Health Record (EHR) data. Unlike prior approaches which provide one delirium prediction label per entire ICU stay, our approach provides predictions every 12 hours. We use the latest 12 hours of ICU data, along with patient demographic and medical history data, to predict delirium risk in the next 12-hour window. This enables delirium risk prediction as soon as 12 hours after ICU admission. We train and test four ML classification algorithms on longitudinal EHR data pertaining to 16,327 ICU stays of 13,395 patients covering a total of 56,297 12-hour windows in the ICU to predict the dynamic incidence of delirium. The best performing algorithm was Categorical Boosting which achieved an area under receiver operating characteristic curve (AUROC) of 0.87 (95% Confidence Interval; C.I, 0.86-0.87). The deployment of this ML system in ICUs can enable early identification of delirium, thereby reducing its deleterious impact on long-term adverse outcomes, such as ICU cost, length of stay and mortality.

Prospective Evaluation of a Dynamic Acuity Score for Regularly Assessing a Critically Ill Patient's Risk of Mortality

OBJECTIVE

Predictive models developed for use in ICUs have been based on retrospectively collected data, which does not take into account the challenges associated with live, clinical data. This study sought to determine if a previously constructed predictive model of ICU mortality (ViSIG) is robust when using data collected prospectively in near real-time.

DESIGN

Prospectively collected data were aggregated and transformed to evaluate a previously developed rolling predictor of ICU mortality.

SETTING

Five adult ICUs at Robert Wood Johnson-Barnabas University Hospital and one adult ICU at Stamford Hospital.

PATIENTS

One thousand eight hundred and ten admissions from August to December 2020.

MEASUREMENTS AND MAIN RESULTS

The ViSIG Score, comprised of severity weights for heart rate, respiratory rate, oxygen saturation, mean arterial pressure, mechanical ventilation, and values for OBS Medical's Visensia Index. This information was collected prospectively, whereas data on discharge disposition was collected retrospectively to measure the ViSIG Score's accuracy. The distribution of patients' maximum ViSIG Score was compared with ICU mortality rate, and cut points determined where changes in mortality probability were greatest. The ViSIG Score was validated on new admissions. The ViSIG Score was able to stratify patients into three groups: 0-37 (low risk), 38-58 (moderate risk), and 59-100 (high risk), with mortality of 1.7%, 12.0%, and 39.8%, respectively ( p < 0.001). The sensitivity and specificity of the model to predict mortality for the high-risk group were 51% and 91%. Performance on the validation dataset remained high. There were similar increases across risk groups for length of stay, estimated costs, and readmission.

CONCLUSIONS

Using prospectively collected data, the ViSIG Score produced risk groups for mortality with good sensitivity and excellent specificity. A future study will evaluate making the ViSIG Score visible to clinicians to determine whether this metric can influence clinician behavior to reduce adverse outcomes.

Predictive Modeling Using Artificial Intelligence and Machine Learning Algorithms on Electronic Health Record Data: Advantages and Challenges

The rapid adoption of electronic health record (EHR) systems in US hospitals from 2008 to 2014 produced novel data elements for analysis. Concurrent innovations in computing architecture and machine learning (ML) algorithms have made rapid consumption of health data feasible and a powerful engine for clinical innovation. In critical care research, the net convergence of these trends has resulted in an exponential increase in outcome prediction research. In the following article, we explore the history of outcome prediction in the intensive care unit (ICU), the growing use of EHR data, and the rise of artificial intelligence and ML (AI) in critical care.

Assessment of Prediction Tasks and Time Window Selection in Temporal Modeling of Electronic Health Record Data: a Systematic Review

Temporal electronic health record (EHR) data are often preferred for clinical prediction tasks because they offer more complete representations of a patient's pathophysiology than static data. A challenge when working with temporal EHR data is problem formulation, which includes defining the time windows of interest and the prediction task. Our objective was to conduct a systematic review that assessed the definition and reporting of concepts relevant to temporal clinical prediction tasks. We searched PubMed® and IEEE Xplore® databases for studies from January 1, 2010 applying machine learning models to EHR data for patient outcome prediction. Publications applying time-series methods were selected for further review. We identified 92 studies and summarized them by clinical context and definition and reporting of the prediction problem. For the time windows of interest, 12 studies did not discuss window lengths, 57 used a single set of window lengths, and 23 evaluated the relationship between window length and model performance. We also found that 72 studies had appropriate reporting of the prediction task. However, evaluation of prediction problem formulation for temporal EHR data was complicated by heterogeneity in assessing and reporting of these concepts. Even among studies modeling similar clinical outcomes, there were variations in terminology used to describe the prediction problem, rationale for window lengths, and determination of the outcome of interest. As temporal modeling using EHR data expands, minimal reporting standards should include time-series specific concerns to promote rigor and reproducibility in future studies and facilitate model implementation in clinical settings.

Supplementary Information

The online version contains supplementary material available at 10.1007/s41666-023-00143-4.

Automated APACHE II and SOFA score calculation using real-world electronic medical record data in a single center

The integration of illness severity and organ dysfunction scores into clinical practice, including the APACHE II and SOFA scores, has been challenging due to constraints associated to manual score calculation. With electronic medical records (EMR), score calculation automation using data extraction scripts has emerged as a solution. We aimed to demonstrate that APACHE II and SOFA scores calculated with an automated EMR-based data extraction script predict important clinical endpoints. In this retrospective cohort study, every adult patient admitted to one of our three ICUs, between July 1, 2019, and December 31, 2020, were enrolled. For every patient, an automated ICU admission APACHE II score was calculated using EMR data and minimal clinician input. Fully automated daily SOFA scores were calculated for every patient. 4 794 ICU admissions met our selection criteria. Of these ICU admissions, 522 deaths were recorded (10.9% in-hospital mortality rate). The automated APACHE II was discriminant for in-hospital mortality (AU-ROC = 0.83 (95% CI 0.81-0.85)). We observed an association between the APACHE II score and ICU LOS, with a statistically significant mean increase of 1.1 days of ICU LOS (1.1 [1-1.2]; p =  < .0001) for each 10 units increase in APACHE score. SOFA score curves did not discrimate significantly between survivors and non-survivors. A partially automated APACHE II score, calculated with real-world EMR data using an extraction script, is associated with in-hospital mortality risk. The automated APACHE II score could potentially constitute an acceptable surrogate of ICU acuity to be used in resource allocation and triaging, especially in time of high demand for ICU beds.

Predicting outcomes of continuous renal replacement therapy using body composition monitoring: a deep-learning approach

Fluid balance is a critical prognostic factor for patients with severe acute kidney injury (AKI) requiring continuous renal replacement therapy (CRRT). This study evaluated whether repeated fluid balance monitoring could improve prognosis in this clinical population. This was a multicenter retrospective study that included 784 patients (mean age, 67.8 years; males, 66.4%) with severe AKI requiring CRRT during 2017-2019 who were treated in eight tertiary hospitals in Korea. Sequential changes in total body water were compared between patients who died (event group) and those who survived (control group) using mixed-effects linear regression analyses. The performance of various machine learning methods, including recurrent neural networks, was compared to that of existing prognostic clinical scores. After adjusting for confounding factors, a marginal benefit of fluid balance was identified for the control group compared to that for the event group (p = 0.074). The deep-learning model using a recurrent neural network with an autoencoder and including fluid balance monitoring provided the best differentiation between the groups (area under the curve, 0.793) compared to 0.604 and 0.606 for SOFA and APACHE II scores, respectively. Our prognostic, deep-learning model underlines the importance of fluid balance monitoring for prognosis assessment among patients receiving CRRT.

Model Choice for Quantitative Health Impact Assessment and Modelling: An Expert Consultation and Narrative Literature Review

BACKGROUND

Health impact assessment (HIA) is a widely used process that aims to identify the health impacts, positive or negative, of a policy or intervention that is not necessarily placed in the health sector. Most HIAs are done prospectively and aim to forecast expected health impacts under assumed policy implementation. HIAs may quantitatively and/ or qualitatively assess health impacts, with this study focusing on the former. A variety of quantitative modelling methods exist that are used for forecasting health impacts, however, they differ in application area, data requirements, assumptions, risk modelling, complexities, limitations, strengths, and comprehensibility. We reviewed relevant models, so as to provide public health researchers with considerations for HIA model choice.

METHODS

Based on an HIA expert consultation, combined with a narrative literature review, we identified the most relevant models that can be used for health impact forecasting. We narratively and comparatively reviewed the models, according to their fields of application, their configuration and purposes, counterfactual scenarios, underlying assumptions, health risk modelling, limitations and strengths.

RESULTS

Seven relevant models for health impacts forecasting were identified, consisting of (i) comparative risk assessment (CRA), (ii) time series analysis (TSA), (iii) compartmental models (CMs), (iv) structural models (SMs), (v) agent-based models (ABMs), (vi) microsimulations (MS), and (vii) artificial intelligence (AI)/machine learning (ML). These models represent a variety in approaches and vary in the fields of HIA application, complexity and comprehensibility. We provide a set of criteria for HIA model choice. Researchers must consider that model input assumptions match the available data and parameter structures, the available resources, and that model outputs match the research question, meet expectations and are comprehensible to end-users.

CONCLUSION

The reviewed models have specific characteristics, related to available data and parameter structures, computational implementation, interpretation and comprehensibility, which the researcher should critically consider before HIA model choice.

Building an automated, machine learning-enabled platform for predicting post-operative complications

Objective. In 2019, the University of Florida College of Medicine launched theMySurgeryRiskalgorithm to predict eight major post-operative complications using automatically extracted data from the electronic health record.Approach. This project was developed in parallel with our Intelligent Critical Care Center and represents a culmination of efforts to build an efficient and accurate model for data processing and predictive analytics.Main Results and Significance. This paper discusses how our model was constructed and improved upon. We highlight the consolidation of the database, processing of fixed and time-series physiologic measurements, development and training of predictive models, and expansion of those models into different aspects of patient assessment and treatment. We end by discussing future directions of the model.

Overtriage, Undertriage, and Value of Care after Major Surgery: An Automated, Explainable Deep Learning-Enabled Classification System

BACKGROUND

In single-institution studies, overtriaging low-risk postoperative patients to ICUs has been associated with a low value of care; undertriaging high-risk postoperative patients to general wards has been associated with increased mortality and morbidity. This study tested the reproducibility of an automated postoperative triage classification system to generating an actionable, explainable decision support system.

STUDY DESIGN

This longitudinal cohort study included adults undergoing inpatient surgery at two university hospitals. Triage classifications were generated by an explainable deep learning model using preoperative and intraoperative electronic health record features. Nearest neighbor algorithms identified risk-matched controls. Primary outcomes were mortality, morbidity, and value of care (inverted risk-adjusted mortality/total direct costs).

RESULTS

Among 4,669 ICU admissions, 237 (5.1%) were overtriaged. Compared with 1,021 control ward admissions, overtriaged admissions had similar outcomes but higher costs ($15.9K [interquartile range $9.8K to $22.3K] vs $10.7K [$7.0K to $17.6K], p < 0.001) and lower value of care (0.2 [0.1 to 0.3] vs 1.5 [0.9 to 2.2], p < 0.001). Among 8,594 ward admissions, 1,029 (12.0%) were undertriaged. Compared with 2,498 control ICU admissions, undertriaged admissions had longer hospital length-of-stays (6.4 [3.4 to 12.4] vs 5.4 [2.6 to 10.4] days, p < 0.001); greater incidence of hospital mortality (1.7% vs 0.7%, p = 0.03), cardiac arrest (1.4% vs 0.5%, p = 0.04), and persistent acute kidney injury without renal recovery (5.2% vs 2.8%, p = 0.002); similar costs ($21.8K [$13.3K to $34.9K] vs $21.9K [$13.1K to $36.3K]); and lower value of care (0.8 [0.5 to 1.3] vs 1.2 [0.7 to 2.0], p < 0.001).

CONCLUSIONS

Overtriage was associated with low value of care; undertriage was associated with both low value of care and increased mortality and morbidity. The proposed framework for generating automated postoperative triage classifications is reproducible.

Dynamic predictions of postoperative complications from explainable, uncertainty-aware, and multi-task deep neural networks

Accurate prediction of postoperative complications can inform shared decisions regarding prognosis, preoperative risk-reduction, and postoperative resource use. We hypothesized that multi-task deep learning models would outperform conventional machine learning models in predicting postoperative complications, and that integrating high-resolution intraoperative physiological time series would result in more granular and personalized health representations that would improve prognostication compared to preoperative predictions. In a longitudinal cohort study of 56,242 patients undergoing 67,481 inpatient surgical procedures at a university medical center, we compared deep learning models with random forests and XGBoost for predicting nine common postoperative complications using preoperative, intraoperative, and perioperative patient data. Our study indicated several significant results across experimental settings that suggest the utility of deep learning for capturing more precise representations of patient health for augmented surgical decision support. Multi-task learning improved efficiency by reducing computational resources without compromising predictive performance. Integrated gradients interpretability mechanisms identified potentially modifiable risk factors for each complication. Monte Carlo dropout methods provided a quantitative measure of prediction uncertainty that has the potential to enhance clinical trust. Multi-task learning, interpretability mechanisms, and uncertainty metrics demonstrated potential to facilitate effective clinical implementation.

Predictive Modeling for Readmission to Intensive Care: A Systematic Review

To evaluate the methodologic rigor and predictive performance of models predicting ICU readmission; to understand the characteristics of ideal prediction models; and to elucidate relationships between appropriate triage decisions and patient outcomes.

DATA SOURCES

PubMed, Web of Science, Cochrane, and Embase.

STUDY SELECTION

Primary literature that reported the development or validation of ICU readmission prediction models within from 2010 to 2021.

DATA EXTRACTION

Relevant study information was extracted independently by two authors using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies checklist. Bias was evaluated using the Prediction model Risk Of Bias ASsessment Tool. Data sources, modeling methodology, definition of outcomes, performance, and risk of bias were critically evaluated to elucidate relevant relationships.

DATA SYNTHESIS

Thirty-three articles describing models were included. Six studies had a high overall risk of bias due to improper inclusion criteria or omission of critical analysis details. Four other studies had an unclear overall risk of bias due to lack of detail describing the analysis. Overall, the most common (50% of studies) source of bias was the filtering of candidate predictors via univariate analysis. The poorest performing models used existing clinical risk or acuity scores such as Acute Physiologic Assessment and Chronic Health Evaluation II, Sequential Organ Failure Assessment, or Stability and Workload Index for Transfer as the sole predictor. The higher-performing ICU readmission prediction models used homogenous patient populations, specifically defined outcomes, and routinely collected predictors that were analyzed over time.

CONCLUSIONS

Models predicting ICU readmission can achieve performance advantages by using longitudinal time series modeling, homogenous patient populations, and predictor variables tailored to those populations.

An explainable knowledge distillation method with XGBoost for ICU mortality prediction

BACKGROUND AND OBJECTIVE

Mortality prediction is an important task in intensive care unit (ICU) for quantifying the severity of patients' physiological condition. Currently, scoring systems are widely applied for mortality prediction, while the performance is unsatisfactory in many clinical conditions due to the non-specificity and linearity characteristics of the used model. As the availability of the large volume of data recorded in electronic health records (EHRs), deep learning models have achieved state-of-art predictive performance. However, deep learning models are hard to meet the requirement of explainability in clinical conditions. Hence, an explainable Knowledge Distillation method with XGBoost (XGB-KD) is proposed to improve the predictive performance of XGBoost while supporting better explainability.

METHODS

In this method, we first use outperformed deep learning teacher models to learn the complex patterns hidden in high-dimensional multivariate time series data. Then, we distill knowledge from soft labels generated by the ensemble of teacher models to guide the training of XGBoost student model, whose inputs are meaningful features obtained from feature engineering. Finally, we conduct model calibration to obtain predicted probabilities reflecting the true posterior probabilities and use SHapley Additive exPlanations (SHAP) to obtain insights about the trained model.

RESULTS

We conduct comprehensive experiments on MIMIC-III dataset to evaluate our method. The results demonstrate that our method achieves better predictive performance than vanilla XGBoost, deep learning models and several state-of-art baselines from related works. Our method can also provide intuitive explanations.

CONCLUSIONS

Our method is useful for improving the predictive performance of XGBoost by distilling knowledge from deep learning models and can provide meaningful explanations for predictions.

Multi-dimensional patient acuity estimation with longitudinal EHR tokenization and flexible transformer networks

Transformer model architectures have revolutionized the natural language processing (NLP) domain and continue to produce state-of-the-art results in text-based applications. Prior to the emergence of transformers, traditional NLP models such as recurrent and convolutional neural networks demonstrated promising utility for patient-level predictions and health forecasting from longitudinal datasets. However, to our knowledge only few studies have explored transformers for predicting clinical outcomes from electronic health record (EHR) data, and in our estimation, none have adequately derived a health-specific tokenization scheme to fully capture the heterogeneity of EHR systems. In this study, we propose a dynamic method for tokenizing both discrete and continuous patient data, and present a transformer-based classifier utilizing a joint embedding space for integrating disparate temporal patient measurements. We demonstrate the feasibility of our clinical AI framework through multi-task ICU patient acuity estimation, where we simultaneously predict six mortality and readmission outcomes. Our longitudinal EHR tokenization and transformer modeling approaches resulted in more accurate predictions compared with baseline machine learning models, which suggest opportunities for future multimodal data integrations and algorithmic support tools using clinical transformer networks.

Improving Fairness in the Prediction of Heart Failure Length of Stay and Mortality by Integrating Social Determinants of Health

BACKGROUND

Machine learning (ML) approaches have been broadly applied to the prediction of length of stay and mortality in hospitalized patients. ML may also reduce societal health burdens, assist in health resources planning and improve health outcomes. However, the fairness of these ML models across ethnoracial or socioeconomic subgroups is rarely assessed or discussed. In this study, we aim (1) to quantify the algorithmic bias of ML models when predicting the probability of long-term hospitalization or in-hospital mortality for different heart failure (HF) subpopulations, and (2) to propose a novel method that can improve the fairness of our models without compromising predictive power.

METHODS

We built 5 ML classifiers to predict the composite outcome of hospitalization length-of-stay and in-hospital mortality for 210 368 HF patients extracted from the Get With The Guidelines-Heart Failure registry data set. We integrated 15 social determinants of health variables, including the Social Deprivation Index and the Area Deprivation Index, into the feature space of ML models based on patients' geographies to mitigate the algorithmic bias.

RESULTS

The best-performing random forest model demonstrated modest predictive power but selectively underdiagnosed underserved subpopulations, for example, female, Black, and socioeconomically disadvantaged patients. The integration of social determinants of health variables can significantly improve fairness without compromising model performance.

CONCLUSIONS

We quantified algorithmic bias against underserved subpopulations in the prediction of the composite outcome for HF patients. We provide a potential direction to reduce disparities of ML-based predictive models by integrating social determinants of health variables. We urge fellow researchers to strongly consider ML fairness when developing predictive models for HF patients.

A Survey on Medical Explainable AI (XAI): Recent Progress, Explainability Approach, Human Interaction and Scoring System

The emerging field of eXplainable AI (XAI) in the medical domain is considered to be of utmost importance. Meanwhile, incorporating explanations in the medical domain with respect to legal and ethical AI is necessary to understand detailed decisions, results, and current status of the patient's conditions. Successively, we will be presenting a detailed survey for the medical XAI with the model enhancements, evaluation methods, significant overview of case studies with open box architecture, medical open datasets, and future improvements. Potential differences in AI and XAI methods are provided with the recent XAI methods stated as (i) local and global methods for preprocessing, (ii) knowledge base and distillation algorithms, and (iii) interpretable machine learning. XAI characteristics details with future healthcare explainability is included prominently, whereas the pre-requisite provides insights for the brainstorming sessions before beginning a medical XAI project. Practical case study determines the recent XAI progress leading to the advance developments within the medical field. Ultimately, this survey proposes critical ideas surrounding a user-in-the-loop approach, with an emphasis on human-machine collaboration, to better produce explainable solutions. The surrounding details of the XAI feedback system for human rating-based evaluation provides intelligible insights into a constructive method to produce human enforced explanation feedback. For a long time, XAI limitations of the ratings, scores and grading are present. Therefore, a novel XAI recommendation system and XAI scoring system are designed and approached from this work. Additionally, this paper encourages the importance of implementing explainable solutions into the high impact medical field.

Exploring Socioeconomic Status as a Global Determinant of COVID-19 Prevalence, Using Exploratory Data Analytic and Supervised Machine Learning Techniques: Algorithm Development and Validation Study

BACKGROUND

The COVID-19 pandemic represents the most unprecedented global challenge in recent times. As the global community attempts to manage the pandemic in the long term, it is pivotal to understand what factors drive prevalence rates and to predict the future trajectory of the virus.

OBJECTIVE

This study had 2 objectives. First, it tested the statistical relationship between socioeconomic status and COVID-19 prevalence. Second, it used machine learning techniques to predict cumulative COVID-19 cases in a multicountry sample of 182 countries. Taken together, these objectives will shed light on socioeconomic status as a global risk factor of the COVID-19 pandemic.

METHODS

This research used exploratory data analysis and supervised machine learning methods. Exploratory analysis included variable distribution, variable correlations, and outlier detection. Following this, the following 3 supervised regression techniques were applied: linear regression, random forest, and adaptive boosting (AdaBoost). Results were evaluated using k-fold cross-validation and subsequently compared to analyze algorithmic suitability. The analysis involved 2 models. First, the algorithms were trained to predict 2021 COVID-19 prevalence using only 2020 reported case data. Following this, socioeconomic indicators were added as features and the algorithms were trained again. The Human Development Index (HDI) metrics of life expectancy, mean years of schooling, expected years of schooling, and gross national income were used to approximate socioeconomic status.

RESULTS

All variables correlated positively with the 2021 COVID-19 prevalence, with R2 values ranging from 0.55 to 0.85. Using socioeconomic indicators, COVID-19 prevalence was predicted with a reasonable degree of accuracy. Using 2020 reported case rates as a lone predictor to predict 2021 prevalence rates, the average predictive accuracy of the algorithms was low (R2=0.543). When socioeconomic indicators were added alongside 2020 prevalence rates as features, the average predictive performance improved considerably (R2=0.721) and all error statistics decreased. Thus, adding socioeconomic indicators alongside 2020 reported case data optimized the prediction of COVID-19 prevalence to a considerable degree. Linear regression was the strongest learner with R2=0.693 on the first model and R2=0.763 on the second model, followed by random forest (0.481 and 0.722) and AdaBoost (0.454 and 0.679). Following this, the second model was retrained using a selection of additional COVID-19 risk factors (population density, median age, and vaccination uptake) instead of the HDI metrics. However, average accuracy dropped to 0.649, which highlights the value of socioeconomic status as a predictor of COVID-19 cases in the chosen sample.

CONCLUSIONS

The results show that socioeconomic status is an important variable to consider in future epidemiological modeling, and highlights the reality of the COVID-19 pandemic as a social phenomenon and a health care phenomenon. This paper also puts forward new considerations about the application of statistical and machine learning techniques to understand and combat the COVID-19 pandemic.

Uncertainty-aware deep learning in healthcare: A scoping review

Mistrust is a major barrier to implementing deep learning in healthcare settings. Entrustment could be earned by conveying model certainty, or the probability that a given model output is accurate, but the use of uncertainty estimation for deep learning entrustment is largely unexplored, and there is no consensus regarding optimal methods for quantifying uncertainty. Our purpose is to critically evaluate methods for quantifying uncertainty in deep learning for healthcare applications and propose a conceptual framework for specifying certainty of deep learning predictions. We searched Embase, MEDLINE, and PubMed databases for articles relevant to study objectives, complying with PRISMA guidelines, rated study quality using validated tools, and extracted data according to modified CHARMS criteria. Among 30 included studies, 24 described medical imaging applications. All imaging model architectures used convolutional neural networks or a variation thereof. The predominant method for quantifying uncertainty was Monte Carlo dropout, producing predictions from multiple networks for which different neurons have dropped out and measuring variance across the distribution of resulting predictions. Conformal prediction offered similar strong performance in estimating uncertainty, along with ease of interpretation and application not only to deep learning but also to other machine learning approaches. Among the six articles describing non-imaging applications, model architectures and uncertainty estimation methods were heterogeneous, but predictive performance was generally strong, and uncertainty estimation was effective in comparing modeling methods. Overall, the use of model learning curves to quantify epistemic uncertainty (attributable to model parameters) was sparse. Heterogeneity in reporting methods precluded the performance of a meta-analysis. Uncertainty estimation methods have the potential to identify rare but important misclassifications made by deep learning models and compare modeling methods, which could build patient and clinician trust in deep learning applications in healthcare. Efficient maturation of this field will require standardized guidelines for reporting performance and uncertainty metrics.

Association Between Glycemic Gap and Mortality in Critically Ill Patients with Diabetes

OBJECTIVES

Dysglycemia is associated with poor outcomes in critically ill patients,which is uncertain in patients with diabetes regarding to the situation of glucose control before hospitalization. This study was aimed to investigate the effect of the difference between the level of blood glucose during ICU stay and before admission to ICU upon the outcomes of critically ill patients with diabetes.

METHOD

Patients with diabetes expected to stay for more than 24hs were enrolled, HbA1c was converted to A1C-derived average glucose (ADAG) by the equation: ADAG  =  [ (HbA1c * 28.7) - 46.7 ] * 18^-1, blood glucose were measured four times a day during the first 7 days after admission, the mean glucose level(MGL) and SOFA (within 3, 5, and 7days) were calculated for each person, GAP_adm and GAP_mean was calculated as admission blood glucose and MGL minus ADAG, the incidence of moderate hypoglycemia(MH), severe hypoglycemia (SH), total dosage of glucocorticoids and average daily dosage of insulin, duration of renal replacement therapy(RRT), ventilator-free hours, and non-ICU days were also collected. Patients were divided into survival group and nonsurvival group according to survival or not at 28-day, the relationship between GAP and mortality were analyzed.

RESULTS

431 patients were divided into survival group and nonsurvival group. The two groups had a comparable level of HbA1c, the nonsurvivors had greater APACHE II, SOFA, GAP_adm, GAP_mean-3, GAP_mean-5, GAP_mean-7 and higher MH and SH incidences. Less duration of ventilator-free, non-ICU stay and longer duration of RRT were recorded in the nonsurvival group. GAP_mean-5 had the greatest predictive power with an AUC of 0.807(95%CI: 0.762-0.851), the cut-off value was 3.6 mmol/L (sensitivity 77.7% and specificity 76.6%). The AUC was increased to 0.852(95%CI: 0.814-0.889) incorporated with SOFA₅ (NRI  =  11.34%).

CONCLUSION

Glycemic GAP between the MGL within 5 days and ADAG was independently associated with 28-day mortality of critically ill patients with diabetes. The predictive power was optimized with addition of SOFA₅.

Demystifying the Black Box: The Importance of Interpretability of Predictive Models in Neurocritical Care

Neurocritical care patients are a complex patient population, and to aid clinical decision-making, many models and scoring systems have previously been developed. More recently, techniques from the field of machine learning have been applied to neurocritical care patient data to develop models with high levels of predictive accuracy. However, although these recent models appear clinically promising, their interpretability has often not been considered and they tend to be black box models, making it extremely difficult to understand how the model came to its conclusion. Interpretable machine learning methods have the potential to provide the means to overcome some of these issues but are largely unexplored within the neurocritical care domain. This article examines existing models used in neurocritical care from the perspective of interpretability. Further, the use of interpretable machine learning will be explored, in particular the potential benefits and drawbacks that the techniques may have when applied to neurocritical care data. Finding a solution to the lack of model explanation, transparency, and accountability is important because these issues have the potential to contribute to model trust and clinical acceptance, and, increasingly, regulation is stipulating a right to explanation for decisions made by models and algorithms. To ensure that the prospective gains from sophisticated predictive models to neurocritical care provision can be realized, it is imperative that interpretability of these models is fully considered.

E-CatBoost: An efficient machine learning framework for predicting ICU mortality using the eICU Collaborative Research Database

Improving the Intensive Care Unit (ICU) management network and building cost-effective and well-managed healthcare systems are high priorities for healthcare units. Creating accurate and explainable mortality prediction models helps identify the most critical risk factors in the patients' survival/death status and early detect the most in-need patients. This study proposes a highly accurate and efficient machine learning model for predicting ICU mortality status upon discharge using the information available during the first 24 hours of admission. The most important features in mortality prediction are identified, and the effects of changing each feature on the prediction are studied. We used supervised machine learning models and illness severity scoring systems to benchmark the mortality prediction. We also implemented a combination of SHAP, LIME, partial dependence, and individual conditional expectation plots to explain the predictions made by the best-performing model (CatBoost). We proposed E-CatBoost, an optimized and efficient patient mortality prediction model, which can accurately predict the patients' discharge status using only ten input features. We used eICU-CRD v2.0 to train and validate the models; the dataset contains information on over 200,000 ICU admissions. The patients were divided into twelve disease groups, and models were fitted and tuned for each group. The models' predictive performance was evaluated using the area under a receiver operating curve (AUROC). The AUROC scores were 0.86 [std:0.02] to 0.92 [std:0.02] for CatBoost and 0.83 [std:0.02] to 0.91 [std:0.03] for E-CatBoost models across the defined disease groups; if measured over the entire patient population, their AUROC scores were 7 to 18 and 2 to 12 percent higher than the baseline models, respectively. Based on SHAP explanations, we found age, heart rate, respiratory rate, blood urine nitrogen, and creatinine level as the most critical cross-disease features in mortality predictions.

Performance of a Machine Learning Algorithm Using Electronic Health Record Data to Predict Postoperative Complications and Report on a Mobile Platform

IMPORTANCE

Predicting postoperative complications has the potential to inform shared decisions regarding the appropriateness of surgical procedures, targeted risk-reduction strategies, and postoperative resource use. Realizing these advantages requires that accurate real-time predictions be integrated with clinical and digital workflows; artificial intelligence predictive analytic platforms using automated electronic health record (EHR) data inputs offer an intriguing possibility for achieving this, but there is a lack of high-level evidence from prospective studies supporting their use.

OBJECTIVE

To examine whether the MySurgeryRisk artificial intelligence system has stable predictive performance between development and prospective validation phases and whether it is feasible to provide automated outputs directly to surgeons' mobile devices.

DESIGN, SETTING, AND PARTICIPANTS

In this prognostic study, the platform used automated EHR data inputs and machine learning algorithms to predict postoperative complications and provide predictions to surgeons, previously through a web portal and currently through a mobile device application. All patients 18 years or older who were admitted for any type of inpatient surgical procedure (74 417 total procedures involving 58 236 patients) between June 1, 2014, and September 20, 2020, were included. Models were developed using retrospective data from 52 117 inpatient surgical procedures performed between June 1, 2014, and November 27, 2018. Validation was performed using data from 22 300 inpatient surgical procedures collected prospectively from November 28, 2018, to September 20, 2020.

MAIN OUTCOMES AND MEASURES

Algorithms for generalized additive models and random forest models were developed and validated using real-time EHR data. Model predictive performance was evaluated primarily using area under the receiver operating characteristic curve (AUROC) values.

RESULTS

Among 58 236 total adult patients who received 74 417 major inpatient surgical procedures, the mean (SD) age was 57 (17) years; 29 226 patients (50.2%) were male. Results reported in this article focus primarily on the validation cohort. The validation cohort included 22 300 inpatient surgical procedures involving 19 132 patients (mean [SD] age, 58 [17] years; 9672 [50.6%] male). A total of 2765 patients (14.5%) were Black or African American, 14 777 (77.2%) were White, 1235 (6.5%) were of other races (including American Indian or Alaska Native, Asian, Native Hawaiian or Pacific Islander, and multiracial), and 355 (1.9%) were of unknown race because of missing data; 979 patients (5.1%) were Hispanic, 17 663 (92.3%) were non-Hispanic, and 490 (2.6%) were of unknown ethnicity because of missing data. A greater number of input features was associated with stable or improved model performance. For example, the random forest model trained with 135 input features had the highest AUROC values for predicting acute kidney injury (0.82; 95% CI, 0.82-0.83); cardiovascular complications (0.81; 95% CI, 0.81-0.82); neurological complications, including delirium (0.87; 95% CI, 0.87-0.88); prolonged intensive care unit stay (0.89; 95% CI, 0.88-0.89); prolonged mechanical ventilation (0.91; 95% CI, 0.90-0.91); sepsis (0.86; 95% CI, 0.85-0.87); venous thromboembolism (0.82; 95% CI, 0.81-0.83); wound complications (0.78; 95% CI, 0.78-0.79); 30-day mortality (0.84; 95% CI, 0.82-0.86); and 90-day mortality (0.84; 95% CI, 0.82-0.85), with accuracy similar to surgeons' predictions. Compared with the original web portal, the mobile device application allowed efficient fingerprint login access and loaded data approximately 10 times faster. The application output displayed patient information, risk of postoperative complications, top 3 risk factors for each complication, and patterns of complications for individual surgeons compared with their colleagues.

CONCLUSIONS AND RELEVANCE

In this study, automated real-time predictions of postoperative complications with mobile device outputs had good performance in clinical settings with prospective validation, matching surgeons' predictive accuracy.

Time Series Prediction Using Deep Learning Methods in Healthcare

Traditional Machine Learning (ML) methods face unique challenges when applied to healthcare predictive analytics. The high-dimensional nature of healthcare data necessitates labor-intensive and time-consuming processes when selecting an appropriate set of features for each new task. Furthermore, ML methods depend heavily on feature engineering to capture the sequential nature of patient data, oftentimes failing to adequately leverage the temporal patterns of medical events and their dependencies. In contrast, recent Deep Learning (DL) methods have shown promising performance for various healthcare prediction tasks by specifically addressing the high-dimensional and temporal challenges of medical data. DL techniques excel at learning useful representations of medical concepts and patient clinical data as well as their nonlinear interactions from high-dimensional raw or minimally-processed healthcare data.

In this paper we systematically reviewed research works that focused on advancing deep neural networks to leverage patient structured time series data for healthcare prediction tasks. To identify relevant studies, we searched MEDLINE, IEEE, Scopus, and ACM digital library for relevant publications through November 4 th , 2021. Overall, we found that researchers have contributed to deep time series prediction literature in ten identifiable research streams: DL models, missing value handling, addressing temporal irregularity, patient representation, static data inclusion, attention mechanisms, interpretation, incorporation of medical ontologies, learning strategies, and scalability. This study summarizes research insights from these literature streams, identifies several critical research gaps, and suggests future research opportunities for DL applications using patient time series data.

A time-incorporated SOFA score-based machine learning model for predicting mortality in critically ill patients: A multicenter, real-world study

BACKGROUND

Organ dysfunction (OD) assessment is essential in intensive care units (ICUs). However, current OD assessment scores merely describe the number and the severity of each OD, without evaluating the duration of organ injury. The objective of this study is to develop and validate a machine learning model based on the Sequential Organ Failure Assessment (SOFA) score for the prediction of mortality in critically ill patients.

MATERIAL AND METHODS

Data from the eICU Collaborative Research Database and Medical Information Mart for Intensive Care (MIMIC) -III were mixed for model development. The MIMIC-IV and Nanjing Jinling Hospital Surgical ICU database were used as external test set A and set B, respectively. The outcome of interest was in-ICU mortality. A modified SOFA model incorporating time-dimension (T-SOFA) was stepwise developed to predict ICU mortality using extreme gradient boosting (XGBoost), support vector machine, random forest and logistic regression algorithms. Time-dimensional features were calculated based on six consecutive SOFA scores collected every 12 h within the first three days of admission. The predictive performance was assessed with the area under the receiver operating characteristic curves (AUROC) and calibration plot.

RESULTS

A total of 82,132 patients from the real-world datasets were included in this study, and 7,494 patients (9.12%) died during their ICU stay. The T-SOFA M3 that incorporated the time-dimension features and age, using the XGBoost algorithm, significantly outperformed the original SOFA score in the validation set (AUROC 0.800 95% CI [0.787-0.813] vs. 0.693 95% CI [0.678-0.709], p < 0.01). Good discrimination and calibration were maintained in the test set A and B, with AUROC of 0.803, 95% CI [0.791-0.815] and 0.830, 95% CI [0.789-0.870], respectively.

CONCLUSIONS

The time-incorporated T-SOFA model could significantly improve the prediction performance of the original SOFA score and is of potential for identifying high-risk patients in future clinical application.

State of the Art of Machine Learning-Enabled Clinical Decision Support in Intensive Care Units: Literature Review

Background

Modern clinical care in intensive care units is full of rich data, and machine learning has great potential to support clinical decision-making. The development of intelligent machine learning–based clinical decision support systems is facing great opportunities and challenges. Clinical decision support systems may directly help clinicians accurately diagnose, predict outcomes, identify risk events, or decide treatments at the point of care.

Objective

We aimed to review the research and application of machine learning–enabled clinical decision support studies in intensive care units to help clinicians, researchers, developers, and policy makers better understand the advantages and limitations of machine learning–supported diagnosis, outcome prediction, risk event identification, and intensive care unit point-of-care recommendations.

Methods

We searched papers published in the PubMed database between January 1980 and October 2020. We defined selection criteria to identify papers that focused on machine learning–enabled clinical decision support studies in intensive care units and reviewed the following aspects: research topics, study cohorts, machine learning models, analysis variables, and evaluation metrics.

Results

A total of 643 papers were collected, and using our selection criteria, 97 studies were found. Studies were categorized into 4 topics—monitoring, detection, and diagnosis (13/97, 13.4%), early identification of clinical events (32/97, 33.0%), outcome prediction and prognosis assessment (46/97, 47.6%), and treatment decision (6/97, 6.2%). Of the 97 papers, 82 (84.5%) studies used data from adult patients, 9 (9.3%) studies used data from pediatric patients, and 6 (6.2%) studies used data from neonates. We found that 65 (67.0%) studies used data from a single center, and 32 (33.0%) studies used a multicenter data set; 88 (90.7%) studies used supervised learning, 3 (3.1%) studies used unsupervised learning, and 6 (6.2%) studies used reinforcement learning. Clinical variable categories, starting with the most frequently used, were demographic (n=74), laboratory values (n=59), vital signs (n=55), scores (n=48), ventilation parameters (n=43), comorbidities (n=27), medications (n=18), outcome (n=14), fluid balance (n=13), nonmedicine therapy (n=10), symptoms (n=7), and medical history (n=4). The most frequently adopted evaluation metrics for clinical data modeling studies included area under the receiver operating characteristic curve (n=61), sensitivity (n=51), specificity (n=41), accuracy (n=29), and positive predictive value (n=23).

Conclusions

Early identification of clinical and outcome prediction and prognosis assessment contributed to approximately 80% of studies included in this review. Using new algorithms to solve intensive care unit clinical problems by developing reinforcement learning, active learning, and time-series analysis methods for clinical decision support will be greater development prospects in the future.

Aligning Patient Acuity With Resource Intensity After Major Surgery: A Scoping Review

OBJECTIVE

Develop unifying definitions and paradigms for data-driven methods to augment postoperative resource intensity decisions.

SUMMARY BACKGROUND DATA

Postoperative level-of-care assignments and frequency of vital sign and laboratory measurements (ie, resource intensity) should align with patient acuity. Effective, data-driven decision-support platforms could improve value of care for millions of patients annually, but their development is hindered by the lack of salient definitions and paradigms.

METHODS

Embase, PubMed, and Web of Science were searched for articles describing patient acuity and resource intensity after inpatient surgery. Study quality was assessed using validated tools. Thirty-five studies were included and assimilated according to PRISMA guidelines.

RESULTS

Perioperative patient acuity is accurately represented by combinations of demographic, physiologic, and hospital-system variables as input features in models that capture complex, non-linear relationships. Intraoperative physiologic data enriche these representations. Triaging high-acuity patients to low-intensity care is associated with increased risk for mortality; triaging low-acuity patients to intensive care units (ICUs) has low value and imparts harm when other, valid requests for ICU admission are denied due to resource limitations, increasing their risk for unrecognized decompensation and failure-to-rescue. Providing high-intensity care for low-acuity patients may also confer harm through unnecessary testing and subsequent treatment of incidental findings, but there is insufficient evidence to evaluate this hypothesis. Compared with data-driven models, clinicians exhibit volatile performance in predicting complications and making postoperative resource intensity decisions.

CONCLUSION

To optimize value, postoperative resource intensity decisions should align with precise, data-driven patient acuity assessments augmented by models that accurately represent complex, non-linear relationships among risk factors.

Ideal algorithms in healthcare: Explainable, dynamic, precise, autonomous, fair, and reproducible

Established guidelines describe minimum requirements for reporting algorithms in healthcare; it is equally important to objectify the characteristics of ideal algorithms that confer maximum potential benefits to patients, clinicians, and investigators. We propose a framework for ideal algorithms, including 6 desiderata: explainable (convey the relative importance of features in determining outputs), dynamic (capture temporal changes in physiologic signals and clinical events), precise (use high-resolution, multimodal data and aptly complex architecture), autonomous (learn with minimal supervision and execute without human input), fair (evaluate and mitigate implicit bias and social inequity), and reproducible (validated externally and prospectively and shared with academic communities). We present an ideal algorithms checklist and apply it to highly cited algorithms. Strategies and tools such as the predictive, descriptive, relevant (PDR) framework, the Standard Protocol Items: Recommendations for Interventional Trials-Artificial Intelligence (SPIRIT-AI) extension, sparse regression methods, and minimizing concept drift can help healthcare algorithms achieve these objectives, toward ideal algorithms in healthcare.

Physiologic signatures within six hours of hospitalization identify acute illness phenotypes

During the early stages of hospital admission, clinicians use limited information to make decisions as patient acuity evolves. We hypothesized that clustering analysis of vital signs measured within six hours of hospital admission would reveal distinct patient phenotypes with unique pathophysiological signatures and clinical outcomes. We created a longitudinal electronic health record dataset for 75,762 adult patient admissions to a tertiary care center in 2014-2016 lasting six hours or longer. Physiotypes were derived via unsupervised machine learning in a training cohort of 41,502 patients applying consensus k-means clustering to six vital signs measured within six hours of admission. Reproducibility and correlation with clinical biomarkers and outcomes were assessed in validation cohort of 17,415 patients and testing cohort of 16,845 patients. Training, validation, and testing cohorts had similar age (54-55 years) and sex (55% female), distributions. There were four distinct clusters. Physiotype A had physiologic signals consistent with early vasoplegia, hypothermia, and low-grade inflammation and favorable short-and long-term clinical outcomes despite early, severe illness. Physiotype B exhibited early tachycardia, tachypnea, and hypoxemia followed by the highest incidence of prolonged respiratory insufficiency, sepsis, acute kidney injury, and short- and long-term mortality. Physiotype C had minimal early physiological derangement and favorable clinical outcomes. Physiotype D had the greatest prevalence of chronic cardiovascular and kidney disease, presented with severely elevated blood pressure, and had good short-term outcomes but suffered increased 3-year mortality. Comparing sequential organ failure assessment (SOFA) scores across physiotypes demonstrated that clustering did not simply recapitulate previously established acuity assessments. In a heterogeneous cohort of hospitalized patients, unsupervised machine learning techniques applied to routine, early vital sign data identified physiotypes with unique disease categories and distinct clinical outcomes. This approach has the potential to augment understanding of pathophysiology by distilling thousands of disease states into a few physiological signatures.

Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond

Explainable Artificial Intelligence (XAI) is an emerging research topic of machine learning aimed at unboxing how AI systems' black-box choices are made. This research field inspects the measures and models involved in decision-making and seeks solutions to explain them explicitly. Many of the machine learning algorithms cannot manifest how and why a decision has been cast. This is particularly true of the most popular deep neural network approaches currently in use. Consequently, our confidence in AI systems can be hindered by the lack of explainability in these black-box models. The XAI becomes more and more crucial for deep learning powered applications, especially for medical and healthcare studies, although in general these deep neural networks can return an arresting dividend in performance. The insufficient explainability and transparency in most existing AI systems can be one of the major reasons that successful implementation and integration of AI tools into routine clinical practice are uncommon. In this study, we first surveyed the current progress of XAI and in particular its advances in healthcare applications. We then introduced our solutions for XAI leveraging multi-modal and multi-centre data fusion, and subsequently validated in two showcases following real clinical scenarios. Comprehensive quantitative and qualitative analyses can prove the efficacy of our proposed XAI solutions, from which we can envisage successful applications in a broader range of clinical questions.

Mortality Prediction in the ICU: The Daunting Task of Predicting the Unpredictable

How to cite this article: Kumar AAK. Mortality Prediction in the ICU: The Daunting Task of Predicting the Unpredictable. Indian J Crit Care Med 2022;26(1):13-14.

Computational Signatures for Post-Cardiac Arrest Trajectory Prediction: Importance of Early Physiological Time Series

BACKGROUND

There is an unmet need for timely and reliable prediction of post-cardiac arrest (CA) clinical trajectories. We hypothesized that physiological time series (PTS) data recorded on the first day of intensive care would contribute significantly to discrimination of outcomes at discharge.

PATIENTS AND METHODS

Adult patients in the multicenter eICU database who were mechanically ventilated after resuscitation from out-of-hospital CA were included. Outcomes of interest were survival, neurological status based on Glasgow motor subscore (mGCS) and surrogate functional status based on discharge location (DL), at hospital discharge. Three machine learning predictive models were trained, one with features from the electronic health records (EHR), the second using features derived from PTS collected in the first 24 hours after ICU admission (PTS₂₄), and the third combining PTS₂₄ and EHR. Model performances were compared, and the best performing model was externally validated in the MIMIC-III dataset.

RESULTS

Data from 2,216 admissions were included in the analysis. Discrimination of prediction models combining EHR and PTS₂₄ features was higher than models using either EHR or PTS₂₄ for prediction of survival (AUROC 0.83, 0.82 and 0.79 respectively), neurological outcome (0.87, 0.86 and 0.79 respectively), and DL (0.80, 0.78 and 0.76 respectively). External validation in MIMIC-III (n = 86) produced similar model performance. Feature analysis suggested prognostic significance of previously unknown EHR and PTS₂₄ variables.

CONCLUSION

These results indicate that physiological data recorded in the early phase after CA resuscitation contain signatures that are linked to post-CA outcome. Additionally, they attest to the effectiveness of ML for post-CA predictive modeling.

On Missingness Features in Machine Learning Models for Critical Care: Observational Study

Background

Missing data in electronic health records is inevitable and considered to be nonrandom. Several studies have found that features indicating missing patterns (missingness) encode useful information about a patient’s health and advocate for their inclusion in clinical prediction models. But their effectiveness has not been comprehensively evaluated.

Objective

The goal of the research is to study the effect of including informative missingness features in machine learning models for various clinically relevant outcomes and explore robustness of these features across patient subgroups and task settings.

Methods

A total of 48,336 electronic health records from the 2012 and 2019 PhysioNet Challenges were used, and mortality, length of stay, and sepsis outcomes were chosen. The latter dataset was multicenter, allowing external validation. Gated recurrent units were used to learn sequential patterns in the data and classify or predict labels of interest. Models were evaluated on various criteria and across population subgroups evaluating discriminative ability and calibration.

Results

Generally improved model performance in retrospective tasks was observed on including missingness features. Extent of improvement depended on the outcome of interest (area under the curve of the receiver operating characteristic [AUROC] improved from 1.2% to 7.7%) and even patient subgroup. However, missingness features did not display utility in a simulated prospective setting, being outperformed (0.9% difference in AUROC) by the model relying only on pathological features. This was despite leading to earlier detection of disease (true positives), since including these features led to a concomitant rise in false positive detections.

Conclusions

This study comprehensively evaluated effectiveness of missingness features on machine learning models. A detailed understanding of how these features affect model performance may lead to their informed use in clinical settings especially for administrative tasks like length of stay prediction where they present the greatest benefit. While missingness features, representative of health care processes, vary greatly due to intra- and interhospital factors, they may still be used in prediction models for clinically relevant outcomes. However, their use in prospective models producing frequent predictions needs to be explored further.

A Comparative Analysis of the Respiratory Subscore of the Sequential Organ Failure Assessment Scoring System

Rationale: The Sequential Organ Failure Assessment (SOFA) tool is a commonly used measure of illness severity. Calculation of the respiratory subscore of SOFA is frequently limited by missing arterial oxygen pressure (PaO2) data. Although missing PaO2 data are commonly replaced with normal values, the performance of different methods of substituting PaO2 for SOFA calculation is unclear. Objectives: The study objective was to compare the performance of different substitution strategies for missing PaO2 data for SOFA score calculation. Methods: This retrospective cohort study was performed using the Weill Cornell Critical Care Database for Advanced Research from a tertiary care hospital in the United States. All adult patients admitted to an intensive care unit (ICU) from 2011 to 2019 with an available respiratory SOFA score were included. We analyzed the availability of the PaO2/fraction of inspired oxygen (FiO2) ratio on the first day of ICU admission. In those without a PaO2/FiO2 ratio available, the ratio of oxygen saturation as measured by pulse oximetry to FiO2 was used to calculate a respiratory SOFA subscore according to four methods (linear substitution [Rice], nonlinear substitution [Severinghaus], modified respiratory SOFA, and multiple imputation by chained equations [MICE]) as well as the missing-as-normal technique. We then compared how well the different total SOFA scores discriminated in-hospital mortality. We performed several subgroup and sensitivity analyses. Results: We identified 35,260 unique visits, of which 9,172 included predominant respiratory failure. PaO2 data were available for 14,939 (47%). The area under the receiver operating characteristic curve for each substitution technique for discriminating in-hospital mortality was higher than that for the missing-as-normal technique (0.78 [0.77-0.79]) in all analyses (modified, 0.80 [0.79-0.81]; Rice, 0.80 [0.79-0.81]; Severinghaus, 0.80 [0.79-0.81]; and MICE, 0.80 [0.79-0.81]) (P < 0.01). Each substitution method had a higher accuracy for discriminating in-hospital mortality (MICE, 0.67; Rice, 0.67; modified, 0.66; and Severinghaus, 0.66) than the missing-as-normal technique. Model calibration for in-hospital mortality was less precise for the missing-as-normal technique than for the other substitution techniques at the lower range of SOFA and among the subgroups. Conclusions: Using physiologic and statistical substitution methods improved the total SOFA score's ability to discriminate mortality compared with the missing-as-normal technique. Treating missing data as normal may result in underreporting the severity of illness compared with using substitution. The simplicity of a direct oxygen saturation as measured by pulse oximetry/FiO2 ratio-modified SOFA technique makes it an attractive choice for electronic health record-based research. This knowledge can inform comparisons of severity of illness across studies that used different techniques.