Prediction of Clinical Deterioration in Hospitalized Adult Patients with Hematologic Malignancies Using a Neural Network Model

Real-Time Machine Learning Alerts to Prevent Escalation of Care: A Nonrandomized Clustered Pragmatic Clinical Trial

OBJECTIVES

Machine learning algorithms can outperform older methods in predicting clinical deterioration, but rigorous prospective data on their real-world efficacy are limited. We hypothesized that real-time machine learning generated alerts sent directly to front-line providers would reduce escalations.

DESIGN

Single-center prospective pragmatic nonrandomized clustered clinical trial.

SETTING

Academic tertiary care medical center.

PATIENTS

Adult patients admitted to four medical-surgical units. Assignment to intervention or control arms was determined by initial unit admission.

INTERVENTIONS

Real-time alerts stratified according to predicted likelihood of deterioration sent either to the primary team or directly to the rapid response team (RRT). Clinical care and interventions were at the providers' discretion. For the control units, alerts were generated but not sent, and standard RRT activation criteria were used.

MEASUREMENTS AND MAIN RESULTS

The primary outcome was the rate of escalation per 1000 patient bed days. Secondary outcomes included the frequency of orders for fluids, medications, and diagnostic tests, and combined in-hospital and 30-day mortality. Propensity score modeling with stabilized inverse probability of treatment weight (IPTW) was used to account for differences between groups. Data from 2740 patients enrolled between July 2019 and March 2020 were analyzed (1488 intervention, 1252 control). Average age was 66.3 years and 1428 participants (52%) were female. The rate of escalation was 12.3 vs. 11.3 per 1000 patient bed days (difference, 1.0; 95% CI, -2.8 to 4.7) and IPTW adjusted incidence rate ratio 1.43 (95% CI, 1.16-1.78; p < 0.001). Patients in the intervention group were more likely to receive cardiovascular medication orders (16.1% vs. 11.3%; 4.7%; 95% CI, 2.1-7.4%) and IPTW adjusted relative risk (RR) (1.74; 95% CI, 1.39-2.18; p < 0.001). Combined in-hospital and 30-day-mortality was lower in the intervention group (7% vs. 9.3%; -2.4%; 95% CI, -4.5% to -0.2%) and IPTW adjusted RR (0.76; 95% CI, 0.58-0.99; p = 0.045).

CONCLUSIONS

Real-time machine learning alerts do not reduce the rate of escalation but may reduce mortality.

RAPID-ED: A predictive model for risk assessment of patient's early in-hospital deterioration from emergency department

Introduction

The objective of this multi-center retrospective cohort study was to devise a predictive tool known as RAPID-ED. This model identifies non-traumatic adult patients at significant risk for cardiac arrest within 48 hours post-admission from the emergency department.

Methods

Data from 224,413 patients admitted through the emergency department (2016-2020) was analyzed, incorporating vital signs, lab tests, and administered therapies. A multivariable regression model was devised to anticipate early cardiac arrest. The efficacy of the RAPID-ED model was evaluated against traditional scoring systems like National Early Warning Score (NEWS) and Modified Early Warning Score (MEWS) and its predictive ability was gauged via the area under the receiver operating characteristic curve (AUC) in both hold-out validation set and external validation set.

Results

RAPID-ED outperformed traditional models in predicting cardiac arrest with an AUC of 0.819 in the hold-out validation set and 0.807 in the external validation set. In this critical care update, RAPID-ED offers an innovative approach to assessing patient risk, aiding emergency physicians in post-discharge care decisions from the emergency department. High-risk score patients (≥13) may benefit from early ICU admission for intensive monitoring.

Conclusion

As we progress with advancements in critical care, tools like RAPID-ED will prove instrumental in refining care strategies for critically ill patients, fostering an improved prognosis and potentially mitigating mortality rates.

Development and validation of predictive model based on deep learning method for classification of dyslipidemia in Chinese medicine

Backgrounds

Dyslipidemia is a prominent risk factor for cardiovascular diseases and one of the primary independent modifiable factors of diabetes and stroke. Statins can significantly improve the prognosis of dyslipidemia, but its side effects cannot be ignored. Traditional Chinese Medicine (TCM) has been used in clinical practice for more than 2000 years in China and has certain traits in treating dyslipidemia with little side effect. Previous research has shown that Mutual Obstruction of Phlegm and Stasis (MOPS) is the most common dyslipidemia type classified in TCM. However, how to compose diagnostic factors in TCM into diagnostic rules relies heavily on the doctor's experience, falling short in standardization and objectiveness. This is a limit for TCM to play its advantages of treating dyslipidemia with MOPS.

Methods

In this study, the syndrome diagnosis in TCM was transformed into the prediction and classification problem in artificial intelligence The deep learning method was employed to build the classification prediction models for dyslipidemia. The models were built and trained with a large amount of multi-centered clinical data on MOPS. The optimal model was screened out by evaluating the performance of prediction models through loss, accuracy, precision, recall, confusion matrix, PR and ROC curve (including AUC).

Results

A total of 20 models were constructed through the deep learning method. All of them performed well in the prediction of dyslipidemia with MOPS. The model-11 is the optimal model. The evaluation indicators of model-11 are as follows: The true positive (TP), false positive (FP), true negative (TN) and false negative (FN) are 51, 15, 129, and 9, respectively. The loss is 0.3241, accuracy is 0.8672, precision is 0.7138, recall is 0.8286, and the AUC is 0.9268. After screening through 89 diagnostic factors of TCM, we identified 36 significant diagnosis factors for dyslipidemia with MOPS. The most outstanding diagnostic factors from the importance were dark purple tongue, slippery pulse and slimy fur, etc.

Conclusions

This study successfully developed a well-performing classification prediction model for dyslipidemia with MOPS, transforming the syndrome diagnosis problem in TCM into a prediction and classification problem in artificial intelligence. Patients with dyslipidemia of MOPS can be accurately recognized through limited information from patients. We also screened out significant diagnostic factors for composing diagnostic rules of dyslipidemia with MOPS. The study is an avant-garde attempt at introducing the deep-learning method into the research of TCM, which provides a useful reference for the extension of deep learning method to other diseases and the construction of disease diagnosis model in TCM, contributing to the standardization and objectiveness of TCM diagnosis.

Sepsis and acute respiratory failure in patients with cancer: how can we improve care and outcomes even further?

PURPOSE OF REVIEW

Care and outcomes of critically ill patients with cancer have improved over the past decade. This selective review will discuss recent updates in sepsis and acute respiratory failure among patients with cancer, with particular focus on important opportunities to improve outcomes further through attention to phenotyping, predictive analytics, and improved outcome measures.

RECENT FINDINGS

The prevalence of cancer diagnoses in intensive care units (ICUs) is nontrivial and increasing. Sepsis and acute respiratory failure remain the most common critical illness syndromes affecting these patients, although other complications are also frequent. Recent research in oncologic sepsis has described outcome variation - including ICU, hospital, and 28-day mortality - across different types of cancer (e.g., solid vs. hematologic malignancies) and different sepsis definitions (e.g., Sepsis-3 vs. prior definitions). Research in acute respiratory failure in oncology patients has highlighted continued uncertainty in the value of diagnostic bronchoscopy for some patients and in the optimal respiratory support strategy. For both of these syndromes, specific challenges include multifactorial heterogeneity (e.g. in etiology and/or underlying cancer), delayed recognition of clinical deterioration, and complex outcomes measurement.

SUMMARY

Improving outcomes in oncologic critical care requires attention to the heterogeneity of cancer diagnoses, timely recognition and management of critical illness, and defining appropriate ICU outcomes.

Evaluation of machine learning-based models for prediction of clinical deterioration: A systematic literature review

BACKGROUND AND OBJECTIVE

Early identification of patients at risk of deterioration can prevent life-threatening adverse events and shorten length of stay. Although there are numerous models applied to predict patient clinical deterioration, most are based on vital signs and have methodological shortcomings that are not able to provide accurate estimates of deterioration risk. The aim of this systematic review is to examine the effectiveness, challenges, and limitations of using machine learning (ML) techniques to predict patient clinical deterioration in hospital settings.

METHODS

A systematic review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and meta-Analyses (PRISMA) guidelines using EMBASE, MEDLINE Complete, CINAHL Complete, and IEEExplore databases. Citation searching was carried out for studies that met inclusion criteria. Two reviewers used the inclusion/exclusion criteria to independently screen studies and extract data. To address any discrepancies in the screening process, the two reviewers discussed their findings and a third reviewer was consulted as needed to reach a consensus. Studies focusing on use of ML in predicting patient clinical deterioration that were published from inception to July 2022 were included.

RESULTS

A total of 29 primary studies that evaluated ML models to predict patient clinical deterioration were identified. After reviewing these studies, we found that 15 types of ML techniques have been employed to predict patient clinical deterioration. While six studies used a single technique exclusively, several others utilised a combination of classical techniques, unsupervised and supervised learning, as well as other novel techniques. Depending on which ML model was applied and the type of input features, ML models predicted outcomes with an area under the curve from 0.55 to 0.99.

CONCLUSIONS

Numerous ML methods have been employed to automate the identification of patient deterioration. Despite these advancements, there is still a need for further investigation to examine the application and effectiveness of these methods in real-world situations.

Artificial Intelligence in Resuscitation: A Scoping Review

Introduction: Cardiac arrest is a significant cause of premature mortality and severe disability. Despite the death rate steadily decreasing over the previous decade, only 22% of survivors achieve good clinical status and only 25% of patients survive until their discharge from the hospital. The objective of this scoping review was to review relevant AI modalities and the main potential applications of AI in resuscitation. Methods: We conducted the literature search for related studies in PubMed, EMBASE, and Google Scholar. We included peer-reviewed publications and articles in the press, pooling and characterizing the data by their model types, goals, and benefits. Results: After identifying 268 original studies, we chose 59 original studies (reporting 1,817,419 patients) to include in the qualitative synthesis. AI-based methods appear to be superior to traditional methods in achieving high-level performance. Conclusion: AI might be useful in predicting cardiac arrest, heart rhythm disorders, and post-cardiac arrest outcomes, as well as in the delivery of drone-delivered defibrillators and notification of dispatchers. AI-powered technologies could be valuable assistants to continuously track patient conditions. Healthcare professionals should assist in the research and development of AI-powered technologies as well as their implementation into clinical practice.

Prediction of extubation failure in the paediatric cardiac ICU using machine learning and high-frequency physiologic data

BACKGROUND

Cardiac intensivists frequently assess patient readiness to wean off mechanical ventilation with an extubation readiness trial despite it being no more effective than clinician judgement alone. We evaluated the utility of high-frequency physiologic data and machine learning for improving the prediction of extubation failure in children with cardiovascular disease.

METHODS

This was a retrospective analysis of clinical registry data and streamed physiologic extubation readiness trial data from one paediatric cardiac ICU (12/2016-3/2018). We analysed patients' final extubation readiness trial. Machine learning methods (classification and regression tree, Boosting, Random Forest) were performed using clinical/demographic data, physiologic data, and both datasets. Extubation failure was defined as reintubation within 48 hrs. Classifier performance was assessed on prediction accuracy and area under the receiver operating characteristic curve.

RESULTS

Of 178 episodes, 11.2% (N = 20) failed extubation. Using clinical/demographic data, our machine learning methods identified variables such as age, weight, height, and ventilation duration as being important in predicting extubation failure. Best classifier performance with this data was Boosting (prediction accuracy: 0.88; area under the receiver operating characteristic curve: 0.74). Using physiologic data, our machine learning methods found oxygen saturation extremes and descriptors of dynamic compliance, central venous pressure, and heart/respiratory rate to be of importance. The best classifier in this setting was Random Forest (prediction accuracy: 0.89; area under the receiver operating characteristic curve: 0.75). Combining both datasets produced classifiers highlighting the importance of physiologic variables in determining extubation failure, though predictive performance was not improved.

CONCLUSION

Physiologic variables not routinely scrutinised during extubation readiness trials were identified as potential extubation failure predictors. Larger analyses are necessary to investigate whether these markers can improve clinical decision-making.

Computational approaches to alleviate alarm fatigue in intensive care medicine: A systematic literature review

Patient monitoring technology has been used to guide therapy and alert staff when a vital sign leaves a predefined range in the intensive care unit (ICU) for decades. However, large amounts of technically false or clinically irrelevant alarms provoke alarm fatigue in staff leading to desensitisation towards critical alarms. With this systematic review, we are following the Preferred Reporting Items for Systematic Reviews (PRISMA) checklist in order to summarise scientific efforts that aimed to develop IT systems to reduce alarm fatigue in ICUs. 69 peer-reviewed publications were included. The majority of publications targeted the avoidance of technically false alarms, while the remainder focused on prediction of patient deterioration or alarm presentation. The investigated alarm types were mostly associated with heart rate or arrhythmia, followed by arterial blood pressure, oxygen saturation, and respiratory rate. Most publications focused on the development of software solutions, some on wearables, smartphones, or headmounted displays for delivering alarms to staff. The most commonly used statistical models were tree-based. In conclusion, we found strong evidence that alarm fatigue can be alleviated by IT-based solutions. However, future efforts should focus more on the avoidance of clinically non-actionable alarms which could be accelerated by improving the data availability.

Systematic Review Registration:https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021233461, identifier: CRD42021233461.

Comparison of Deep Learning Algorithms in Predicting Expert Assessments of Pain Scores during Surgical Operations Using Analgesia Nociception Index

There are many surgical operations performed daily in operation rooms worldwide. Adequate anesthesia is needed during an operation. Besides hypnosis, adequate analgesia is critical to prevent autonomic reactions. Clinical experience and vital signs are usually used to adjust the dosage of analgesics. Analgesia nociception index (ANI), which ranges from 0 to 100, is derived from heart rate variability (HRV) via electrocardiogram (ECG) signals, for pain evaluation in a non-invasive manner. It represents parasympathetic activity. In this study, we compared the performance of multilayer perceptron (MLP) and long short-term memory (LSTM) algorithms in predicting expert assessment of pain score (EAPS) based on patient's HRV during surgery. The objective of this study was to analyze how deep learning models differed from the medical doctors' predictions of EAPS. As the input and output features of the deep learning models, the opposites of ANI and EAPS were used. This study included 80 patients who underwent operations at National Taiwan University Hospital. Using MLP and LSTM, a holdout method was first applied to 60 training patients, 10 validation patients, and 10 testing patients. As compared to the LSTM model, which had a testing mean absolute error (MAE) of 2.633 ± 0.542, the MLP model had a testing MAE of 2.490 ± 0.522, with a more appropriate shape of its prediction curves. The model based on MLP was selected as the best. Using MLP, a seven-fold cross validation method was then applied. The first fold had the lowest testing MAE of 2.460 ± 0.634, while the overall MAE for the seven-fold cross validation method was 2.848 ± 0.308. In conclusion, HRV analysis using MLP algorithm had a good correlation with EAPS; therefore, it can play role as a continuous monitor to predict intraoperative pain levels, to assist physicians in adjusting analgesic agent dosage. Further studies may consider obtaining more input features, such as photoplethysmography (PPG) and other kinds of continuous variable, to improve the prediction performance.

Definition and Classification of Postoperative Complications After Cardiac Surgery: A Pilot Delphi Study (Preprint)

Background

Postoperative complications following cardiac surgery are common and represent a serious burden to health services and society. However, there is a lack of consensus among experts on what events should be considered as a “complication” and how to assess their severity.

Objective

This study aimed to consult domain experts to pilot the development of a definition and classification system for complications following cardiac surgery with the goal to allow the progression of standardized clinical processes and systems in cardiac surgery.

Methods

We conducted a Delphi study, which is a well-established method to reach expert consensus on complex topics. We sent 2 rounds of surveys to domain experts, including cardiac surgeons and anesthetists, to define and classify postoperative complications following cardiac surgery. The responses to open-ended questions were analyzed using a thematic analysis framework.

Results

In total, 71 and 37 experts’ opinions were included in the analysis in Round 1 and Round 2 of the study, respectively. Cardiac anesthetists and cardiac critical care specialists took part in the study. Cardiac surgeons did not participate. Experts agreed that a classification for postoperative complications for cardiac surgery is useful, and consensus was reached for the generic definition of a postoperative complication in cardiac surgery. Consensus was also reached on classification of complications according to the following 4 levels: “Mild,” “Moderate,” “Severe,” and “Death.” Consensus was also reached on definitions for “Mild” and “Severe” categories of complications.

Conclusions

Domain experts agreed on the definition and classification of complications in cardiac surgery for “Mild” and “Severe” complications. The standardization of complication identification, recording, and reporting in cardiac surgery should help the development of quality benchmarks, clinical audit, care quality assessment, resource planning, risk management, communication, and research.

Digitally enabled hemovigilance allows real time response to transfusion reactions

BACKGROUND

Transfusion carries a risk of transfusion reaction that is often underdiagnosed due to reliance on passive reporting. The study investigated the utility of digital methods to identify potential transfusion reactions, thus allowing real-time intervention for affected patients.

METHOD

The hemovigilance unit monitored 3856 patients receiving 43,515 transfusions under the hemovigilance program. Retrospective comparison data included 298,498 transfusions. Transfusion medicine physicians designed and validated algorithms in the electronic health record that analyze discrete data, such as vital sign changes, to assign a risk score during each transfusion. Dedicated hemovigilance nurses remotely monitor all patients and perform real-time chart reviews prioritized by risk score. When a reaction is suspected, a hemovigilance trained licensed clinician responds to manage the patient and ensure data collection. Board-certified transfusion medicine physicians reviewed data and classified transfusion reactions under various categories according to the Centers for Disease Control hemovigilance definitions.

RESULTS

Transfusion medicine physicians diagnosed 564 transfusion reactions (1.3% of transfusions)-a 524% increase compared to the previous passive reporting. The rapid response provider reached the bedside on average at 12.4 min demonstrating logistic feasibility. While febrile reactions were most diagnosed, recognition of transfusion-associated circulatory overload demonstrated the greatest relative increase. Auditing and education programs further enhanced transfusion reaction awareness.

DISCUSSION

The model of digitally-enabled expert real-time review of clinical data that prompts rapid response improved recognition of transfusion reactions. This approach could be applied to other patient deterioration events such as early identification of sepsis.

A Clinical Prognostic Model Based on Machine Learning from the Fondazione Italiana Linfomi (FIL) MCL0208 Phase III Trial

Simple Summary

The interest in using Machine-Learning (ML) techniques in clinical research is growing. We applied ML to build up a novel prognostic model from patients affected with Mantle Cell Lymphoma (MCL) enrolled in a phase III open-labeled, randomized clinical trial from the Fondazione Italiana Linfomi (FIL)—MCL0208. This is the first application of ML in a prospective clinical trial on MCL lymphoma. We applied a novel ML pipeline to a large cohort of patients for which several clinical variables have been collected at baseline, and assessed their prognostic value based on overall survival. We validated it on two independent data series provided by European MCL Network. Due to its flexibility, we believe that ML would be of tremendous help in the development of a novel MCL prognostic score aimed at re-defining risk stratification.

Abstract

Background: Multicenter clinical trials are producing growing amounts of clinical data. Machine Learning (ML) might facilitate the discovery of novel tools for prognostication and disease-stratification. Taking advantage of a systematic collection of multiple variables, we developed a model derived from data collected on 300 patients with mantle cell lymphoma (MCL) from the Fondazione Italiana Linfomi-MCL0208 phase III trial (NCT02354313). Methods: We developed a score with a clustering algorithm applied to clinical variables. The candidate score was correlated to overall survival (OS) and validated in two independent data series from the European MCL Network (NCT00209222, NCT00209209); Results: Three groups of patients were significantly discriminated: Low, Intermediate (Int), and High risk (High). Seven discriminants were identified by a feature reduction approach: albumin, Ki-67, lactate dehydrogenase, lymphocytes, platelets, bone marrow infiltration, and B-symptoms. Accordingly, patients in the Int and High groups had shorter OS rates than those in the Low and Int groups, respectively (Int→Low, HR: 3.1, 95% CI: 1.0–9.6; High→Int, HR: 2.3, 95% CI: 1.5–4.7). Based on the 7 markers, we defined the engineered MCL international prognostic index (eMIPI), which was validated and confirmed in two independent cohorts; Conclusions: We developed and validated a ML-based prognostic model for MCL. Even when currently limited to baseline predictors, our approach has high scalability potential.

Algorithms for Prediction of Clinical Deterioration on the General Wards: A Scoping Review

OBJECTIVE

The primary objective of this scoping review was to identify and describe state-of-the-art models that use vital sign monitoring to predict clinical deterioration on the general ward. The secondary objective was to identify facilitators, barriers, and effects of implementing these models.

DATA SOURCES

PubMed, Embase, and CINAHL databases until November 2020.

STUDY SELECTION

We selected studies that compared vital signs-based automated real-time predictive algorithms to current track-and-trace protocols in regard to the outcome of clinical deterioration in a general ward population.

DATA EXTRACTION

Study characteristics, predictive characteristics and barriers, facilitators, and effects.

RESULTS

We identified 1741 publications, 21 of which were included in our review. Two of the these were clinical trials, 2 were prospective observational studies, and the remaining 17 were retrospective studies. All of the studies focused on hospitalized adult patients. The reported area under the receiver operating characteristic curves ranged between 0.65 and 0.95 for the outcome of clinical deterioration. Positive predictive value and sensitivity ranged between 0.223 and 0.773 and from 7.2% to 84.0%, respectively. Input variables differed widely, and predicted endpoints were inconsistently defined. We identified 57 facilitators and 48 barriers to the implementation of these models. We found 68 reported effects, 57 of which were positive.

CONCLUSION

Predictive algorithms can detect clinical deterioration on the general ward earlier and more accurately than conventional protocols, which in one recent study led to lower mortality. Consensus is needed on input variables, predictive time horizons, and definitions of endpoints to better facilitate comparative research.

Machine Learning for Clinical Outcome Prediction

Clinical decision-making in healthcare is already being influenced by predictions or recommendations made by data-driven machines. Numerous machine learning applications have appeared in the latest clinical literature, especially for outcome prediction models, with outcomes ranging from mortality and cardiac arrest to acute kidney injury and arrhythmia. In this review article, we summarize the state-of-the-art in related works covering data processing, inference, and model evaluation, in the context of outcome prediction models developed using data extracted from electronic health records. We also discuss limitations of prominent modeling assumptions and highlight opportunities for future research.

Toward Modernization of Geriatric Oncology by Digital Health Technologies

The number of older adults with cancer is increasing. Over the past 3 decades, geriatric oncology research has focused on improving the assessment of frailty and fitness of older adults with cancer as well as methods of improving their outcomes. At the same time, advances in digital health technologies have opened new frontiers for reaching this goal. Digital health technologies encompass a variety of solutions, from electronic patient-reported outcomes (ePROs) to Big Data and wireless sensors. These solutions have the potential to further advance our understanding of patients' experiences during cancer treatment. Whereas the data on the feasibility and utility of such solutions in the care of older adults with cancer are limited, interest from digital health oncology researchers to further explore the benefits of these products is increasing. In this article, we describe the focus of geriatric oncology, the rationale behind the need to explore digital health technologies in this setting, and emerging data and ongoing studies, as well as provide guidelines for proper selection, implementation, and testing of digital health solutions in the context of geriatric oncology.

Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review

BACKGROUND

Artificial intelligence (AI) provides opportunities to identify the health risks of patients and thus influence patient safety outcomes.

OBJECTIVE

The purpose of this systematic literature review was to identify and analyze quantitative studies utilizing or integrating AI to address and report clinical-level patient safety outcomes.

METHODS

We restricted our search to the PubMed, PubMed Central, and Web of Science databases to retrieve research articles published in English between January 2009 and August 2019. We focused on quantitative studies that reported positive, negative, or intermediate changes in patient safety outcomes using AI apps, specifically those based on machine-learning algorithms and natural language processing. Quantitative studies reporting only AI performance but not its influence on patient safety outcomes were excluded from further review.

RESULTS

We identified 53 eligible studies, which were summarized concerning their patient safety subcategories, the most frequently used AI, and reported performance metrics. Recognized safety subcategories were clinical alarms (n=9; mainly based on decision tree models), clinical reports (n=21; based on support vector machine models), and drug safety (n=23; mainly based on decision tree models). Analysis of these 53 studies also identified two essential findings: (1) the lack of a standardized benchmark and (2) heterogeneity in AI reporting.

CONCLUSIONS

This systematic review indicates that AI-enabled decision support systems, when implemented correctly, can aid in enhancing patient safety by improving error detection, patient stratification, and drug management. Future work is still needed for robust validation of these systems in prospective and real-world clinical environments to understand how well AI can predict safety outcomes in health care settings.

Machine learning and artificial intelligence in haematology

Digitalization of the medical record and integration of genomic methods into clinical practice have resulted in an unprecedented wealth of data. Machine learning is a subdomain of artificial intelligence that attempts to computationally extract meaningful insights from complex data structures. Applications of machine learning in haematological scenarios are steadily increasing. However, basic concepts are often unfamiliar to clinicians and investigators. The purpose of this review is to provide readers with tools to interpret and critically appraise machine learning literature. We begin with the elucidation of standard terminology and then review examples in haematology. Guidelines for designing and evaluating machine-learning studies are provided. Finally, we discuss limitations of the machine-learning approach.

Early warning score validation methodologies and performance metrics: a systematic review

BACKGROUND

Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation.

METHODS

A systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity.

RESULTS

The key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies.

CONCLUSIONS

Methodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.

MEWS++: Enhancing the Prediction of Clinical Deterioration in Admitted Patients through a Machine Learning Model

Early detection of patients at risk for clinical deterioration is crucial for timely intervention. Traditional detection systems rely on a limited set of variables and are unable to predict the time of decline. We describe a machine learning model called MEWS++ that enables the identification of patients at risk of escalation of care or death six hours prior to the event. A retrospective single-center cohort study was conducted from July 2011 to July 2017 of adult (age > 18) inpatients excluding psychiatric, parturient, and hospice patients. Three machine learning models were trained and tested: random forest (RF), linear support vector machine, and logistic regression. We compared the models’ performance to the traditional Modified Early Warning Score (MEWS) using sensitivity, specificity, and Area Under the Curve for Receiver Operating Characteristic (AUC-ROC) and Precision-Recall curves (AUC-PR). The primary outcome was escalation of care from a floor bed to an intensive care or step-down unit, or death, within 6 h. A total of 96,645 patients with 157,984 hospital encounters and 244,343 bed movements were included. Overall rate of escalation or death was 3.4%. The RF model had the best performance with sensitivity 81.6%, specificity 75.5%, AUC-ROC of 0.85, and AUC-PR of 0.37. Compared to traditional MEWS, sensitivity increased 37%, specificity increased 11%, and AUC-ROC increased 14%. This study found that using machine learning and readily available clinical data, clinical deterioration or death can be predicted 6 h prior to the event. The model we developed can warn of patient deterioration hours before the event, thus helping make timely clinical decisions.

Cell Population Data-Driven Acute Promyelocytic Leukemia Flagging Through Artificial Neural Network Predictive Modeling

A targeted and timely offered treatment can be a benefitting tool for patients with acute promyelocytic leukemia (APML). Current round of study made use of potential morphological and immature fraction-related parameters (cell population data) generated during complete blood cell count (CBC), through artificial neural network (ANN) predictive modeling for early flagging of APML cases. We collected classical CBC items along with cell population data (CPD) from hematology analyzer at diagnosis of 1067 study subjects with hematological neoplasms. For morphological assessment, peripheral blood films were examined. Statistical and machine learning tools including principal component analysis (PCA) helped in the evaluation of predictive capacity of routine and CPD items. Then selected CBC item-driven ANN predictive modeling was developed to smartly use the hidden trend by increasing the auguring accuracy of these parameters in differentiation of APML cases. We found a characteristic triad based on lower (53.73) platelet count (PLT) with decreased/normal (4.72) immature fraction of platelet (IPF) with addition of significantly higher value (65.5) of DNA/RNA content-related neutrophil (NE-SFL) parameter in patients with APML against other hematological neoplasm's groups. On PCA, APML showed exceptionally significant variance for PLT, IPF, and NE-SFL. Through training of ANN predictive modeling, our selected CBC items successfully classify the APML group from non-APML groups at highly significant (0.894) AUC value with lower (2.3 percent) false prediction rate. Practical results of using our ANN model were found acceptable with value of 95.7% and 97.7% for training and testing data sets, respectively. We proposed that PLT, IPF, and NE-SFL could potentially be used for early flagging of APML cases in the hematology-oncology unit. CBC item-driven ANN modeling is a novel approach that substantially strengthen the predictive potential of CBC items, allowing the clinicians to be confident by the typical trend raised by these studied parameters.

Predicting Emergency Visits and Hospital Admissions During Radiation and Chemoradiation: An Internally Validated Pretreatment Machine Learning Algorithm

PURPOSE

Patients undergoing radiotherapy (RT) or chemoradiotherapy (CRT) may require emergency department evaluation or hospitalization. Early identification may direct preventative supportive care, improving outcomes and reducing health care costs. We developed and evaluated a machine learning (ML) approach to predict these events.

METHODS

A total of 8,134 outpatient courses of RT and CRT from a single institution from 2013 to 2016 were identified. Extensive pretreatment data were programmatically extracted and processed from the electronic health record (EHR). Training and internal validation cohorts were randomly generated (3:1 ratio). Gradient tree boosting (GTB), random forest, support vector machine, and least absolute shrinkage and selection operator logistic regression approaches were trained and internally validated based on area under receiver operating characteristic (AUROC) curve. The most predictive ML approach was also evaluated using only disease- and treatment-related factors to assess predictive gain of extensive EHR data.

RESULTS

All methods had high predictive accuracy, particularly GTB (validation AUROC, 0.798). Extensive EHR data beyond disease and treatment information improved accuracy (delta AUROC, 0.056). A Youden-based cutoff corresponded to validation sensitivity of 81.0% (175 of 216 courses with events) and specificity of 67.3% (1,218 of 1811 courses without events). Interpretability is an important advantage of GTB. Variable importance identified top predictive factors, including treatment (planned RT and systemic therapy), pretreatment encounters (emergency department visits and admissions in the year before treatment), vital signs (weight loss and pain score in the year before treatment), and laboratory values (albumin level at weeks before treatment).

CONCLUSION

ML predicts emergency visits and hospitalization during cancer therapy. Incorporating predictions into clinical care algorithms may help direct personalized supportive care, improve quality of care, and reduce costs. A prospective trial investigating ML-assisted direction of increased clinical assessments during RT is planned.

Remote Surveillance Technologies: Realizing the Aim of Right Patient, Right Data, Right Time

The convergence of multiple recent developments in health care information technology and monitoring devices has made possible the creation of remote patient surveillance systems that increase the timeliness and quality of patient care. More convenient, less invasive monitoring devices, including patches, wearables, and biosensors, now allow for continuous physiological data to be gleaned from patients in a variety of care settings across the perioperative experience. These data can be bound into a single data repository, creating so-called data lakes. The high volume and diversity of data in these repositories must be processed into standard formats that can be queried in real time. These data can then be used by sophisticated prediction algorithms currently under development, enabling the early recognition of patterns of clinical deterioration otherwise undetectable to humans. Improved predictions can reduce alarm fatigue. In addition, data are now automatically queriable on a real-time basis such that they can be fed back to clinicians in a time frame that allows for meaningful intervention. These advancements are key components of successful remote surveillance systems. Anesthesiologists have the opportunity to be at the forefront of remote surveillance in the care they provide in the operating room, postanesthesia care unit, and intensive care unit, while also expanding their scope to include high-risk preoperative and postoperative patients on the general care wards. These systems hold the promise of enabling anesthesiologists to detect and intervene upon changes in the clinical status of the patient before adverse events have occurred. Importantly, however, significant barriers still exist to the effective deployment of these technologies and their study in impacting patient outcomes. Studies demonstrating the impact of remote surveillance on patient outcomes are limited. Critical to the impact of the technology are strategies of implementation, including who should receive and respond to alerts and how they should respond. Moreover, the lack of cost-effectiveness data and the uncertainty of whether clinical activities surrounding these technologies will be financially reimbursed remain significant challenges to future scale and sustainability. This narrative review will discuss the evolving technical components of remote surveillance systems, the clinical use cases relevant to the anesthesiologist's practice, the existing evidence for their impact on patients, the barriers that exist to their effective implementation and study, and important considerations regarding sustainability and cost-effectiveness.

Factors Associated With Clinical Deterioration Among Patients Hospitalized on the Wards at a Tertiary Cancer Hospital

PURPOSE

Patients hospitalized outside the intensive care unit (ICU) frequently experience clinical deterioration. Little has been done to describe the landscape of clinical deterioration among inpatients with cancer. We aimed to describe the frequency of clinical deterioration among patients with cancer hospitalized on the wards at a major academic hospital and to identify independent risk factors for clinical deterioration among these patients.

METHODS

This was a retrospective cohort study at a 1,300-bed urban academic hospital with a 138-bed inpatient cancer center. We included consecutive admissions to the oncology wards between January 1, 2014, and June 30, 2017. We defined clinical deterioration as the composite of ward death and transfer to the ICU.

RESULTS

We evaluated 21,219 admissions from 9,058 patients. The composite outcome occurred during 1,945 admissions (9.2%): 1,365 (6.4%) had at least one ICU transfer, and 580 (2.7%) involved ward death. Logistic regression identified several independent risk factors for clinical deterioration, including the following: age (odds ratio [OR], 1.33 per decade; 95% CI, 1.07 to 1.67), male sex (OR, 1.15; 95% CI, 1.05 to 1.33), comorbidities, illness severity (OR, 1.11; 95% CI, 1.10 to 1.13), emergency admission (OR, 1.45; 95% CI, 1.26 to 1.67), hospitalization on particular wards (OR, 1.525; 95% CI, 1.326 to 1.67), bacteremia (OR, 1.24; 95% CI, 1.01 to 1.52), fungemia (OR, 3.76; 95% CI, 1.90 to 7.41), tumor lysis syndrome (OR, 3.01; 95% CI, 2.41 to 3.76), and receipt of antimicrobials (OR, 2.04; 95% CI, 1.72 to 2.42) and transfusions (OR, 1.65; 95% CI, 1.42 to 1.92).

CONCLUSION

Clinical deterioration was common; it occurred in more than 9% of admissions. Factors independently associated with deterioration included comorbidities, admission source, infections, and blood product transfusion.

Nurses' Perceptions of Barriers to Rapid Response System Activation: A Systematic Review

BACKGROUND

The rapid response system (RRS) was designed to identify and intervene on patients exhibiting clinical deterioration in the non-critical-care setting but is not always effectively activated by nurses, leading to adverse patient outcomes.

OBJECTIVES

The objective of this systematic review was to explore nurses' perceived barriers to RRS activation in the acute adult inpatient setting.

METHOD

A systematic review was completed utilizing the Preferred Reporting Items for Systematic Reviews and Meta-analysis checklist. Six different search terms were used in the following electronic databases: Academic Search Premier, the Cumulative Index to Nursing and Allied Health Literature, Healthsource: Nursing/Academic Edition, MEDLINE, and PubMed. Limiters applied to search methods included years 2007 to current, full text, scholarly (peer reviewed), and English language. This review was further limited to quantitative studies in the adult inpatient setting.

RESULTS

The initial electronic database search yielded 149 articles. After duplicate exclusion, 87 article abstracts were reviewed for inclusion and eligibility, and a total of 8 articles were used for this systematic review. Themes to nurses' perceived barriers to RRS activation include RRS activator-responder interaction, physician influence, nurse education, and nurse experience.

DISCUSSION

Nurses play a vital role in patient care by providing continuous surveillance and are the frontline for early detection including prompt intervention should a patient's condition deteriorate. Inconsistent RRS activation has been associated with negative patient outcomes. Exploring nurses' perceived barriers to RRS activation may contribute to interventions that lead to nurses appropriately activating the RRS and potentially decreasing adverse patient outcomes.

Predicting Intensive Care Unit Readmission with Machine Learning Using Electronic Health Record Data

RATIONALE

Patients transferred from the intensive care unit to the wards who are later readmitted to the intensive care unit have increased length of stay, healthcare expenditure, and mortality compared with those who are never readmitted. Improving risk stratification for patients transferred to the wards could have important benefits for critically ill hospitalized patients.

OBJECTIVES

We aimed to use a machine-learning technique to derive and validate an intensive care unit readmission prediction model with variables available in the electronic health record in real time and compare it to previously published algorithms.

METHODS

This observational cohort study was conducted at an academic hospital in the United States with approximately 600 inpatient beds. A total of 24,885 intensive care unit transfers to the wards were included, with 14,962 transfers (60%) in the training cohort and 9,923 transfers (40%) in the internal validation cohort. Patient characteristics, nursing assessments, International Classification of Diseases, Ninth Revision codes from prior admissions, medications, intensive care unit interventions, diagnostic tests, vital signs, and laboratory results were extracted from the electronic health record and used as predictor variables in a gradient-boosted machine model. Accuracy for predicting intensive care unit readmission was compared with the Stability and Workload Index for Transfer score and Modified Early Warning Score in the internal validation cohort and also externally using the Medical Information Mart for Intensive Care database (n = 42,303 intensive care unit transfers).

RESULTS

Eleven percent (2,834) of discharges to the wards were later readmitted to the intensive care unit. The machine-learning-derived model had significantly better performance (area under the receiver operating curve, 0.76) than either the Stability and Workload Index for Transfer score (area under the receiver operating curve, 0.65), or Modified Early Warning Score (area under the receiver operating curve, 0.58; P value < 0.0001 for all comparisons). At a specificity of 95%, the derived model had a sensitivity of 28% compared with 15% for Stability and Workload Index for Transfer score and 7% for the Modified Early Warning Score. Accuracy improvements with the derived model over Modified Early Warning Score and Stability and Workload Index for Transfer were similar in the Medical Information Mart for Intensive Care-III cohort.

CONCLUSIONS

A machine learning approach to predicting intensive care unit readmission was significantly more accurate than previously published algorithms in both our internal validation and the Medical Information Mart for Intensive Care-III cohort. Implementation of this approach could target patients who may benefit from additional time in the intensive care unit or more frequent monitoring after transfer to the hospital ward.

Opening the black box of neural networks: methods for interpreting neural network models in clinical applications

Artificial neural networks (ANNs) are powerful tools for data analysis and are particularly suitable for modeling relationships between variables for best prediction of an outcome. While these models can be used to answer many important research questions, their utility has been critically limited because the interpretation of the "black box" model is difficult. Clinical investigators usually employ ANN models to predict the clinical outcomes or to make a diagnosis; the model however is difficult to interpret for clinicians. To address this important shortcoming of neural network modeling methods, we describe several methods to help subject-matter audiences (e.g., clinicians, medical policy makers) understand neural network models. Garson's algorithm describes the relative magnitude of the importance of a descriptor (predictor) in its connection with outcome variables by dissecting the model weights. The Lek's profile method explores the relationship of the outcome variable and a predictor of interest, while holding other predictors at constant values (e.g., minimum, 20th quartile, maximum). While Lek's profile was developed specifically for neural networks, partial dependence plot is a more generic version that visualize the relationship between an outcome and one or two predictors. Finally, the local interpretable model-agnostic explanations (LIME) method can show the predictions of any classification or regression, by approximating it locally with an interpretable model. R code for the implementations of these methods is shown by using example data fitted with a standard, feed-forward neural network model. We offer codes and step-by-step description on how to use these tools to facilitate better understanding of ANN.

Current monitoring and innovative predictive modeling to improve care in the pediatric cardiac intensive care unit

The objectives of this review are (I) to describe the challenges associated with monitoring patients in the pediatric cardiac intensive care unit (PCICU) and (II) to discuss the use of innovative statistical and artificial intelligence (AI) software programs to attempt to predict significant clinical events. Patients cared for in the PCICU are clinically fragile and at risk for fatal decompensation. Current monitoring modalities are often ineffective, sometimes inaccurate, and fail to detect a deteriorating clinical status in a timely manner. Predictive models created by AI and machine learning may lead to earlier detection of patients at risk for clinical decompensation and thereby improve care for critically ill pediatric cardiac patients.