Machine Learning-Based Prediction of Clinical Outcomes for Children During Emergency Department Triage

Introduction to supervised machine learning in clinical epidemiology

Machine learning refers to a series of processes in which a computer finds rules from a vast amount of data. With recent advances in computer technology and the availability of a wide variety of health data, machine learning has rapidly developed and been applied in medical research. Currently, there are three types of machine learning: supervised, unsupervised, and reinforcement learning. In medical research, supervised learning is commonly used for diagnoses and prognoses, while unsupervised learning is used for phenotyping a disease, and reinforcement learning for maximizing favorable results, such as optimization of total patients' waiting time in the emergency department. The present article focuses on the concept and application of supervised learning in medicine, the most commonly used machine learning approach in medicine, and provides a brief explanation of four algorithms widely used for prediction (random forests, gradient-boosted decision tree, support vector machine, and neural network). Among these algorithms, the neural network has further developed into deep learning algorithms to solve more complex tasks. Along with simple classification problems, deep learning is commonly used to process medical imaging, such as retinal fundus photographs for diabetic retinopathy diagnosis. Although machine learning can bring new insights into medicine by processing a vast amount of data that are often beyond human capacity, algorithms can also fail when domain knowledge is neglected. The combination of algorithms and human cognitive ability is a key to the successful application of machine learning in medicine.

Application of Machine Learning Approaches to Predict Postnatal Growth Failure in Very Low Birth Weight Infants

PURPOSE

The aims of the study were to develop and evaluate a machine learning model with which to predict postnatal growth failure (PGF) among very low birth weight (VLBW) infants.

MATERIALS AND METHODS

Of 10425 VLBW infants registered in the Korean Neonatal Network between 2013 and 2017, 7954 infants were included. PGF was defined as a decrease in Z score >1.28 at discharge, compared to that at birth. Six metrics [area under the receiver operating characteristic curve (AUROC), accuracy, precision, sensitivity, specificity, and F1 score] were obtained at five time points (at birth, 7 days, 14 days, 28 days after birth, and at discharge). Machine learning models were built using four different techniques [extreme gradient boosting (XGB), random forest, support vector machine, and convolutional neural network] to compare against the conventional multiple logistic regression (MLR) model.

RESULTS

The XGB algorithm showed the best performance with all six metrics across the board. When compared with MLR, XGB showed a significantly higher AUROC (p=0.03) for Day 7, which was the primary performance metric. Using optimal cut-off points, for Day 7, XGB still showed better performances in terms of AUROC (0.74), accuracy (0.68), and F1 score (0.67). AUROC values seemed to increase slightly from birth to 7 days after birth with significance, almost reaching a plateau after 7 days after birth.

CONCLUSION

We have shown the possibility of predicting PGF through machine learning algorithms, especially XGB. Such models may help neonatologists in the early diagnosis of high-risk infants for PGF for early intervention.

Machine learning-based suggestion for critical interventions in the management of potentially severe conditioned patients in emergency department triage

Providing timely intervention to critically ill patients is a challenging task in emergency departments (ED). Our study aimed to predict early critical interventions (CrIs), which can be used as clinical recommendations. This retrospective observational study was conducted in the ED of a tertiary hospital located in a Korean metropolitan city. Patient who visited ED from January 1, 2016, to December 31, 2018, were included. Need of six CrIs were selected as prediction outcomes, namely, arterial line (A-line) insertion, oxygen therapy, high-flow nasal cannula (HFNC), intubation, Massive Transfusion Protocol (MTP), and inotropes and vasopressor. Extreme gradient boosting (XGBoost) prediction model was built by using only data available at the initial stage of ED. Overall, 137,883 patients were included in the study. The areas under the receiver operating characteristic curve for the prediction of A-line insertion was 0·913, oxygen therapy was 0.909, HFNC was 0.962, intubation was 0.945, MTP was 0.920, and inotropes or vasopressor administration was 0.899 in the XGBoost method. In addition, an increase in the need for CrIs was associated with worse ED outcomes. The CrIs model was integrated into the study site's electronic medical record and could be used to suggest early interventions for emergency physicians.

Validation of a Classification Model Using Complete Blood Count to Predict Severe Human Adenovirus Lower Respiratory Tract Infections in Pediatric Cases

Background

Human adenovirus (HAdV) lower respiratory tract infections (LRTIs) are prone to severe cases and even cause death in children. Here, we aimed to develop a classification model to predict severity in pediatric patients with HAdV LRTIs using complete blood count (CBC).

Methods

The CBC parameters from pediatric patients with a diagnosis of HAdV LRTIs from 2013 to 2019 were collected during the disease's course. The data were analyzed as potential predictors for severe cases and were selected using a random forest model.

Results

We enrolled 1,652 CBC specimens from 1,069 pediatric patients with HAdV LRTIs in the present study. Four hundred and seventy-four patients from 2017 to 2019 were used as the discovery cohort, and 470 patients from 2013 to 2016 were used as the validation cohort. The monocyte ratio (MONO%) was the most obvious difference between the mild and severe groups at onset, and could be used as a marker for the early accurate prediction of the severity [area under the subject operating characteristic curve (AUROC): 0.843]. Four risk factors [MONO%, hematocrit (HCT), red blood cell count (RBC), and platelet count (PLT)] were derived to construct a classification model of severe and mild cases using a random forest model (AUROC: 0.931 vs. 0.903).

Conclusion

Monocyte ratio can be used as an individual predictor of severe cases in the early stages of HAdV LRTIs. The four risk factors model is a simple and accurate risk assessment tool that can predict severe cases in the early stages of HAdV LRTIs.

Prehospital Factors Predict Outcomes in Pediatric Trauma: A Principal Component Analysis

BACKGROUND

Trauma team activation leveling decisions are complex and based on many variables. Accurate triage decisions improve patient safety and resource utilization. Our purpose was to establish proof-of-concept for using principal component analysis (PCA) to identify multivariate predictors of injury severity and to assess their ability to predict outcomes in pediatric trauma patients. We hypothesized that we could identify significant principal components (PCs) among variables used for decisions regarding trauma team activation and that PC scores would be predictive of outcomes in pediatric trauma.

METHODS

We conducted a retrospective review of the trauma registry (1/2014-12/2020) at our pediatric trauma center, including all pediatric patients (age < 18 y) who triggered a trauma team activation. Data included patient demographics, prehospital report, Injury Severity Score, and outcomes. Four significant principal components were identified using PCA. Differences in outcome variables between the highest and lowest quartile for PC score were examined.

RESULTS

1090 pediatric patients were included. The 4 significant PCs accounted for >96% of the overall date variance. The first PC was a composite of prehospital Glasgow Coma Scale and Revised Trauma Score (RTS) and was predictive of outcomes, including injury severity, length of stay, and mortality. The second PC was characterized primarily by prehospital systolic blood pressure (SBP) and high PC scores were associated with increased length of stay. The third and fourth PCs were characterized by patient age and by prehospital RTS and SBP, respectively.

CONCLUSIONS

We demonstrate that, using information available at the time of trauma team activation, PCA can be used to identify key predictors of patient outcome. While the ultimate goal is to create a machine learning-based predictive tool to support and improve clinical decision making, this study serves as a crucial step toward developing a deep understanding of the features of the model and their behavior with actual clinical data.

LEVEL OF EVIDENCE

III; Diagnostic test.

The Use of Biomarkers in the Early Diagnosis of Septic Arthritis and Osteomyelitis-A Pilot Study

BACKGROUND

The diagnosis of septic arthritis (SA) and osteomyelitis (OM) has remained challenging in the pediatric population, often accompanied by delays and requiring invasive interventions. The purpose of this pilot study is to identify a novel panel of biomarkers and cytokines that can accurately differentiate SA and OM at initial presentation using serum alone.

METHODS

Twenty patients below 18 years old whose working diagnosis included SA (n=10) and OM (n=10) were identified. Serum was collected at initial evaluation. Each sample underwent seven ELISA [C1-C2, COMP, CS-846, hyaluronan, procalcitonin, PIIANP, C-terminal telopeptide of type II collagen (CTX-II)] and 65-plex cytokine panels. Principal component and Lasso regression analysis were performed to identify a limited set of predictive biomarkers.

RESULTS

Mean age was 4.7 and 9.5 years in SA and OM patients, respectively (P=0.029). 50% of SA patients presented within 24 hours of symptom onset, compared with 0% of OM patients (P=0.033). 30% of SA patients were discharged home with an incorrect diagnosis and re-presented to the emergency department days later. At time of presentation: temperature ≥38.5°C was present in 10% of SA and 40% of OM patients (P=0.12), mean erythrocyte sedimentation rate (mm/h) was 51.6 in SA and 44.9 in OM patients (P=0.63), mean C-reactive protein (mg/dL) was 55.8 in SA and 71.8 in OM patients (P=0.53), and mean white blood cells (K/mm3) was 12.5 in SA and 10.4 in OM patients (P=0.34). 90% of SA patients presented with ≤2 of the Kocher criteria. 100% of SA and 40% of OM patients underwent surgery. 70% of SA cultures were culture negative, 10% MSSA, 10% Kingella, and 10% Strep pyogenes. 40% of OM cultures were culture negative, 50% MSSA, and 10% MRSA. Four biomarkers [CTx-II, transforming growth factor alpha (TGF-α), monocyte chemoattractant protein 1 (MCP-1), B cell-attracting chemokine 1] were identified that were able to classify and differentiate 18 of the 20 SA and OM cases correctly, with 90% sensitivity and 80% specificity.

CONCLUSIONS

This pilot study identifies a panel of biomarkers that can differentiate between SA and OM at initial presentation using serum alone.

LEVEL OF EVIDENCE

Level II-diagnostic study.

Implementation of prognostic machine learning algorithms in paediatric chronic respiratory conditions: a scoping review

Machine learning (ML) holds great potential for predicting clinical outcomes in heterogeneous chronic respiratory diseases (CRD) affecting children, where timely individualised treatments offer opportunities for health optimisation. This paper identifies rate-limiting steps in ML prediction model development that impair clinical translation and discusses regulatory, clinical and ethical considerations for ML implementation. A scoping review of ML prediction models in paediatric CRDs was undertaken using the PRISMA extension scoping review guidelines. From 1209 results, 25 articles published between 2013 and 2021 were evaluated for features of a good clinical prediction model using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines.

Most of the studies were in asthma (80%), with few in cystic fibrosis (12%), bronchiolitis (4%) and childhood wheeze (4%). There were inconsistencies in model reporting and studies were limited by a lack of validation, and absence of equations or code for replication. Clinician involvement during ML model development is essential and diversity, equity and inclusion should be assessed at each step of the ML pipeline to ensure algorithms do not promote or amplify health disparities among marginalised groups. As ML prediction studies become more frequent, it is important that models are rigorously developed using published guidelines and take account of regulatory frameworks which depend on model complexity, patient safety, accountability and liability.

A Framework for Using Real-World Data and Health Outcomes Modeling to Evaluate Machine Learning-Based Risk Prediction Models

OBJECTIVES

We propose a framework of health outcomes modeling with dynamic decision making and real-world data (RWD) to evaluate the potential utility of novel risk prediction models in clinical practice. Lung transplant (LTx) referral decisions in cystic fibrosis offer a complex case study.

METHODS

We used longitudinal RWD for a cohort of adults (n = 4247) from the Cystic Fibrosis Foundation Patient Registry to compare outcomes of an LTx referral policy based on machine learning (ML) mortality risk predictions to referral based on (1) forced expiratory volume in 1 second (FEV₁) alone and (2) heterogenous usual care (UC). We then developed a patient-level simulation model to project number of patients referred for LTx and 5-year survival, accounting for transplant availability, organ allocation policy, and heterogenous treatment effects.

RESULTS

Only 12% of patients (95% confidence interval 11%-13%) were referred for LTx over 5 years under UC, compared with 19% (18%-20%) under FEV₁ and 20% (19%-22%) under ML. Of 309 patients who died before LTx referral under UC, 31% (27%-36%) would have been referred under FEV₁ and 40% (35%-45%) would have been referred under ML. Given a fixed supply of organs, differences in referral time did not lead to significant differences in transplants, pretransplant or post-transplant deaths, or overall survival in 5 years.

CONCLUSIONS

Health outcomes modeling with RWD may help to identify novel ML risk prediction models with high potential real-world clinical utility and rule out further investment in models that are unlikely to offer meaningful real-world benefits.

Future of machine learning in paediatrics

Machine learning (ML) is a branch of artificial intelligence (AI) that enables computers to learn without being explicitly programmed, through a combination of statistics and computer science. It encompasses a variety of techniques used to analyse and interpret extremely large amounts of data, which can then be applied to create predictive models. Such applications of this technology are now ubiquitous in our day-to-day lives: predictive text, spam filtering, and recommendation systems in social media, streaming video and e-commerce to name a few examples. It is only more recently that ML has started to be implemented against the vast amount of data generated in healthcare. The emerging role of AI in refining healthcare delivery was recently highlighted in the 'National Health Service Long Term Plan 2019'. In paediatrics, workforce challenges, rising healthcare attendance and increased patient complexity and comorbidity mean that demands on paediatric services are also growing. As healthcare moves into this digital age, this review considers the potential impact ML can have across all aspects of paediatric care from improving workforce efficiency and aiding clinical decision-making to precision medicine and drug development.

Development and Validation of Simple Age-Adjusted Objectified Korean Triage and Acuity Scale for Adult Patients Visiting the Emergency Department

PURPOSE

The study aimed to develop an objectified Korean Triage and Acuity Scale (OTAS) that can objectively and quickly classify severity, as well as a simple age-adjusted OTAS (S-OTAS) that reflects age and evaluate its usefulness.

MATERIALS AND METHODS

A retrospective analysis was performed of all adult patients who had visited the emergency department at three teaching hospitals. Sex, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, O₂ saturation, and consciousness level were collected from medical records. The OTAS was developed with objective criterion and minimal OTAS level, and S-OTAS was developed by adding the age variable. For usefulness evaluation, the 30-day mortality, the rates of computed tomography scan and emergency procedures were compared between Korean Triage and Acuity Scale (KTAS) and OTAS.

RESULTS

A total of 44402 patients were analyzed. For 30-day mortality, S-OTAS showed a higher area under the curve (AUC) compared to KTAS (0.751 vs. 0.812 for KTAS and S-OTAS, respectively, p<0.001). Regarding the rates of emergency procedures, AUC was significantly higher in S-OTAS, compared to KTAS (0.807 vs. 0.830, for KTAS and S-OTAS, respectively, p=0.013).

CONCLUSION

S-OTAS showed comparative usefulness for adult patients visiting the emergency department as a triage tool compared to KTAS.

Machine learning-based prediction of critical illness in children visiting the emergency department

OBJECTIVES

Triage is an essential emergency department (ED) process designed to provide timely management depending on acuity and severity; however, the process may be inconsistent with clinical and hospitalization outcomes. Therefore, studies have attempted to augment this process with machine learning models, showing advantages in predicting critical conditions and hospitalization outcomes. The aim of this study was to utilize nationwide registry data to develop a machine learning-based classification model to predict the clinical course of pediatric ED visits.

METHODS

This cross-sectional observational study used data from the National Emergency Department Information System on emergency visits of children under 15 years of age from January 1, 2016, to December 31, 2017. The primary and secondary outcomes were to identify critically ill children and predict hospitalization from triage data, respectively. We developed and tested a random forest model with the under sampled dataset and validated the model using the entire dataset. We compared the model's performance with that of the conventional triage system.

RESULTS

A total of 2,621,710 children were eligible for the analysis and included 12,951 (0.5%) critical outcomes and 303,808 (11.6%) hospitalizations. After validation, the area under the receiver operating characteristic curve was 0.991 (95% confidence interval [CI] 0.991-0.992) for critical outcomes and 0.943 (95% CI 0.943-0.944) for hospitalization, which were higher than those of the conventional triage system.

CONCLUSIONS

The machine learning-based model using structured triage data from a nationwide database can effectively predict critical illness and hospitalizations among children visiting the ED.

Evaluation of the Need for Intensive Care in Children With Pneumonia: Machine Learning Approach

Background

Timely decision-making regarding intensive care unit (ICU) admission for children with pneumonia is crucial for a better prognosis. Despite attempts to establish a guideline or triage system for evaluating ICU care needs, no clinically applicable paradigm is available.

Objective

The aim of this study was to develop machine learning (ML) algorithms to predict ICU care needs for pediatric pneumonia patients within 24 hours of admission, evaluate their performance, and identify clinical indices for making decisions for pediatric pneumonia patients.

Methods

Pneumonia patients admitted to National Taiwan University Hospital from January 2010 to December 2019 aged under 18 years were enrolled. Their underlying diseases, clinical manifestations, and laboratory data at admission were collected. The outcome of interest was ICU transfer within 24 hours of hospitalization. We compared clinically relevant features between early ICU transfer patients and patients without ICU care. ML algorithms were developed to predict ICU admission. The performance of the algorithms was evaluated using sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and average precision. The relative feature importance of the best-performing algorithm was compared with physician-rated feature importance for explainability.

Results

A total of 8464 pediatric hospitalizations due to pneumonia were recorded, and 1166 (1166/8464, 13.8%) hospitalized patients were transferred to the ICU within 24 hours. Early ICU transfer patients were younger (P<.001), had higher rates of underlying diseases (eg, cardiovascular, neuropsychological, and congenital anomaly/genetic disorders; P<.001), had abnormal laboratory data, had higher pulse rates (P<.001), had higher breath rates (P<.001), had lower oxygen saturation (P<.001), and had lower peak body temperature (P<.001) at admission than patients without ICU transfer. The random forest (RF) algorithm achieved the best performance (sensitivity 0.94, 95% CI 0.92-0.95; specificity 0.94, 95% CI 0.92-0.95; AUC 0.99, 95% CI 0.98-0.99; and average precision 0.93, 95% CI 0.90-0.96). The lowest systolic blood pressure and presence of cardiovascular and neuropsychological diseases ranked in the top 10 in both RF relative feature importance and clinician judgment.

Conclusions

The ML approach could provide a clinically applicable triage algorithm and identify important clinical indices, such as age, underlying diseases, abnormal vital signs, and laboratory data for evaluating the need for intensive care in children with pneumonia.

Machine Learning Approaches for Predicting Difficult Airway and First-Pass Success in the Emergency Department: Multicenter Prospective Observational Study

Background

There is still room for improvement in the modified LEMON (look, evaluate, Mallampati, obstruction, neck mobility) criteria for difficult airway prediction and no prediction tool for first-pass success in the emergency department (ED).

Objective

We applied modern machine learning approaches to predict difficult airways and first-pass success.

Methods

In a multicenter prospective study that enrolled consecutive patients who underwent tracheal intubation in 13 EDs, we developed 7 machine learning models (eg, random forest model) using routinely collected data (eg, demographics, initial airway assessment). The outcomes were difficult airway and first-pass success. Model performance was evaluated using c-statistics, calibration slopes, and association measures (eg, sensitivity) in the test set (randomly selected 20% of the data). Their performance was compared with the modified LEMON criteria for difficult airway success and a logistic regression model for first-pass success.

Results

Of 10,741 patients who underwent intubation, 543 patients (5.1%) had a difficult airway, and 7690 patients (71.6%) had first-pass success. In predicting a difficult airway, machine learning models—except for k-point nearest neighbor and multilayer perceptron—had higher discrimination ability than the modified LEMON criteria (all, P≤.001). For example, the ensemble method had the highest c-statistic (0.74 vs 0.62 with the modified LEMON criteria; P<.001). Machine learning models—except k-point nearest neighbor and random forest models—had higher discrimination ability for first-pass success. In particular, the ensemble model had the highest c-statistic (0.81 vs 0.76 with the reference regression; P<.001).

Conclusions

Machine learning models demonstrated greater ability for predicting difficult airway and first-pass success in the ED.

Prediction of acute appendicitis among patients with undifferentiated abdominal pain at emergency department

Background

Early screening and accurately identifying Acute Appendicitis (AA) among patients with undifferentiated symptoms associated with appendicitis during their emergency visit will improve patient safety and health care quality. The aim of the study was to compare models that predict AA among patients with undifferentiated symptoms at emergency visits using both structured data and free-text data from a national survey.

Methods

We performed a secondary data analysis on the 2005-2017 United States National Hospital Ambulatory Medical Care Survey (NHAMCS) data to estimate the association between emergency department (ED) patients with the diagnosis of AA, and the demographic and clinical factors present at ED visits during a patient’s ED stay. We used binary logistic regression (LR) and random forest (RF) models incorporating natural language processing (NLP) to predict AA diagnosis among patients with undifferentiated symptoms.

Results

Among the 40,441 ED patients with assigned International Classification of Diseases (ICD) codes of AA and appendicitis-related symptoms between 2005 and 2017, 655 adults (2.3%) and 256 children (2.2%) had AA. For the LR model identifying AA diagnosis among adult ED patients, the c-statistic was 0.72 (95% CI: 0.69–0.75) for structured variables only, 0.72 (95% CI: 0.69–0.75) for unstructured variables only, and 0.78 (95% CI: 0.76–0.80) when including both structured and unstructured variables. For the LR model identifying AA diagnosis among pediatric ED patients, the c-statistic was 0.84 (95% CI: 0.79–0.89) for including structured variables only, 0.78 (95% CI: 0.72–0.84) for unstructured variables, and 0.87 (95% CI: 0.83–0.91) when including both structured and unstructured variables. The RF method showed similar c-statistic to the corresponding LR model.

Conclusions

We developed predictive models that can predict the AA diagnosis for adult and pediatric ED patients, and the predictive accuracy was improved with the inclusion of NLP elements and approaches.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01490-9.

Development of artificial neural network models for paediatric critical illness in South Africa

OBJECTIVES

Failures in identification, resuscitation and appropriate referral have been identified as significant contributors to avoidable severity of illness and mortality in South African children. In this study, artificial neural network models were developed to predict a composite outcome of death before discharge from hospital or admission to the PICU. These models were compared to logistic regression and XGBoost models developed on the same data in cross-validation.

DESIGN

Prospective, analytical cohort study.

SETTING

A single centre tertiary hospital in South Africa providing acute paediatric services.

PATIENTS

Children, under the age of 13 years presenting to the Paediatric Referral Area for acute consultations.

OUTCOMES

Predictive models for a composite outcome of death before discharge from hospital or admission to the PICU.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

765 patients were included in the data set with 116 instances (15.2%) of the study outcome. Models were developed on three sets of features. Two derived from sequential floating feature selection (one inclusive, one parsimonious) and one from the Akaike information criterion to yield 9 models. All developed models demonstrated good discrimination on cross-validation with mean ROC AUCs greater than 0.8 and mean PRC AUCs greater than 0.53. ANN1, developed on the inclusive feature-et demonstrated the best discrimination with a ROC AUC of 0.84 and a PRC AUC of 0.64 Model calibration was variable, with most models demonstrating weak calibration. Decision curve analysis demonstrated that all models were superior to baseline strategies, with ANN1 demonstrating the highest net benefit.

CONCLUSIONS

All models demonstrated satisfactory performance, with the best performing model in cross-validation being an ANN model. Given the good performance of less complex models, however, these models should also be considered, given their advantage in ease of implementation in practice. An internal validation study is now being conducted to further assess performance with a view to external validation.

Machine learning methods applied to triage in emergency services: A systematic review

BACKGROUND

In emergency services is important to accurately assess and classify symptoms, which may be improved with the help of technology. One mechanism that could help and improve predictions from health records or patient flow is machine learning (ML).

AIM

To analyse the effectiveness of ML systems in triage for making predictions at the emergency department in comparison with other triage scales/scores.

METHODS

Following the PRISMA recommendations, a systematic review was conducted using CINAHL, Cochrane, Cuiden, Medline and Scopus databases with the search equation "Machine learning AND triage AND emergency".

RESULTS

Eleven studies were identified. The studies show that the use of ML methods consistently predict important outcomes like mortality, critical care outcomes and admission, and the need for hospitalization in comparison with scales like Emergency Severity Index or others. Among the ML models considered, XGBoost and Deep Neural Networks obtained the highest levels of prediction accuracy, while Logistic Regression performed obtained the worst values.

CONCLUSIONS

Machine learning methods can be a good instrument for helping triage process with the prediction of important emergency variables like mortality or the need for critical care or hospitalization.

Machine learning can identify patients at risk of hyperparathyroidism without known calcium and intact parathyroid hormone

BACKGROUND

To prove the concept of diagnosing primary hyperparathyroidism (pHPT) without calcium and parathyroid hormone (PTH) values and identifying potential risk factors for pHPT.

METHODS

Data were extracted from the clinical data warehouse (CDW) at the University of Arkansas for Medical Sciences (UAMS) Epic EHR (2014-2019).

RESULTS

1737 patients with over 185 000 rows of clinical data were provided in a relational structure and processed/flattened to facilitate modeling. Phenotype elements were identified for pHPT without advance knowledge of calcium and PTH levels. The area under the curve (AUC) for the prediction of pHPT using our model was 0.86 with sensitivity and specificity of 0.8953 and 0.6686, respectively, using a 0.45 probability threshold.

CONCLUSION

Primary hyperparathyroidism was predicted from a dataset excluding calcium and PTH data with 86% accuracy. This approach needs to be validated/refined on larger samples of data and plans are in place to do this with other regional/national datasets.

Development and validation of a practical machine-learning triage algorithm for the detection of patients in need of critical care in the emergency department

Identifying critically ill patients is a key challenge in emergency department (ED) triage. Mis-triage errors are still widespread in triage systems around the world. Here, we present a machine learning system (MLS) to assist ED triage officers better recognize critically ill patients and provide a text-based explanation of the MLS recommendation. To derive the MLS, an existing dataset of 22,272 patient encounters from 2012 to 2019 from our institution’s electronic emergency triage system (EETS) was used for algorithm training and validation. The area under the receiver operating characteristic curve (AUC) was 0.875 ± 0.006 (CI:95%) in retrospective dataset using fivefold cross validation, higher than that of reference model (0.843 ± 0.005 (CI:95%)). In the prospective cohort study, compared to the traditional triage system’s 1.2% mis-triage rate, the mis-triage rate in the MLS-assisted group was 0.9%. This MLS method with a real-time explanation for triage officers was able to lower the mis-triage rate of critically ill ED patients.

Diagnosis of Suspected Scaphoid Fractures

»

Suspected scaphoid fractures are a diagnostic and therapeutic challenge despite the advances in knowledge regarding these injuries and imaging techniques. The risks and restrictions of routine immobilization as well as the restriction of activities in a young and active population must be weighed against the risks of nonunion that are associated with a missed fracture.

»

The prevalence of true fractures among suspected fractures is low. This greatly reduces the statistical probability that a positive diagnostic test will correspond with a true fracture, reducing the positive predictive value of an investigation.

»

There is no consensus reference standard for a true fracture; therefore, alternative statistical methods for calculating sensitivity, specificity, and positive and negative predictive values are required.

»

Clinical prediction rules that incorporate a set of demographic and clinical factors may allow stratification of secondary imaging, which, in turn, could increase the pretest probability of a scaphoid fracture and improve the diagnostic performance of the sophisticated radiographic investigations that are available.

»

Machine-learning-derived probability calculators may augment risk stratification and can improve through retraining, although these theoretical benefits need further prospective evaluation.

»

Convolutional neural networks (CNNs) are a form of artificial intelligence that have demonstrated great promise in the recognition of scaphoid fractures on radiographs. However, in the more challenging diagnostic scenario of a suspected or so-called "clinical" scaphoid fracture, CNNs have not yet proven superior to a diagnosis that has been made by an experienced surgeon.

Validating machine learning models for the prediction of labour induction intervention using routine data: a registry-based retrospective cohort study at a tertiary hospital in northern Tanzania

OBJECTIVES

We aimed at identifying the important variables for labour induction intervention and assessing the predictive performance of machine learning algorithms.

SETTING

We analysed the birth registry data from a referral hospital in northern Tanzania. Since July 2000, every birth at this facility has been recorded in a specific database.

PARTICIPANTS

21 578 deliveries between 2000 and 2015 were included. Deliveries that lacked information regarding the labour induction status were excluded.

PRIMARY OUTCOME

Deliveries involving labour induction intervention.

RESULTS

Parity, maternal age, body mass index, gestational age and birth weight were all found to be important predictors of labour induction. Boosting method demonstrated the best discriminative performance (area under curve, AUC=0.75: 95% CI (0.73 to 0.76)) while logistic regression presented the least (AUC=0.71: 95% CI (0.70 to 0.73)). Random forest and boosting algorithms showed the highest net-benefits as per the decision curve analysis.

CONCLUSION

All of the machine learning algorithms performed well in predicting the likelihood of labour induction intervention. Further optimisation of these classifiers through hyperparameter tuning may result in an improved performance. Extensive research into the performance of other classifier algorithms is warranted.

Modification and Validation of a Complaint-Oriented Emergency Department Triage System: A Multicenter Observational Study

PURPOSE

The objective of this study was to modify and validate an emergency department (ED) triage system with improved prediction performance on hospital outcomes by modifying the Korean Triage and Acuity Scale (KTAS).

MATERIALS AND METHODS

We performed a retrospective observational study at three academic universities in South Korea. The KTAS code, determined by the chief complaint and the selected modifier of a patient, was used to derive the Modified KTAS (MKTAS). We calculated the area under the receiver operating characteristics curve (AUC) and the test characteristics to evaluate the performance of MKTAS to predict hospital mortality, critical outcome, and admission.

RESULTS

A total of 272402 and 128831 ED visits were used for the derivation and validation of MKTAS, respectively. Compared to KTAS, MKTAS had significantly higher AUC values for the prediction of hospital mortality [MKTAS 0.826 (0.818-0.835) vs. KTAS 0.794 (0.784-0.803)], critical outcome [MKTAS 0.836 (0.830-0.841) vs. 0.798 (0.792-0.804)], and admission [MKTAS 0.725 (0.723-0.728) vs. KTAS 0.685 (0.682-0.688)]. The sensitivity for predicting hospital mortality and critical outcome, as well as the specificity for predicting admission, were significantly improved.

CONCLUSION

MKTAS was derived by modifying the KTAS, and then validated. Compared with KTAS, MKTAS showed better discriminating ability to predict hospital outcomes. Continuous efforts to evaluate and modify widely used triage systems are required to improve their performance.

Understanding, explaining, and utilizing medical artificial intelligence

Medical artificial intelligence is cost-effective and scalable and often outperforms human providers, yet people are reluctant to use it. We show that resistance to the utilization of medical artificial intelligence is driven by both the subjective difficulty of understanding algorithms (the perception that they are a 'black box') and by an illusory subjective understanding of human medical decision-making. In five pre-registered experiments (1-3B: N = 2,699), we find that people exhibit an illusory understanding of human medical decision-making (study 1). This leads people to believe they better understand decisions made by human than algorithmic healthcare providers (studies 2A,B), which makes them more reluctant to utilize algorithmic than human providers (studies 3A,B). Fortunately, brief interventions that increase subjective understanding of algorithmic decision processes increase willingness to utilize algorithmic healthcare providers (studies 3A,B). A sixth study on Google Ads for an algorithmic skin cancer detection app finds that the effectiveness of such interventions generalizes to field settings (study 4: N = 14,013).

The Safety INdEx of Prehospital On Scene Triage (SINEPOST) study: the development and validation of a risk prediction model to support ambulance clinical transport decisions on-scene-a protocol

BACKGROUND

Demand for both the ambulance service and the emergency department (ED) is rising every year and when this demand is excessive in both systems, ambulance crews queue at the ED waiting to hand patients over. Some transported ambulance patients are 'low-acuity' and do not require the treatment of the ED. However, paramedics can find it challenging to identify these patients accurately. Decision support tools have been developed using expert opinion to help identify these low acuity patients but have failed to show a benefit beyond regular decision-making. Predictive algorithms may be able to build accurate models, which can be used in the field to support the decision not to take a low-acuity patient to an ED.

METHODS AND ANALYSIS

All patients in Yorkshire who were transported to the ED by ambulance between July 2019 and February 2020 will be included. Ambulance electronic patient care record (ePCR) clinical data will be used as candidate predictors for the model. These will then be linked to the corresponding ED record, which holds the outcome of a 'non-urgent attendance'. The estimated sample size is 52,958, with 4767 events and an EPP of 7.48. An XGBoost algorithm will be used for model development. Initially, a model will be derived using all the data and the apparent performance will be assessed. Then internal-external validation will use non-random nested cross-validation (CV) with test sets held out for each ED (spatial validation). After all models are created, a random-effects meta-analysis will be undertaken. This will pool performance measures such as goodness of fit, discrimination and calibration. It will also generate a prediction interval and measure heterogeneity between clusters. The performance of the full model will be updated with the pooled results.

DISCUSSION

Creating a risk prediction model in this area will lead to further development of a clinical decision support tool that ensures every ambulance patient can get to the right place of care, first time. If this study is successful, it could help paramedics evaluate the benefit of transporting a patient to the ED before they leave the scene. It could also reduce congestion in the urgent and emergency care system.

TRIAL REGISTRATION

This study was retrospectively registered with the ISRCTN: 12121281.

Novelelectronic health records applied for prediction of pre-eclampsia: Machine-learning algorithms

OBJECTIVE

To predict risk of pre-eclampsia (PE) in women using machine learning (ML) algorithms, based on electronic health records (EHR) collected at the early second trimester.

STUDY DESIGN

A total of 3759 cases of pregnancy who received antenatal care at Xinhua hospital Chongming branch Affiliated to Shanghai Jiaotong University were included in this retrospective EHR-based study. Thirty-eight candidate clinical parameters routinely available at the first visit in antenatal care were collected by manual chart review. Logistic regression (LR), random forest (RF), support vector machine (SVM) and extreme gradient boosting (XGBoost) were used to construct the prediction model. Features that contributed to the model predictions were identified using XGBoost.

OUTCOME MEASURES

The performance of ML models to predict women at risk of PE was quantified in terms of accuracy, precision, recall, false negative score, f1_score, brier score and the area under the receiver operating curve (auROC).

RESULTS

The XGboost model had the best prediction performance (accuracy = 0.920, precision = 0.447, recall = 0.789, f1_score = 0.571, auROC = 0.955). The most predictive feature of PE development was fasting plasma glucose, followed by mean blood pressure and body mass index. An easy-to-use model that a patient could answer independently still enabled accurate prediction, with auROC of 0.83.

CONCLUSION

risk of PE development can be predicted with excellent discriminative ability using ML algorithms based on EHR collected at the early second trimester. Future studies are needed to assess the real-world clinical utility of the model.

A machine learning model to predict critical care outcomes in patient with chest pain visiting the emergency department

Background

Currently, the risk stratification of critically ill patient with chest pain is a challenge. We aimed to use machine learning approach to predict the critical care outcomes in patients with chest pain, and simultaneously compare its performance with HEART, GRACE, and TIMI scores.

Methods

This was a retrospective, case-control study in patients with acute non-traumatic chest pain who presented to the emergency department (ED) between January 2017 and December 2019. The outcomes included cardiac arrest, transfer to ICU, and death during treatment in ED. In the randomly sampled training set (70%), a LASSO regression model was developed, and presented with nomogram. The performance was measured in both training set (70% participants) and testing set (30% participants), and findings were compared with the three widely used scores.

Results

We proposed a LASSO regression model incorporating mode of arrival, reperfusion therapy, Killip class, systolic BP, serum creatinine, creatine kinase-MB, and brain natriuretic peptide as independent predictors of critical care outcomes in patients with chest pain. Our model significantly outperformed the HEART, GRACE, TIMI score with AUC of 0.953 (95%CI: 0.922–0.984), 0.754 (95%CI: 0.675–0.832), 0.747 (95%CI: 0.664–0.829), 0.735 (95%CI: 0.655–0.815), respectively. Consistently, our model demonstrated better outcomes regarding the metrics of accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and F1 score. Similarly, the decision curve analysis elucidated a greater net benefit of our model over the full ranges of clinical thresholds.

Conclusion

We present an accurate model for predicting the critical care outcomes in patients with chest pain, and provide substantial support to its application as a decision-making tool in ED.

Machine learning for outcome predictions of patients with trauma during emergency department care

OBJECTIVES

To develop and evaluate a machine learning model for predicting patient with trauma mortality within the US emergency departments.

METHODS

This was a retrospective prognostic study using deidentified patient visit data from years 2007 to 2014 of the National Trauma Data Bank. The predictive model intelligence building process is designed based on patient demographics, vital signs, comorbid conditions, arrival mode and hospital transfer status. The mortality prediction model was evaluated on its sensitivity, specificity, area under receiver operating curve (AUC), positive and negative predictive value, and Matthews correlation coefficient.

RESULTS

Our final dataset consisted of 2 007 485 patient visits (36.45% female, mean age of 45), 8198 (0.4%) of which resulted in mortality. Our model achieved AUC and sensitivity-specificity gap of 0.86 (95% CI 0.85 to 0.87), 0.44 for children and 0.85 (95% CI 0.85 to 0.85), 0.44 for adults. The all ages model characteristics indicate it generalised, with an AUC and gap of 0.85 (95% CI 0.85 to 0.85), 0.45. Excluding fall injuries weakened the child model (AUC 0.85, 95% CI 0.84 to 0.86) but strengthened adult (AUC 0.87, 95% CI 0.87 to 0.87) and all ages (AUC 0.86, 95% CI 0.86 to 0.86) models.

CONCLUSIONS

Our machine learning model demonstrates similar performance to contemporary machine learning models without requiring restrictive criteria or extensive medical expertise. These results suggest that machine learning models for trauma outcome prediction can generalise to patients with trauma across the USA and may be able to provide decision support to medical providers in any healthcare setting.

Can the adverse childhood experiences (ACEs) checklist be utilized to predict emergency department visits among children and adolescents?

Background

Extensive literature has shown an association of Adverse Childhood Experiences (ACEs) with adverse health outcomes; however, its ability to predict events or stratify risks is less known. Individuals with mental illness and ACE exposure have been shown to visit emergency departments (ED) more often than those in the general population. This study thus examined the ability of the ACEs checklist to predict ED visits within the subsequent year among children and adolescents presenting to mental health clinics with pre-existing mental health issues.

Methods

The study analyzed linked data (n = 6100) from two databases provided by Alberta Health Services (AHS). The Regional Access and Intake System (RAIS 2016–2018) database provided data on the predictors (ACE items, age, sex, residence, mental health program type, and primary diagnosis) regarding children and adolescents (aged 0–17 years) accessing addiction and mental health services within Calgary Zone, and the National Ambulatory Care Reporting System (NACRS 2016–2019) database provided data on ED visits. A 25% random sample of the data was reserved for validation purposes. Two Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression models, each employing a different method to tune the shrinkage parameter lambda (namely cross-validated and adaptive) and performing 10-fold cross-validation for a set of 100 lambdas in each model were examined.

Results

The adaptive LASSO model had a slightly better fit in the validation dataset than the cross-validated model; however, it still demonstrated poor discrimination (AUC 0.60, sensitivity 37.8%, PPV 49.6%) and poor calibration (over-triaged in low-risk and under-triaged in high-risk subgroups). The model’s poor performance was evident from an out-of-sample deviance ratio of − 0.044.

Conclusion

The ACEs checklist did not perform well in predicting ED visits among children and adolescents with existing mental health concerns. The diverse causes of ED visits may have hindered accurate predictions, requiring more advanced statistical procedures. Future studies exploring other machine learning approaches and including a more extensive set of childhood adversities and other important predictors may produce better predictions. Furthermore, despite highly significant associations being observed, ACEs may not be deterministic in predicting health-related events at the individual level, such as general ED use.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01392-w.

Could a Multi-Marker and Machine Learning Approach Help Stratify Patients with Heart Failure?

Half of the patients with heart failure (HF) have preserved ejection fraction (HFpEF). To date, there are no specific markers to distinguish this subgroup. The main objective of this work was to stratify HF patients using current biochemical markers coupled with clinical data. The cohort study included HFpEF (n = 24) and heart failure with reduced ejection fraction (HFrEF) (n = 34) patients as usually considered in clinical practice based on cardiac imaging (EF ≥ 50% for HFpEF; EF < 50% for HFrEF). Routine blood tests consisted of measuring biomarkers of renal and heart functions, inflammation, and iron metabolism. A multi-test approach and analysis of peripheral blood samples aimed to establish a computerized Machine Learning strategy to provide a blood signature to distinguish HFpEF and HFrEF. Based on logistic regression, demographic characteristics and clinical biomarkers showed no statistical significance to differentiate the HFpEF and HFrEF patient subgroups. Hence a multivariate factorial discriminant analysis, performed blindly using the data set, allowed us to stratify the two HF groups. Consequently, a Machine Learning (ML) strategy was developed using the same variables in a genetic algorithm approach. ML provided very encouraging explorative results when considering the small size of the samples applied. The accuracy and the sensitivity were high for both validation and test groups (69% and 100%, 64% and 75%, respectively). Sensitivity was 100% for the validation and 75% for the test group, whereas specificity was 44% and 55% for the validation and test groups because of the small number of samples. Lastly, the precision was acceptable, with 58% in the validation and 60% in the test group. Combining biochemical and clinical markers is an excellent entry to develop a computer classification tool to diagnose HFpEF. This translational approach is a springboard for improving new personalized treatment methods and identifying “high-yield” populations for clinical trials.

New machine learning scoring system for predicting postoperative mortality in gastroduodenal ulcer perforation: A study using a Japanese nationwide inpatient database

BACKGROUND

Conventional prediction models for estimating risk of postoperative mortality in gastroduodenal ulcer perforation have suboptimal prediction ability. We aimed to develop and validate new machine learning models and an integer-based score for predicting the postoperative mortality.

METHODS

We retrospectively identified patients with gastroduodenal ulcer perforation who underwent surgical repair, using a nationwide Japanese inpatient database. In a derivation cohort from July 2010 to March 2016, we developed 2 machine learning-based models, Lasso and XGBoost, using 45 candidate predictors, and also developed an integer-based score for clinical use by including important variables in Lasso. In a validation cohort from April 2016 to March 2017, we measured the prediction performances of the models by computing area under the curve and comparing it to the conventional American Society of Anesthesiology risk score.

RESULTS

Of 25,886 patients, 1,176 (4.5%) died after surgical repair. For the validation cohort, Lasso and XGBoost had significantly higher prediction abilities than the American Society of Anesthesiology score (Lasso area under the curve = 0.84; 95% confidence interval 0.81-0.86; American Society of Anesthesiology score area under the curve = 0.70; 95% confidence interval 0.65-0.74, P < .001). The integer-based risk score, which had 13 factors, had a prediction ability similar to those of Lasso and XGBoost (area under the curve = 0.83; 95% confidence interval 0.81-0.86). According to the integer-based score, the mortalities were 0.1%, 2.3%, 9.3%, and 29.0% for the low (score, 0), moderate (1-2), high (3-4), and very high (≥5) score groups, respectively.

CONCLUSION

Machine learning models and the integer-based risk score performed well in predicting risk of postoperative mortality in gastroduodenal ulcer perforation. These models will help in decision making.

The Impact of Weather and Air Pollution on Viral Infection and Disease Outcome Among Pediatric Pneumonia Patients in Chongqing, China, from 2009 to 2018: A Prospective Observational Study

BACKGROUND

For pediatric pneumonia, the meteorological and air pollution indicators have been frequently investigated for their association with viral circulation but not for their impact on disease severity.

METHODS

We performed a 10-year prospective, observational study in 1 hospital in Chongqing, China, to recruit children with pneumonia. Eight commonly seen respiratory viruses were tested. Autoregressive distributed lag (ADL) and random forest (RF) models were used to fit monthly detection rates of each virus at the population level and to predict the possibility of severe pneumonia at the individual level, respectively.

RESULTS

Between 2009 and 2018, 6611 pediatric pneumonia patients were included, and 4846 (73.3%) tested positive for at least 1 respiratory virus. The patient median age was 9 months (interquartile range, 4‒20). ADL models demonstrated a decent fitting of detection rates of R2 > 0.7 for respiratory syncytial virus, human rhinovirus, parainfluenza virus, and human metapneumovirus. Based on the RF models, the area under the curve for host-related factors alone was 0.88 (95% confidence interval [CI], .87‒.89) and 0.86 (95% CI, .85‒.88) for meteorological and air pollution indicators alone and 0.62 (95% CI, .60‒.63) for viral infections alone. The final model indicated that 9 weather and air pollution indicators were important determinants of severe pneumonia, with a relative contribution of 62.53%, which is significantly higher than respiratory viral infections (7.36%).

CONCLUSIONS

Meteorological and air pollution predictors contributed more to severe pneumonia in children than did respiratory viruses. These meteorological data could help predict times when children would be at increased risk for severe pneumonia and when interventions, such as reducing outdoor activities, may be warranted.

Development and Assessment of an Interpretable Machine Learning Triage Tool for Estimating Mortality After Emergency Admissions

IMPORTANCE

Triage in the emergency department (ED) is a complex clinical judgment based on the tacit understanding of the patient's likelihood of survival, availability of medical resources, and local practices. Although a scoring tool could be valuable in risk stratification, currently available scores have demonstrated limitations.

OBJECTIVES

To develop an interpretable machine learning tool based on a parsimonious list of variables available at ED triage; provide a simple, early, and accurate estimate of patients' risk of death; and evaluate the tool's predictive accuracy compared with several established clinical scores.

DESIGN, SETTING, AND PARTICIPANTS

This single-site, retrospective cohort study assessed all ED patients between January 1, 2009, and December 31, 2016, who were subsequently admitted to a tertiary hospital in Singapore. The Score for Emergency Risk Prediction (SERP) tool was derived using a machine learning framework. To estimate mortality outcomes after emergency admissions, SERP was compared with several triage systems, including Patient Acuity Category Scale, Modified Early Warning Score, National Early Warning Score, Cardiac Arrest Risk Triage, Rapid Acute Physiology Score, and Rapid Emergency Medicine Score. The initial analyses were completed in October 2020, and additional analyses were conducted in May 2021.

MAIN OUTCOMES AND MEASURES

Three SERP scores, namely SERP-2d, SERP-7d, and SERP-30d, were developed using the primary outcomes of interest of 2-, 7-, and 30-day mortality, respectively. Secondary outcomes included 3-day mortality and inpatient mortality. The SERP's predictive power was measured using the area under the curve in the receiver operating characteristic analysis.

RESULTS

The study included 224 666 ED episodes in the model training cohort (mean [SD] patient age, 63.60 [16.90] years; 113 426 [50.5%] female), 56 167 episodes in the validation cohort (mean [SD] patient age, 63.58 [16.87] years; 28 427 [50.6%] female), and 42 676 episodes in the testing cohort (mean [SD] patient age, 64.85 [16.80] years; 21 556 [50.5%] female). The mortality rates in the training cohort were 0.8% at 2 days, 2.2% at 7 days, and 5.9% at 30 days. In the testing cohort, the areas under the curve of SERP-30d were 0.821 (95% CI, 0.796-0.847) for 2-day mortality, 0.826 (95% CI, 0.811-0.841) for 7-day mortality, and 0.823 (95% CI, 0.814-0.832) for 30-day mortality and outperformed several benchmark scores.

CONCLUSIONS AND RELEVANCE

In this retrospective cohort study, SERP had better prediction performance than existing triage scores while maintaining easy implementation and ease of ascertainment in the ED. It has the potential to be widely applied and validated in different circumstances and health care settings.

A Simulated Prospective Evaluation of a Deep Learning Model for Real-Time Prediction of Clinical Deterioration Among Ward Patients

OBJECTIVES

The National Early Warning Score, Modified Early Warning Score, and quick Sepsis-related Organ Failure Assessment can predict clinical deterioration. These scores exhibit only moderate performance and are often evaluated using aggregated measures over time. A simulated prospective validation strategy that assesses multiple predictions per patient-day would provide the best pragmatic evaluation. We developed a deep recurrent neural network deterioration model and conducted a simulated prospective evaluation.

DESIGN

Retrospective cohort study.

SETTING

Four hospitals in Pennsylvania.

PATIENTS

Inpatient adults discharged between July 1, 2017, and June 30, 2019.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We trained a deep recurrent neural network and logistic regression model using data from electronic health records to predict hourly the 24-hour composite outcome of transfer to ICU or death. We analyzed 146,446 hospitalizations with 16.75 million patient-hours. The hourly event rate was 1.6% (12,842 transfers or deaths, corresponding to 260,295 patient-hours within the predictive horizon). On a hold-out dataset, the deep recurrent neural network achieved an area under the precision-recall curve of 0.042 (95% CI, 0.04-0.043), comparable with logistic regression model (0.043; 95% CI 0.041 to 0.045), and outperformed National Early Warning Score (0.034; 95% CI, 0.032-0.035), Modified Early Warning Score (0.028; 95% CI, 0.027- 0.03), and quick Sepsis-related Organ Failure Assessment (0.021; 95% CI, 0.021-0.022). For a fixed sensitivity of 50%, the deep recurrent neural network achieved a positive predictive value of 3.4% (95% CI, 3.4-3.5) and outperformed logistic regression model (3.1%; 95% CI 3.1-3.2), National Early Warning Score (2.0%; 95% CI, 2.0-2.0), Modified Early Warning Score (1.5%; 95% CI, 1.5-1.5), and quick Sepsis-related Organ Failure Assessment (1.5%; 95% CI, 1.5-1.5).

CONCLUSIONS

Commonly used early warning scores for clinical decompensation, along with a logistic regression model and a deep recurrent neural network model, show very poor performance characteristics when assessed using a simulated prospective validation. None of these models may be suitable for real-time deployment.

Pediatric quality measures: The leap from process to outcomes

Value-based reimbursement arrangements tie financial incentives to achieving quality measures to ensure savings are not from withholding care. For patients and their families, the delivery of high-quality care is simply the expectation. Defining and measuring pediatric quality, however, is not standardized which has led to a large proliferation of metrics across multiple stakeholders. The majority of these measures are process rather than outcomes metrics often chosen for the ease at which the data can be obtained. In order to drive greater value, outcomes measures should be preferentially selected. However, measuring outcomes in children presents multiple unique challenges. Compared to adults, children are generally healthier, their outcomes may take more time to manifest, and their clinical variability is greater. Another challenge is the amount of healthcare data being generated by providers, provider networks, payors, government agencies, and many others. This should help in understanding pediatric quality outcomes, but the massive volume of data requires new analytic tools. Artificial intelligence techniques such as machine learning offer faster, more precise, and larger scale evaluation of quality outcomes. Its implementation necessitates identifying expertise in the way of data scientists as well as additional infrastructure components to evaluate data governance, security, regulatory compliance, and ethics. Despite these prerequisites, much progress is being made in outcome insights that drive value benefiting children and families.

Predictive Risk Models for Wound Infection-Related Hospitalization or ED Visits in Home Health Care Using Machine-Learning Algorithms

OBJECTIVE

Wound infection is prevalent in home healthcare (HHC) and often leads to hospitalizations. However, none of the previous studies of wounds in HHC have used data from clinical notes. Therefore, the authors created a more accurate description of a patient's condition by extracting risk factors from clinical notes to build predictive models to identify a patient's risk of wound infection in HHC.

METHODS

The structured data (eg, standardized assessments) and unstructured information (eg, narrative-free text charting) were retrospectively reviewed for HHC patients with wounds who were served by a large HHC agency in 2014. Wound infection risk factors were identified through bivariate analysis and stepwise variable selection. Risk predictive performance of three machine learning models (logistic regression, random forest, and artificial neural network) was compared.

RESULTS

A total of 754 of 54,316 patients (1.39%) had a hospitalization or ED visit related to wound infection. In the bivariate logistic regression, language describing wound type in the patient's clinical notes was strongly associated with risk (odds ratio, 9.94; P < .05). The areas under the curve were 0.82 in logistic regression, 0.75 in random forest, and 0.78 in artificial neural network. Risk prediction performance of the models improved (by up to 13.2%) after adding risk factors extracted from clinical notes.

CONCLUSIONS

Logistic regression showed the best risk prediction performance in prediction of wound infection-related hospitalization or ED visits in HHC. The use of data extracted from clinical notes can improve the performance of risk prediction models.

XGBoost Algorithm Prediction of Critical Care Outcome for Adult Patients Presenting to Emergency Department Using Initial Triage Information

Background

The emergency department (ED) triage system to classify and prioritize patients from high risk to less urgent continues to be a challenge.

Objective

This study, comprising 80,433 patients, aims to develop a machine learning algorithm prediction model of critical care outcomes for adult patients using information collected during ED triage and compare the performance with that of the baseline model using the Korean Triage and Acuity Scale (KTAS).

Methods

To predict the need for critical care, we used 13 predictors from triage information: age, gender, mode of ED arrival, the time interval between onset and ED arrival, reason of ED visit, chief complaints, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, oxygen saturation, and level of consciousness. The baseline model with KTAS was developed using logistic regression, and the machine learning model with 13 variables was generated using extreme gradient boosting (XGB) and deep neural network (DNN) algorithms. The discrimination was measured by the area under the receiver operating characteristic (AUROC) curve. The ability of calibration with Hosmer–Lemeshow test and reclassification with net reclassification index were evaluated. The calibration plot and partial dependence plot were used in the analysis.

Results

The AUROC of the model with the full set of variables (0.833-0.861) was better than that of the baseline model (0.796). The XGB model of AUROC 0.861 (95% CI 0.848-0.874) showed a higher discriminative performance than the DNN model of 0.833 (95% CI 0.819-0.848). The XGB and DNN models proved better reclassification than the baseline model with a positive net reclassification index. The XGB models were well-calibrated (Hosmer-Lemeshow test; P>.05); however, the DNN showed poor calibration power (Hosmer-Lemeshow test; P<.001). We further interpreted the nonlinear association between variables and critical care prediction.

Conclusions

Our study demonstrated that the performance of the XGB model using initial information at ED triage for predicting patients in need of critical care outperformed the conventional model with KTAS.

Automatic Classification of the Korean Triage Acuity Scale in Simulated Emergency Rooms Using Speech Recognition and Natural Language Processing: a Proof of Concept Study

BACKGROUND

Rapid triage reduces the patients' stay time at an emergency department (ED). The Korean Triage Acuity Scale (KTAS) is mandatorily applied at EDs in South Korea. For rapid triage, we studied machine learning-based triage systems composed of a speech recognition model and natural language processing-based classification.

METHODS

We simulated 762 triage cases that consisted of 18 classes with six types of the main symptom (chest pain, dyspnea, fever, stroke, abdominal pain, and headache) and three levels of KTAS. In addition, we recorded conversations between emergency patients and clinicians during the simulation. We used speech recognition models to transcribe the conversation. Bidirectional Encoder Representation from Transformers (BERT), support vector machine (SVM), random forest (RF), and k-nearest neighbors (KNN) were used for KTAS and symptom classification. Additionally, we evaluated the Shapley Additive exPlanations (SHAP) values of features to interpret the classifiers.

RESULTS

The character error rate of the speech recognition model was reduced to 25.21% through transfer learning. With auto-transcribed scripts, support vector machine (area under the receiver operating characteristic curve [AUROC], 0.86; 95% confidence interval [CI], 0.81-0.9), KNN (AUROC, 0.89; 95% CI, 0.85-0.93), RF (AUROC, 0.86; 95% CI, 0.82-0.9) and BERT (AUROC, 0.82; 95% CI, 0.75-0.87) achieved excellent classification performance. Based on SHAP, we found "stress", "pain score point", "fever", "breath", "head" and "chest" were the important vocabularies for determining KTAS and symptoms.

CONCLUSION

We demonstrated the potential of an automatic KTAS classification system using speech recognition models, machine learning and BERT-based classifiers.

Advanced natural language processing technique to predict patient disposition based on emergency triage notes

OBJECTIVE

To demonstrate the potential of machine learning and capability of natural language processing (NLP) to predict disposition of patients based on triage notes in the ED.

METHODS

A retrospective cohort of ED triage notes from St Vincent's Hospital (Melbourne) was used to develop a deep-learning algorithm that predicts patient disposition. Bidirectional Encoder Representations from Transformers, a recent language representation model developed by Google, was utilised for NLP. Eighty percent of the dataset was used for training the model and 20% was used to test the algorithm performance. Ktrain library, a wrapper for TensorFlow Keras, was employed to develop the model.

RESULTS

The accuracy of the algorithm was 83% and the area under the curve was 0.88. Sensitivity, specificity, precision and F1-score of the algorithm were 72%, 86%, 56% and 63%, respectively.

CONCLUSION

Machine learning and NLP can be together applied to the ED triage note to predict patient disposition with a high level of accuracy. The algorithm can potentially assist ED clinicians in early identification of patients requiring admission by mitigating the cognitive load, thus optimises resource allocation in EDs.

Clinical Profile, Prognostic Factors, and Outcome Prediction in Hospitalized Patients With Bloodstream Infection: Results From a 10-Year Prospective Multicenter Study

Background: Bloodstream infection (BSI) is one of the most common serious bacterial infections worldwide and also a major contributor to in-hospital mortality. Determining the predictors of mortality is crucial for prevention and improving clinical prognosis in patients with nosocomial BSI. Methods: A nationwide prospective cohort study was conducted from 2007 until 2016 in 16 teaching hospitals across China. Microbiological results, clinical information, and patient outcomes were collected to investigate the pathogenic spectrum and mortality rate in patients with BSI and identify outcome predictors using multivariate regression, prediction model, and Kaplan-Meier analysis. Results: No significant change was observed in the causative pathogen distribution during the 10-year period and the overall in-hospital mortality was 12.83% (480/3,741). An increased trend was found in the mortality of patients infected with Pseudomonas aeruginosa or Acinetobacter baumannii, while a decreased mortality rate was noted in Staphylococcus aureus-related BSI. In multivariable-adjusted models, higher mortality rate was significantly associated with older age, cancer, sepsis diagnosis, ICU admission, and prolonged hospital stay prior to BSI onset, which were also determined using machine learning-based predictive model achieved by random forest algorithm with a satisfactory performance in outcome prediction. Conclusions: Our study described the clinical and microbiological characteristics and mortality predictive factors in patients with BSI. These informative predictors would inform clinical practice to adopt effective therapeutic strategies to improve patient outcomes.

Health Care Professional Association Agency in Preparing for Artificial Intelligence: Protocol for a Multi-Case Study

BACKGROUND

The emergence of artificial intelligence (AI) in health care has impacted health care systems, including employment, training, education, and professional regulation. It is incumbent on health professional associations to assist their membership in defining and preparing for AI-related change. Health professional associations, or the national groups convened to represent the interests of the members of a profession, play a unique role in establishing the sociocultural, normative, and regulative elements of health care professions.

OBJECTIVE

The aim of this paper is to present a protocol for a proposed study of how, when faced with AI as a disruptive technology, health professional associations engage in sensemaking and legitimization of change to support their membership in preparing for future practice.

METHODS

An exploratory multi-case study approach will be used. This study will be informed by the normalization process theory (NPT), which suggests behavioral constructs required for complex change, providing a novel lens through which to consider the agency of macrolevel actors in practice change. A total of 4 health professional associations will be studied, each representing an instrumental case and related fields selected for their early consideration of AI technologies. Data collection will consist of key informant interviews, observation of relevant meetings, and document review. Individual and collective sensemaking activities and action toward change will be identified using stakeholder network mapping. A hybrid inductive and deductive model will be used for a concurrent thematic analysis, mapping emergent themes against the NPT framework to assess fit and identify areas of discordance.

RESULTS

As of January 2021, we have conducted 17 interviews, with representation across the 4 health professional associations. Of these 17 interviews, 15 (88%) have been transcribed. Document review is underway and complete for one health professional association and nearly complete for another. Observation opportunities have been challenged by competing priorities during COVID-19 and may require revisiting. A linear cross-case analytic approach will be taken to present the data, highlighting both guidance for the implementation of AI and implications for the application of NPT at the macro level. The ability to inform consideration of AI will depend on the degree to which the engaged health professional associations have considered this topic at the time of the study and, hence, what priority it has been assigned within the health professional association and what actions have been taken to consider or prepare for it. The fact that this may differ between health professional associations and practice environments will require consideration throughout the analysis.

CONCLUSIONS

Ultimately, this protocol outlines a case study approach to understand how, when faced with AI as a disruptive technology, health professional associations engage in sensemaking and legitimization of change to support their membership in preparing for future practice.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/27340.

Development and Validation of Machine Learning Models to Predict Admission From Emergency Department to Inpatient and Intensive Care Units

STUDY OBJECTIVE

This study aimed to develop and validate 2 machine learning models that use historical and current-visit patient data from electronic health records to predict the probability of patient admission to either an inpatient unit or ICU at each hour (up to 24 hours) of an emergency department (ED) encounter. The secondary goal was to provide a framework for the operational implementation of these machine learning models.

METHODS

Data were curated from 468,167 adult patient encounters in 3 EDs (1 academic and 2 community-based EDs) of a large academic health system from August 1, 2015, to October 31, 2018. The models were validated using encounter data from January 1, 2019, to December 31, 2019. An operational user dashboard was developed, and the models were run on real-time encounter data.

RESULTS

For the intermediate admission model, the area under the receiver operating characteristic curve was 0.873 and the area under the precision-recall curve was 0.636. For the ICU admission model, the area under the receiver operating characteristic curve was 0.951 and the area under the precision-recall curve was 0.461. The models had similar performance in both the academic- and community-based settings as well as across the 2019 and real-time encounter data.

CONCLUSION

Machine learning models were developed to accurately make predictions regarding the probability of inpatient or ICU admission throughout the entire duration of a patient's encounter in ED and not just at the time of triage. These models remained accurate for a patient cohort beyond the time period of the initial training data and were integrated to run on live electronic health record data, with similar performance.

Data Science Methods for Nursing-Relevant Patient Outcomes and Clinical Processes: The 2019 Literature Year in Review

Data science continues to be recognized and used within healthcare due to the increased availability of large data sets and advanced analytics. It can be challenging for nurse leaders to remain apprised of this rapidly changing landscape. In this article, we describe our findings from a scoping literature review of papers published in 2019 that use data science to explore, explain, and/or predict 15 phenomena of interest to nurses. Fourteen of the 15 phenomena were associated with at least one paper published in 2019. We identified the use of many contemporary data science methods (eg, natural language processing, neural networks) for many of the outcomes. We found many studies exploring Readmissions and Pressure Injuries. The topics of Artificial Intelligence/Machine Learning Acceptance, Burnout, Patient Safety, and Unit Culture were poorly represented. We hope that the studies described in this article help readers: (1) understand the breadth and depth of data science's ability to improve clinical processes and patient outcomes that are relevant to nurses and (2) identify gaps in the literature that are in need of exploration.

Criticality index conducted in pediatric emergency department triage

OBJECTIVE

To develop and analyze the performance of a machine learning model capable of predicting the disposition of patients presenting to a pediatric emergency department (ED) based on triage assessment and historical information mined from electronic health records.

METHODS

We retrospectively reviewed data from 585,142 ED visits at a pediatric quaternary care institution between 2013 and 2020. An extreme gradient boosting machine learning model was trained on a randomly selected training data set (50%) to stratify patients into 3 classes: (1) high criticality (patients requiring intensive care unit [ICU] care within 4 h of hospital admission, patients who died within 4 h of admission, and patients who died in the ED); (2) moderate criticality (patients requiring hospitalization without the need for ICU care); and (3) low criticality (patients discharged home). Variables considered during model development included triage vital signs, aspects of triage nursing assessment, demographics, and historical information (diagnoses, medication use, and healthcare utilization). Historical factors were limited to the 6 months preceding the index ED visit. The model was tested on a previously withheld test data set (40%), and its performance analyzed.

RESULTS

The distribution of criticality among high, moderate, and low was 1.5%, 7.1%, and 91.4%, respectively. The one-versus-all area under the receiver operating characteristic (AUROC) curve for high and moderate criticality was 0.982 (95% CI 0.980, 0.983) and 0.968 (0.967, 0.969). The multi-class macro average AUROC and area under the receiver operating characteristic curve were 0.976 and 0.754. The features most integral to model performance included history of intravenous medications, capillary refill, emergency severity index level, history of hospitalization, use of a supplemental oxygen device, age, and history of admission to the ICU.

CONCLUSION

Pediatric ED disposition can be accurately predicted using information available at triage, providing an opportunity to improve quality of care and patient outcomes.

Some machine’s doin’ that for you* – elektronische Triagesysteme in der Notaufnahme

Hintergrund

In den letzten 25 Jahren haben sich Triagesysteme zur Dringlichkeitseinschätzung in den Notaufnahmen etabliert. Die bisherigen symptomorientierten Triagesysteme haben allerdings auch Schwächen. Inzwischen ermöglichen die Digitalisierung der Notaufnahmen und die Leistungsfähigkeit der aktuellen Computergeneration bereits zum Triagezeitpunkt einen algorithmenbasierten Datenvergleich und eine Risikostratifizierung für bestimmte klinische Endpunkte über die reine Triagestufe hinaus.

Ziel der Arbeit

Nach selektiver Literaturrecherche erfolgt eine Übersicht über elektronische Triagesysteme (ETS). Das Funktionsprinzip und die aktuellen Möglichkeiten der ETS werden dargestellt. Daneben werden Chancen und Schwierigkeiten einer Etablierung von ETS in deutschen Notaufnahmen betrachtet.

Ergebnisse

Es wurden wesentliche Prädiktorvariablen wie Alter und bestimmte Vitalparameter identifiziert, die bisher nicht standardisiert in die Triagestufen einfließen, aber mithilfe von Modelllernen (ML) in belastbare Vorhersagen für klinische Endpunkte wie stationäre Aufnahme oder Mortalität einfließen können. Die Güte der Ersteinschätzung durch ein ETS ist insgesamt hoch. Ein ETS kann das Triagepersonal evidenzbasiert bei der Disposition der Patienten unterstützen und Über- und Untertriage reduzieren. Es gibt einige Entwicklungen, die günstige Bedingungen für den Einsatz von ETS in deutschen ZNA schaffen. So erleichtern z. B. repräsentative Notaufnahmeregister die Erstellung von Referenzdatensätzen, die zum Aufbau computerbasierter Klassifikationsmodelle benötigt werden. Außerdem müssen individuelle Patientendaten schnell verfügbar sein.

Schlussfolgerung

ETS können zur Erhöhung der Patientensicherheit und zur besseren Ressourcennutzung beitragen. Bislang fehlen allerdings noch objektive Referenzstandards und Leitlinien zum maschinellen Lernen.

Predicting hospitalization of pediatric asthma patients in emergency departments using machine learning

MOTIVATION

The timely identification of patients for hospitalization in emergency departments (EDs) can facilitate efficient use of hospital resources. Machine learning can help the early prediction of ED disposition; however, application of machine learning models requires both computer science skills and domain knowledge. This presents a barrier for those who want to apply machine learning to real-world settings.

OBJECTIVES

The objective of this study was to construct a competitive predictive model with a minimal amount of human effort to facilitate decisions regarding hospitalization of patients.

METHODS

This study used the electronic health record data from five EDs in a single healthcare system, including an academic urban children's hospital ED, from January 2009 to December 2013. We constructed two machine learning models by using automated machine learning algorithm (autoML) which allows non-experts to use machine learning model: one with data only available at ED triage, the other adding information available one hour into the ED visit. Random forest and logistic regression were employed as bench-marking models. The ratio of the training dataset to the test dataset was 8:2, and the area under the receiver operating characteristic curve (AUC), accuracy, and F1 were calculated to assess the quality of the models.

RESULTS

Of the 9,069 ED visits analyzed, the study population was made up of males (62.7 %), median (IQR) age was 6 (4, 10) years, and public insurance coverage (66.0 %). The majority had an Emergency Severity Index score of 3 (52.9 %). The prevalence of hospitalization was 22.5 %. The AUCs were 0.914 and 0.942. AUCs were 0.831 and 0.886 for random forests, and 0.795 and 0.823 for logistic regression. Among the predictors, an outcome at a prior visit, ESI level, time to first medication, and time to triage were the most important features for the prediction of the need for hospitalization.

CONCLUSIONS

In comparison with the conventional approaches, the use of autoML improved the predictive ability for the need for hospitalization. The findings can optimize ED management, hospital-level resource utilization and improve quality. Furthermore, this approach can support the design of a more effective patient ED flow for pediatric asthma care.

Predicting Admissions From a Paediatric Emergency Department - Protocol for Developing and Validating a Low-Dimensional Machine Learning Prediction Model

Introduction: Patients boarding in the Emergency Department can contribute to overcrowding, leading to longer waiting times and patients leaving without being seen or completing their treatment. The early identification of potential admissions could act as an additional decision support tool to alert clinicians that a patient needs to be reviewed for admission and would also be of benefit to bed managers in advance bed planning for the patient. We aim to create a low-dimensional model predicting admissions early from the paediatric Emergency Department.

Methods and Analysis: The methodology Cross Industry Standard Process for Data Mining (CRISP-DM) will be followed. The dataset will comprise of 2 years of data, ~76,000 records. Potential predictors were identified from previous research, comprising of demographics, registration details, triage assessment, hospital usage and past medical history. Fifteen models will be developed comprised of 3 machine learning algorithms (Logistic regression, naïve Bayes and gradient boosting machine) and 5 sampling methods, 4 of which are aimed at addressing class imbalance (undersampling, oversampling, and synthetic oversampling techniques). The variables of importance will then be identified from the optimal model (selected based on the highest Area under the curve) and used to develop an additional low-dimensional model for deployment.

Discussion: A low-dimensional model comprised of routinely collected data, captured up to post triage assessment would benefit many hospitals without data rich platforms for the development of models with a high number of predictors. Novel to the planned study is the use of data from the Republic of Ireland and the application of sampling techniques aimed at improving model performance impacted by an imbalance between admissions and discharges in the outcome variable.

Clinical performance of early warning scoring systems for identifying sepsis among anti-hypertensive agent users

BACKGROUND

Little is known about the accuracy of the quick Sequential Organ Failure Assessment (qSOFA) and the National Early Warning Score (NEWS) in identifying sepsis patients with a history of hypertension on anti-hypertensive agents, which affect vital signs as components of the scoring systems. We aimed to examine the ability of qSOFA and NEWS to predict sepsis among anti-hypertensive agent users by comparing them with non-users.

METHODS

We retrospectively identified adult patients (aged ≥18years) with suspected infection who presented to an emergency department (ED) of a large tertiary medical center in Japan between April 2018 and March 2020. Suspected infection was defined based on the chief complaint of fever, high body temperature, or the clinical context on arrival at the ED. We excluded patients who had trauma or cardiac arrest, those who were transported to other hospitals after arrival at the ED, and those whose vital signs data were mostly missing. The predictive performances of qSOFA and NEWS based on initial vital signs were examined separately for sepsis, ICU admission, and in-hospital mortality and compared between anti-hypertensive agent users and non-users.

RESULTS

Among 2900 patients with suspected infection presenting to the ED, 291 (10%) had sepsis, 1023 (35%) were admitted to the ICU, and 188 (6.5%) died. The prediction performances of qSOFA and NEWS for each outcome among anti-hypertensive agent users were lower than that among non-users (e.g., c-statistics of qSOFA for sepsis, 0.66 vs. 0.71, p = 0.07; and for ICU admission, 0.70 vs. 0.75, p = 0.01). For identifying sepsis, the sensitivity and specificity of qSOFA ≥2 were 0.43 and 0.77 in anti-hypertensive agent users and 0.51 and 0.82 in non-users. Similar associations were observed for identifying ICU admission and in-hospital mortality. Regardless of the use of anti-hypertensive agents, NEWS had better prediction abilities for each outcome than qSOFA.

CONCLUSION

The clinical performance of qSOFA and NEWS for identifying sepsis among anti-hypertensive agent users was likely lower than that among non-users.

Automated remote decision-making algorithm as a primary triage system using machine learning techniques

OBJECTIVE

An objective and convenient primary triage procedure is needed for prioritizing patients who need help in mass casualty incident (MCI) situations, where there is a lack of medical staff and available resources. This study aimed to develop an automated remote decision-making algorithm that remotely categorize a patient's emergency level using clinical parameters that can be measured with a wearable device.

APPROACH

The algorithm was developed according to the following procedures. First, we used the National Trauma Data Bank data set, a large open trauma patient data set assembled by the American College of Surgeons (ACS). In addition, we performed pre-processing to exclude data when the vital sign or consciousness indicator value was missing or physiologically in an abnormal range. Second, we selected the T-RTS method, which classifies emergency levels into four classes (Delayed, Urgent, Immediate and Dead), as the primary outcome. Third, three machine learning methods widely used in the medical field, logistic regression, random forest, and deep neural network (DNN), were applied to build the algorithm. Finally, each method was evaluated using quantitative performance indicators including the macro-averaged f1 score, macro-averaged mean absolute error (MMAE), and the area under the receiver operating characteristic curve (AUC).

MAIN RESULTS

For total sets, the logistic regression had a macro-averaged f1 score of 0.673, an MMAE of 0.387 and an AUC value of 0.844 (95% CI, 0.843-0.845), while the random forest and DNN had macro-averaged f1 scores of 0.783 and 0.784, MMAEs of 0.297 and 0.298 and AUC values of 0.882 (95% CI, 0.881-0.883) and 0.883(95% CI, 0.881-0.884), respectively.

SIGNIFICANCE

In a comprehensive analysis of these results, our algorithm demonstrated a viable approach that could be practically adopted in an MCI. In addition, it can be employed to transfer patients and to redistribute available resources according to their priorities.