Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index

Use of natural language processing to improve predictive models for imaging utilization in children presenting to the emergency department

OBJECTIVE

To examine the association between the medical imaging utilization and information related to patients' socioeconomic, demographic and clinical factors during the patients' ED visits; and to develop predictive models using these associated factors including natural language elements to predict the medical imaging utilization at pediatric ED.

METHODS

Pediatric patients' data from the 2012-2016 United States National Hospital Ambulatory Medical Care Survey was included to build the models to predict the use of imaging in children presenting to the ED. Multivariable logistic regression models were built with structured variables such as temperature, heart rate, age, and unstructured variables such as reason for visit, free text nursing notes and combined data available at triage. NLP techniques were used to extract information from the unstructured data.

RESULTS

Of the 27,665 pediatric ED visits included in the study, 8394 (30.3%) received medical imaging in the ED, including 6922 (25.0%) who had an X-ray and 1367 (4.9%) who had a computed tomography (CT) scan. In the predictive model including only structured variables, the c-statistic was 0.71 (95% CI: 0.70-0.71) for any imaging use, 0.69 (95% CI: 0.68-0.70) for X-ray, and 0.77 (95% CI: 0.76-0.78) for CT. Models including only unstructured information had c-statistics of 0.81 (95% CI: 0.81-0.82) for any imaging use, 0.82 (95% CI: 0.82-0.83) for X-ray, and 0.85 (95% CI: 0.83-0.86) for CT scans. When both structured variables and free text variables were included, the c-statistics reached 0.82 (95% CI: 0.82-0.83) for any imaging use, 0.83 (95% CI: 0.83-0.84) for X-ray, and 0.87 (95% CI: 0.86-0.88) for CT.

CONCLUSIONS

Both CT and X-rays are commonly used in the pediatric ED with one third of the visits receiving at least one. Patients' socioeconomic, demographic and clinical factors presented at ED triage period were associated with the medical imaging utilization. Predictive models combining structured and unstructured variables available at triage performed better than models using structured or unstructured variables alone, suggesting the potential for use of NLP in determining resource utilization.

A validation of machine learning-based risk scores in the prehospital setting

BACKGROUND

The triage of patients in prehospital care is a difficult task, and improved risk assessment tools are needed both at the dispatch center and on the ambulance to differentiate between low- and high-risk patients. This study validates a machine learning-based approach to generating risk scores based on hospital outcomes using routinely collected prehospital data.

METHODS

Dispatch, ambulance, and hospital data were collected in one Swedish region from 2016-2017. Dispatch center and ambulance records were used to develop gradient boosting models predicting hospital admission, critical care (defined as admission to an intensive care unit or in-hospital mortality), and two-day mortality. Composite risk scores were generated based on the models and compared to National Early Warning Scores (NEWS) and actual dispatched priorities in a prospectively gathered dataset from 2018.

RESULTS

A total of 38203 patients were included from 2016-2018. Concordance indexes (or areas under the receiver operating characteristics curve) for dispatched priorities ranged from 0.51-0.66, while those for NEWS ranged from 0.66-0.85. Concordance ranged from 0.70-0.79 for risk scores based only on dispatch data, and 0.79-0.89 for risk scores including ambulance data. Dispatch data-based risk scores consistently outperformed dispatched priorities in predicting hospital outcomes, while models including ambulance data also consistently outperformed NEWS. Model performance in the prospective test dataset was similar to that found using cross-validation, and calibration was comparable to that of NEWS.

CONCLUSIONS

Machine learning-based risk scores outperformed a widely-used rule-based triage algorithm and human prioritization decisions in predicting hospital outcomes. Performance was robust in a prospectively gathered dataset, and scores demonstrated adequate calibration. Future research should explore the robustness of these methods when applied to other settings, establish appropriate outcome measures for use in determining the need for prehospital care, and investigate the clinical impact of interventions based on these methods.

Machine Learning-Based Prediction of Korean Triage and Acuity Scale Level in Emergency Department Patients

Objectives

Triage is a process to accurately assess and classify symptoms to identify and provide rapid treatment to patients. The Korean Triage and Acuity Scale (KTAS) is used as a triage instrument in all emergency centers. The aim of this study was to train and compare machine learning models to predict KTAS levels.

Methods

This was a cross-sectional study using data from a single emergency department of a tertiary university hospital. Information collected during triage was used in the analysis. Logistic regression, random forest, and XGBoost were used to predict the KTAS level.

Results

The models with the highest area under the receiver operating characteristic curve (AUROC) were the random forest and XGBoost models trained on the entire dataset (AUROC = 0.922, 95% confidence interval 0.917-0.925 and AUROC = 0.922, 95% confidence interval 0.918-0.925, respectively). The AUROC of the models trained on the clinical data was higher than that of models trained on text data only, but the models trained on all variables had the highest AUROC among similar machine learning models.

Conclusions

Machine learning can robustly predict the KTAS level at triage, which may have many possibilities for use, and the addition of text data improves the predictive performance compared to that achieved by using structured data alone.

Application of Artificial Intelligence in the Health Care Safety Context: Opportunities and Challenges

There is a growing awareness that artificial intelligence (AI) has been used in the analysis of complicated and big data to provide outputs without human input in various health care contexts, such as bioinformatics, genomics, and image analysis. Although this technology can provide opportunities in diagnosis and treatment processes, there still may be challenges and pitfalls related to various safety concerns. To shed light on such opportunities and challenges, this article reviews AI in health care along with its implication for safety. To provide safer technology through AI, this study shows that safe design, safety reserves, safe fail, and procedural safeguards are key strategies, whereas cost, risk, and uncertainty should be identified for all potential technical systems. It is also suggested that clear guidance and protocols should be identified and shared with all stakeholders to develop and adopt safer AI applications in the health care context.

Readmission Risk Trajectories for Patients With Heart Failure Using a Dynamic Prediction Approach: Retrospective Study

Background

Patients hospitalized with heart failure suffer the highest rates of 30-day readmission among other clinically defined patient populations in the United States. Investigation into the predictability of 30-day readmissions can lead to clinical decision support tools and targeted interventions that can help care providers to improve individual patient care and reduce readmission risk.

Objective

This study aimed to develop a dynamic readmission risk prediction model that yields daily predictions for patients hospitalized with heart failure toward identifying risk trajectories over time and identifying clinical predictors associated with different patterns in readmission risk trajectories.

Methods

A two-stage predictive modeling approach combining logistic and beta regression was applied to electronic health record data accumulated daily to predict 30-day readmission for 534 hospital encounters of patients with heart failure over 2750 patient days. Unsupervised clustering was performed on predictions to uncover time-dependent trends in readmission risk over the patient’s hospital stay. We used data collected between September 1, 2013, and August 31, 2015, from a community hospital in Maryland (United States) for patients with a primary diagnosis of heart failure. Patients who died during the hospital stay or were transferred to other acute care hospitals or hospice care were excluded.

Results

Readmission occurred in 107 (107/534, 20.0%) encounters. The out-of-sample area under curve for the 2-stage predictive model was 0.73 (SD 0.08). Dynamic clinical predictors capturing laboratory results and vital signs had the highest predictive value compared with demographic, administrative, medical, and procedural data included. Unsupervised clustering identified four risk trajectory groups: decreasing risk (131/534, 24.5% encounters), high risk (113/534, 21.2%), moderate risk (177/534, 33.1%), and low risk (113/534, 21.2%). The decreasing risk group demonstrated change in average probability of readmission from admission (0.69) to discharge (0.30), whereas the high risk (0.75), moderate risk (0.61), and low risk (0.39) groups maintained consistency over the hospital course. A higher level of hemoglobin, larger decrease in potassium and diastolic blood pressure from admission to discharge, and smaller number of past hospitalizations are associated with decreasing readmission risk (P<.001).

Conclusions

Dynamically predicting readmission and quantifying trends over patients’ hospital stay illuminated differing risk trajectory groups. Identifying risk trajectory patterns and distinguishing predictors may shed new light on indicators of readmission and the isolated effects of the index hospitalization.

Urinary Stone Detection on CT Images Using Deep Convolutional Neural Networks: Evaluation of Model Performance and Generalization

PURPOSE

To investigate the diagnostic accuracy of cascading convolutional neural network (CNN) for urinary stone detection on unenhanced CT images and to evaluate the performance of pretrained models enriched with labeled CT images across different scanners.

MATERIALS AND METHODS

This HIPAA-compliant, institutional review board-approved, retrospective clinical study used unenhanced abdominopelvic CT scans from 535 adults suspected of having urolithiasis. The scans were obtained on two scanners (scanner 1 [hereafter S1] and scanner 2 [hereafter S2]). A radiologist reviewed clinical reports and labeled cases for determination of reference standard. Stones were present on 279 (S1, 131; S2, 148) and absent on 256 (S1, 158; S2, 98) scans. One hundred scans (50 from each scanner) were randomly reserved as the test dataset, and the rest were used for developing a cascade of two CNNs: The first CNN identified the extent of the urinary tract, and the second CNN detected presence of stone. Nine variations of models were developed through the combination of different training data sources (S1, S2, or both [hereafter SB]) with (ImageNet, GrayNet) and without (Random) pretrained CNNs. First, models were compared for generalizability at the section level. Second, models were assessed by using area under the receiver operating characteristic curve (AUC) and accuracy at the patient level with test dataset from both scanners (n = 100).

RESULTS

The GrayNet-pretrained model showed higher classifier exactness than did ImageNet-pretrained or Random-initialized models when tested by using data from the same or different scanners at section level. At the patient level, the AUC for stone detection was 0.92-0.95, depending on the model. Accuracy of GrayNet-SB (95%) was higher than that of ImageNet-SB (91%) and Random-SB (88%). For stones larger than 4 mm, all models showed similar performance (false-negative results: two of 34). For stones smaller than 4 mm, the number of false-negative results for GrayNet-SB, ImageNet-SB, and Random-SB were one of 16, three of 16, and five of 16, respectively. GrayNet-SB identified stones in all 22 test cases that had obstructive uropathy.

CONCLUSION

A cascading model of CNNs can detect urinary tract stones on unenhanced CT scans with a high accuracy (AUC, 0.954). Performance and generalization of CNNs across scanners can be enhanced by using transfer learning with datasets enriched with labeled medical images.© RSNA, 2019Supplemental material is available for this article. : An earlier incorrect version appeared online. This article was corrected on August 6, 2019.

Prediction of emergency department patient disposition based on natural language processing of triage notes

BACKGROUND

Nursing triage documentation is the first free-form text data created at the start of an emergency department (ED) visit. These 1-3 unstructured sentences reflect the clinical impression of an experienced nurse and are key in gauging a patient's illness. We aimed to predict final ED disposition using three commonly-employed natural language processing (NLP) techniques of nursing triage notes in isolation from other data.

METHODS

We constructed a retrospective cohort of all 260,842 consecutive ED encounters in 2015-16, from three clinically heterogeneous academically-affiliated EDs. After exclusion of 3964 encounters based on completeness of triage, and disposition data, we included 256,878 encounters. We defined the outcome as: 1) admission, transfer, or in-ED death [68,092 encounters] vs. 2) discharge, "left without being seen," and "left against medical advice" [188,786 encounters]. The dataset was divided into training and testing subsets. Neural network regression models were trained using bag-of-words, paragraph vectors, and topic distributions to predict disposition and were evaluated using the testing dataset.

RESULTS

Area under the curve for disposition using triage notes as bag-of-words, paragraph vectors, and topic distributions were 0.737 (95% CI: 0.734 - 0.740), 0.785 (95% CI: 0.782 - 0.788), and 0.687 (95% CI: 0.684 - 0.690), respectively.

CONCLUSIONS

Nursing triage notes can be used to predict final ED patient disposition, even when used separately from other clinical information. These findings have substantial implications for future studies, suggesting that free text from medical records may be considered as a critical predictor in research of patient outcomes.

Association of Vital Signs and Process Outcomes in Emergency Department Patients

Introduction

We sought to determine the association of abnormal vital signs with emergency department (ED) process outcomes in both discharged and admitted patients.

Methods

We performed a retrospective review of five years of operational data at a single site. We identified all visits for patients 18 and older who were discharged home without ancillary services, and separately identified all visits for patients admitted to a floor (ward) bed. We assessed two process outcomes for discharged visits (returns to the ED within 72 hours and returns to the ED within 72 hours resulting in admission) and two process outcomes for admitted patients (transfer to a higher level of care [intermediate care or intensive care] within either six hours or 24 hours of arrival to floor). Last-recorded ED vital signs were obtained for all patients. We report rates of abnormal vital signs in each group, as well as the relative risk of meeting a process outcome for each individual vital sign abnormality.

Results

Patients with tachycardia, tachypnea, or fever more commonly experienced all measured process outcomes compared to patients without these abnormal vitals; admitted hypotensive patients more frequently required transfer to a higher level of care within 24 hours.

Conclusion

In a single facility, patients with abnormal last-recorded ED vital signs experienced more undesirable process outcomes than patients with normal vitals. Vital sign abnormalities may serve as a useful signal in outcome forecasting.

Advanced diagnostic imaging utilization during emergency department visits in the United States: A predictive modeling study for emergency department triage

Background

Emergency department (ED) crowding is associated with negative health outcomes, patient dissatisfaction, and longer length of stay (LOS). The addition of advanced diagnostic imaging (ADI), namely CT, ultrasound (U/S), and MRI to ED encounter work up is a predictor of longer length of stay. Earlier and improved prediction of patients’ need for advanced imaging may improve overall ED efficiency. The aim of the study was to detect the association between ADI utilization and the structured and unstructured information immediately available during ED triage, and to develop and validate models to predict utilization of ADI during an ED encounter.

Methods

We used the United States National Hospital Ambulatory Medical Care Survey data from 2009 to 2014 to examine which sociodemographic and clinical factors immediately available at ED triage were associated with the utilization of CT, U/S, MRI, and multiple ADI during a patient’s ED stay. We used natural language processing (NLP) topic modeling to incorporate free-text reason for visit data available at time of ED triage in addition to other structured patient data to predict the use of ADI using multivariable logistic regression models.

Results

Among the 139,150 adult ED visits from a national probability sample of hospitals across the U.S, 21.9% resulted in ADI use, including 16.8% who had a CT, 3.6% who had an ultrasound, 0.4% who had an MRI, and 1.2% of the population who had multiple types of ADI. The c-statistic of the predictive models was greater than or equal to 0.78 for all imaging outcomes, and the addition of text-based reason for visit information improved the accuracy of all predictive models.

Conclusions

Patient information immediately available during ED triage can accurately predict the eventual use of advanced diagnostic imaging during an ED visit. Such models have the potential to be incorporated into the ED triage workflow in order to more rapidly identify patients who may require advanced imaging during their ED stay and assist with medical decision-making.

Emergency department triage prediction of clinical outcomes using machine learning models

Background

Development of emergency department (ED) triage systems that accurately differentiate and prioritize critically ill from stable patients remains challenging. We used machine learning models to predict clinical outcomes, and then compared their performance with that of a conventional approach—the Emergency Severity Index (ESI).

Methods

Using National Hospital and Ambulatory Medical Care Survey (NHAMCS) ED data, from 2007 through 2015, we identified all adult patients (aged ≥ 18 years). In the randomly sampled training set (70%), using routinely available triage data as predictors (e.g., demographics, triage vital signs, chief complaints, comorbidities), we developed four machine learning models: Lasso regression, random forest, gradient boosted decision tree, and deep neural network. As the reference model, we constructed a logistic regression model using the five-level ESI data. The clinical outcomes were critical care (admission to intensive care unit or in-hospital death) and hospitalization (direct hospital admission or transfer). In the test set (the remaining 30%), we measured the predictive performance, including area under the receiver-operating-characteristics curve (AUC) and net benefit (decision curves) for each model.

Results

Of 135,470 eligible ED visits, 2.1% had critical care outcome and 16.2% had hospitalization outcome. In the critical care outcome prediction, all four machine learning models outperformed the reference model (e.g., AUC, 0.86 [95%CI 0.85–0.87] in the deep neural network vs 0.74 [95%CI 0.72–0.75] in the reference model), with less under-triaged patients in ESI triage levels 3 to 5 (urgent to non-urgent). Likewise, in the hospitalization outcome prediction, all machine learning models outperformed the reference model (e.g., AUC, 0.82 [95%CI 0.82–0.83] in the deep neural network vs 0.69 [95%CI 0.68–0.69] in the reference model) with less over-triages in ESI triage levels 1 to 3 (immediate to urgent). In the decision curve analysis, all machine learning models consistently achieved a greater net benefit—a larger number of appropriate triages considering a trade-off with over-triages—across the range of clinical thresholds.

Conclusions

Compared to the conventional approach, the machine learning models demonstrated a superior performance to predict critical care and hospitalization outcomes. The application of modern machine learning models may enhance clinicians’ triage decision making, thereby achieving better clinical care and optimal resource utilization.

Electronic supplementary material

The online version of this article (10.1186/s13054-019-2351-7) contains supplementary material, which is available to authorized users.

Cytokines and Chemokines in Pediatric Appendicitis: A Multiplex Analysis of Inflammatory Protein Mediators

OBJECTIVES

We aimed to demonstrate the potential of precision medicine to describe the inflammatory landscape present in children with suspected appendicitis. Our primary objective was to determine levels of seven inflammatory protein mediators previously associated with intra-abdominal inflammation (C-reactive protein-CRP, procalcitonin-PCT, interleukin-6 (IL), IL-8, IL-10, monocyte chemoattractant protein-1-MCP-1, and serum amyloid A-SAA) in a cohort of children with suspected appendicitis. Subsequently, using a multiplex proteomics approach, we examined an expansive array of novel candidate cytokine and chemokines within this population.

METHODS

We performed a secondary analysis of targeted proteomics data from Alberta Sepsis Network studies. Plasma mediator levels, analyzed by Luminex multiplex assays, were evaluated in children aged 5-17 years with nonappendicitis abdominal pain (NAAP), acute appendicitis (AA), and nonappendicitis sepsis (NAS). We used multivariate regression analysis to evaluate the seven target proteins, followed by decision tree and heat mapping analyses for all proteins evaluated.

RESULTS

185 children were included: 83 with NAAP, 79 AA, and 23 NAS. Plasma levels of IL-6, CRP, MCP-1, PCT, and SAA were significantly different in children with AA compared to those with NAAP (p < 0.001). Expansive proteomic analysis demonstrated 6 patterns in inflammatory mediator profiles based on severity of illness. A decision tree incorporating the proteins CRP, ferritin, SAA, regulated on activation normal T-cell expressed and secreted (RANTES), monokine induced by gamma interferon (MIG), and PCT demonstrated excellent specificity (0.920) and negative predictive value (0.882) for children with appendicitis.

CONCLUSIONS

Multiplex proteomic analyses described the inflammatory landscape of children presenting to the ED with suspected appendicitis. We have demonstrated the feasibility of this approach to identify potential novel candidate cytokines/chemokine patterns associated with a specific illness (appendicitis) amongst those with a broad ED presentation (abdominal pain). This approach can be modelled for future research initiatives in pediatric emergency medicine.

Machine Learning in Relation to Emergency Medicine Clinical and Operational Scenarios: An Overview

Health informatics is a vital technology that holds great promise in the healthcare setting. We describe two prominent health informatics tools relevant to emergency care, as well as the historical background and the current state of informatics. We also identify recent research findings and practice changes. The recent advances in machine learning and natural language processing (NLP) are a prominent development in health informatics overall and relevant in emergency medicine (EM). A basic comprehension of machine-learning algorithms is the key to understand the recent usage of artificial intelligence in healthcare. We are using NLP more in clinical use for documentation. NLP has started to be used in research to identify clinically important diseases and conditions. Health informatics has the potential to benefit both healthcare providers and patients. We cover two powerful tools from health informatics for EM clinicians and researchers by describing the previous successes and challenges and conclude with their implications to emergency care.

Validité de la FRENCH, l’échelle de tri des urgences hospitalières élaborée par la Société française de médecine d’urgence

Introduction : La Société française de médecine d’urgence a créé en 2016 une échelle de tri pour les infirmier( ière)s d’accueil dans une structure d’urgences. Cette échelle appelée FRENCH (FRench Emergency Nurses Classification in Hospital) classe les tris de 5 à 1 (du moins urgent au plus urgent) en fonction du pronostic et de la complexité/ sévérité des motifs de recours aux soins. Le tri 3, groupe hétérogène dans les échelles de tri internationales, a été subdivisé en deux niveaux pour prioriser les patients qui ont une comorbidité en rapport avec le motif de recours aux soins ou qui sont adressés par un médecin et qui sont prioritaires (3A) par rapport aux autres patients (3B).

Objectif : Évaluer la pertinence de la FRENCH.

Méthode : Nous avons analysé les données démographiques, les paramètres vitaux, les examens complémentaires prescrits et les durées de prise en charge de tous les patients accueillis dans un hôpital universitaire sur une période de neuf mois consécutifs. Le critère de jugement était l’existence d’une relation croissante entre le niveau de complexité/ sévérité des patients — reflétée par le taux d’hospitalisation et la prescription d’examens complémentaires—et le niveau de priorité du tri.

Résultats : L’étude a inclus 27 598 patients. La répartition des patients par niveaux de tri était : 0,4 (tri 1), 6,7 (tri 2), 13,3 (tri 3A), 29,4 (tri 3B), 43,1 (tri 4) et 7,1 % (tri 5). Le taux d’hospitalisation était croissant quand l’ordre de priorité augmentait. La relation entre les niveaux de tri et le taux d’hospitalisation mesurée par l’aire sous la courbe (0,83 : intervalle de confiance à 95 % : [0,82–0,83]) était bonne. La prescription des examens complémentaires était croissante quand l’ordre de priorité augmentait. La corrélation entre les niveaux de tri et un indice global d’examens complémentaires prescrits était modérée (K = 0,51).

Conclusion : Cette étude monocentrique valide la pertinence de l’échelle de tri FRENCH à six niveaux par sa bonne capacité à classer les patients selon leur complexité/sévérité. De nouvelles évaluations sont nécessaires dans d’autres structures d’urgences pour confirmer sa performance et favoriser son évolution.

Machine Learning-Based Prediction of Clinical Outcomes for Children During Emergency Department Triage

IMPORTANCE

While machine learning approaches may enhance prediction ability, little is known about their utility in emergency department (ED) triage.

OBJECTIVES

To examine the performance of machine learning approaches to predict clinical outcomes and disposition in children in the ED and to compare their performance with conventional triage approaches.

DESIGN, SETTING, AND PARTICIPANTS

Prognostic study of ED data from the National Hospital Ambulatory Medical Care Survey from January 1, 2007, through December 31, 2015. A nationally representative sample of 52 037 children aged 18 years or younger who presented to the ED were included. Data analysis was performed in August 2018.

MAIN OUTCOMES AND MEASURES

The outcomes were critical care (admission to an intensive care unit and/or in-hospital death) and hospitalization (direct hospital admission or transfer). In the training set (70% random sample), using routinely available triage data as predictors (eg, demographic characteristics and vital signs), we derived 4 machine learning-based models: lasso regression, random forest, gradient-boosted decision tree, and deep neural network. In the test set (the remaining 30% of the sample), we measured the models' prediction performance by computing C statistics, prospective prediction results, and decision curves. These machine learning models were built for each outcome and compared with the reference model using the conventional triage classification information.

RESULTS

Of 52 037 eligible ED visits by children (median [interquartile range] age, 6 [2-14] years; 24 929 [48.0%] female), 163 (0.3%) had the critical care outcome and 2352 (4.5%) had the hospitalization outcome. For the critical care prediction, all machine learning approaches had higher discriminative ability compared with the reference model, although the difference was not statistically significant (eg, C statistics of 0.85 [95% CI, 0.78-0.92] for the deep neural network vs 0.78 [95% CI, 0.71-0.85] for the reference; P = .16), and lower number of undertriaged critically ill children in the conventional triage levels 3 to 5 (urgent to nonurgent). For the hospitalization prediction, all machine learning approaches had significantly higher discrimination ability (eg, C statistic, 0.80 [95% CI, 0.78-0.81] for the deep neural network vs 0.73 [95% CI, 0.71-0.75] for the reference; P < .001) and fewer overtriaged children who did not require inpatient management in the conventional triage levels 1 to 3 (immediate to urgent). The decision curve analysis demonstrated a greater net benefit of machine learning models over ranges of clinical thresholds.

CONCLUSIONS AND RELEVANCE

Machine learning-based triage had better discrimination ability to predict clinical outcomes and disposition, with reduction in undertriaging critically ill children and overtriaging children who are less ill.

Assessment of a Simulated Case-Based Measurement of Physician Diagnostic Performance

IMPORTANCE

Diagnostic acumen is a fundamental skill in the practice of medicine. Scalable, practical, and objective tools to assess diagnostic performance are lacking.

OBJECTIVE

To validate a new method of assessing diagnostic performance that uses automated techniques to assess physicians' diagnostic performance on brief, open-ended case simulations.

DESIGN, SETTING, AND PARTICIPANTS

Retrospective cohort study of 11 023 unique attempts to solve case simulations on an online software platform, The Human Diagnosis Project (Human Dx). A total of 1738 practicing physicians, residents (internal medicine, family medicine, and emergency medicine), and medical students throughout the United States voluntarily used Human Dx software between January 21, 2016, and January 15, 2017.

MAIN OUTCOMES AND MEASURES

Internal structure validity was assessed by 3 measures of diagnostic performance: accuracy, efficiency, and a combined score (Diagnostic Acumen Precision Performance [DAPP]). These were each analyzed by level of training. Association with other variables' validity evidence was evaluated by correlating diagnostic performance and affiliation with an institution ranked in the top 25 medical schools by US News and World Report.

RESULTS

Data were analyzed for 239 attending physicians, 926 resident physicians, 347 intern physicians, and 226 medical students. Attending physicians had higher mean accuracy scores than medical students (difference, 8.1; 95% CI, 4.2-12.0; P < .001), as did residents (difference, 8.0; 95% CI, 4.8-11.2; P < .001) and interns (difference, 5.9; 95% CI, 2.3-9.6; P < .001). Attending physicians had higher mean efficiency compared with residents (difference, 4.8; 95% CI, 1.8-7.8; P < .001), interns (difference, 5.0; 95% CI, 1.5-8.4; P = .001), and medical students (difference, 5.4; 95% CI, 1.4-9.3; P = .003). Attending physicians also had significantly higher mean DAPP scores than residents (difference, 2.6; 95% CI, 0.0-5.2; P = .05), interns (difference, 3.6; 95% CI, 0.6-6.6; P = .01), and medical students (difference, 6.7; 95% CI, 3.3-10.2; P < .001). Attending physicians affiliated with a US News and World Report-ranked institution had higher mean DAPP scores compared with nonaffiliated attending physicians (80 [95% CI, 77-83] vs 72 [95% CI, 70-74], respectively; P < .001). Resident physicians affiliated with an institution ranked in the top 25 medical schools by US News and World Report also had higher mean DAPP scores compared with nonaffiliated peers (75 [95% CI, 73-77] vs 71 [95% CI, 69-72], respectively; P < .001).

CONCLUSIONS AND RELEVANCE

The data suggest that diagnostic performance is higher in those with more training and that DAPP scores may be a valid measure to appraise diagnostic performance. This diagnostic assessment tool allows individuals to receive immediate feedback on performance through an openly accessible online platform.

A review of triage accuracy and future direction

BACKGROUND

In the emergency department, it is important to identify and prioritize who requires an urgent intervention in a short time. Triage helps recognize the urgency among patients. An accurate triage decision helps patients receive the emergency service in the most appropriate time. Various triage systems have been developed and verified to assist healthcare providers to make accurate triage decisions. The triage accuracy can represent the quality of emergency service, but there is a lack of review studies addressing this topic.

METHODS

A literature search was conducted in four electronic databases where 'emergency nursing' and 'triage accuracy' were used as keywords. Studies published from 2008 January to 2018 August were included as potential subjects. Nine studies were included in this review after the inclusion and exclusion criteria were applied.

RESULTS

Written case scenarios and retrospective review were commonly used to examine the triage accuracy. The triage accuracy from studies was in moderate level. The single-center studies which held better results than those from multi-center studies revealed the need of triage training and consistent training between emergency departments.

CONCLUSIONS

Regular refresher triage training, collaboration between emergency departments and continuous monitoring were necessary to strengthen the use of triage systems and improve nurse's triage performance.

Triage Performance in Emergency Medicine: A Systematic Review

STUDY OBJECTIVE

Rapid growth in emergency department (ED) triage literature has been accompanied by diversity in study design, methodology, and outcome assessment. We aim to synthesize existing ED triage literature by using a framework that enables performance comparisons and benchmarking across triage systems, with respect to clinical outcomes and reliability.

METHODS

PubMed, EMBASE, Scopus, and Web of Science were systematically searched for studies of adult ED triage systems through 2016. Studies evaluating triage systems with evidence of widespread adoption (Australian Triage Scale, Canadian Triage and Acuity Scale, Emergency Severity Index, Manchester Triage Scale, and South African Triage Scale) were cataloged and compared for performance in identifying patients at risk for mortality, critical illness and hospitalization, and interrater reliability. This study was performed and reported in adherence to Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines.

RESULTS

A total of 6,160 publications were identified, with 182 meeting eligibility criteria and 50 with sufficient data for inclusion in comparative analysis. The Canadian Triage and Acuity Scale (32 studies), Emergency Severity Index (43), and Manchester Triage Scale (38) were the most frequently studied triage scales, and all demonstrated similar performance. Most studies (6 of 8) reported high sensitivity (>90%) of triage scales for identifying patients with ED mortality as high acuity at triage. However, sensitivity was low (<80%) for identification of patients who had critical illness outcomes and those who died within days of the ED visit or during the index hospitalization. Sensitivity varied by critical illness and was lower for severe sepsis (36% to 74%), pulmonary embolism (54%), and non-ST-segment elevation myocardial infarction (44% to 85%) compared with ST-segment elevation myocardial infarction (56% to 92%) and general outcomes of ICU admission (58% to 100%) and lifesaving intervention (77% to 98%). Some proportion of hospitalized patients (3% to 45%) were triaged to low acuity (level 4 to 5) in all studies. Reliability measures (κ) were variable across evaluations, with only a minority (11 of 42) reporting κ above 0.8.

CONCLUSION

We found that a substantial proportion of ED patients who die postencounter or are critically ill are not designated as high acuity at triage. Opportunity to improve interrater reliability and triage performance in identifying patients at risk of adverse outcome exists.

Validation of deep-learning-based triage and acuity score using a large national dataset

AIM

Triage is important in identifying high-risk patients amongst many less urgent patients as emergency department (ED) overcrowding has become a national crisis recently. This study aims to validate that a Deep-learning-based Triage and Acuity Score (DTAS) identifies high-risk patients more accurately than existing triage and acuity scores using a large national dataset.

METHODS

We conducted a retrospective observational cohort study using data from the Korean National Emergency Department Information System (NEDIS), which collected data on visits in real time from 151 EDs. The NEDIS data was split into derivation data (January 2014-June 2016) and validation data (July-December 2016). We also used data from the Sejong General Hospital (SGH) for external validation (January-December 2017). We predicted in-hospital mortality, critical care, and hospitalization using initial information of ED patients (age, sex, chief complaint, time from symptom onset to ED visit, arrival mode, trauma, initial vital signs and mental status as predictor variables).

RESULTS

A total of 11,656,559 patients were included in this study. The primary outcome was in-hospital mortality. The Area Under the Receiver Operating Characteristic curve (AUROC) and Area Under the Precision and Recall Curve (AUPRC) of DTAS were 0.935 and 0.264. It significantly outperformed Korean triage and acuity score (AUROC:0.785, AUPRC:0.192), modified early warning score (AUROC:0.810, AUPRC:0.116), logistic regression (AUROC:0.903, AUPRC:0.209), and random forest (AUROC:0.910, AUPRC:0.179).

CONCLUSION

Deep-learning-based Triage and Acuity Score predicted in-hospital mortality, critical care, and hospitalization more accurately than existing triages and acuity, and it was validated using a large, multicenter dataset.

Predicting hospital admission at emergency department triage using machine learning

OBJECTIVE

To predict hospital admission at the time of ED triage using patient history in addition to information collected at triage.

METHODS

This retrospective study included all adult ED visits between March 2014 and July 2017 from one academic and two community emergency rooms that resulted in either admission or discharge. A total of 972 variables were extracted per patient visit. Samples were randomly partitioned into training (80%), validation (10%), and test (10%) sets. We trained a series of nine binary classifiers using logistic regression (LR), gradient boosting (XGBoost), and deep neural networks (DNN) on three dataset types: one using only triage information, one using only patient history, and one using the full set of variables. Next, we tested the potential benefit of additional training samples by training models on increasing fractions of our data. Lastly, variables of importance were identified using information gain as a metric to create a low-dimensional model.

RESULTS

A total of 560,486 patient visits were included in the study, with an overall admission risk of 29.7%. Models trained on triage information yielded a test AUC of 0.87 for LR (95% CI 0.86-0.87), 0.87 for XGBoost (95% CI 0.87-0.88) and 0.87 for DNN (95% CI 0.87-0.88). Models trained on patient history yielded an AUC of 0.86 for LR (95% CI 0.86-0.87), 0.87 for XGBoost (95% CI 0.87-0.87) and 0.87 for DNN (95% CI 0.87-0.88). Models trained on the full set of variables yielded an AUC of 0.91 for LR (95% CI 0.91-0.91), 0.92 for XGBoost (95% CI 0.92-0.93) and 0.92 for DNN (95% CI 0.92-0.92). All algorithms reached maximum performance at 50% of the training set or less. A low-dimensional XGBoost model built on ESI level, outpatient medication counts, demographics, and hospital usage statistics yielded an AUC of 0.91 (95% CI 0.91-0.91).

CONCLUSION

Machine learning can robustly predict hospital admission using triage information and patient history. The addition of historical information improves predictive performance significantly compared to using triage information alone, highlighting the need to incorporate these variables into prediction models.

Artificial intelligence and machine learning in emergency medicine

Interest in artificial intelligence (AI) research has grown rapidly over the past few years, in part thanks to the numerous successes of modern machine learning techniques such as deep learning, the availability of large datasets and improvements in computing power. AI is proving to be increasingly applicable to healthcare and there is a growing list of tasks where algorithms have matched or surpassed physician performance. Despite the successes there remain significant concerns and challenges surrounding algorithm opacity, trust and patient data security. Notwithstanding these challenges, AI technologies will likely become increasingly integrated into emergency medicine in the coming years. This perspective presents an overview of current AI research relevant to emergency medicine.

Machine learning approaches for predicting disposition of asthma and COPD exacerbations in the ED

OBJECTIVE

The prediction of emergency department (ED) disposition at triage remains challenging. Machine learning approaches may enhance prediction. We compared the performance of several machine learning approaches for predicting two clinical outcomes (critical care and hospitalization) among ED patients with asthma or COPD exacerbation.

METHODS

Using the 2007-2015 National Hospital and Ambulatory Medical Care Survey (NHAMCS) ED data, we identified adults with asthma or COPD exacerbation. In the training set (70% random sample), using routinely-available triage data as predictors (e.g., demographics, arrival mode, vital signs, chief complaint, comorbidities), we derived four machine learning-based models: Lasso regression, random forest, boosting, and deep neural network. In the test set (the remaining 30% of sample), we compared their prediction ability against traditional logistic regression with Emergency Severity Index (ESI, reference model).

RESULTS

Of 3206 eligible ED visits, corresponding to weighted estimates of 13.9 million visits, 4% had critical care outcome and 26% had hospitalization outcome. For the critical care prediction, the best performing approach- boosting - achieved the highest discriminative ability (C-statistics 0.80 vs. 0.68), reclassification improvement (net reclassification improvement [NRI] 53%, P = 0.002), and sensitivity (0.79 vs. 0.53) over the reference model. For the hospitalization prediction, random forest provided the highest discriminative ability (C-statistics 0.83 vs. 0.64) reclassification improvement (NRI 92%, P < 0.001), and sensitivity (0.75 vs. 0.33). Results were generally consistent across the asthma and COPD subgroups.

CONCLUSIONS

Based on nationally-representative ED data, machine learning approaches improved the ability to predict disposition of patients with asthma or COPD exacerbation.

An Evolutionary Computation Approach for Optimizing Multilevel Data to Predict Patient Outcomes

Widespread adoption of electronic health records (EHR) and objectives for meaningful use have increased opportunities for data-driven predictive applications in healthcare. These decision support applications are often fueled by large-scale, heterogeneous, and multilevel (i.e., defined at hierarchical levels of specificity) patient data that challenge the development of predictive models. Our objective is to develop and evaluate an approach for optimally specifying multilevel patient data for prediction problems. We present a general evolutionary computational framework to optimally specify multilevel data to predict individual patient outcomes. We evaluate this method for both flattening (single level) and retaining the hierarchical predictor structure (multiple levels) using data collected to predict critical outcomes for emergency department patients across five populations. We find that the performance of both the flattened and hierarchical predictor structures in predicting critical outcomes for emergency department patients improve upon the baseline models for which only a single level of predictor—either more general or more specific—is used (p < 0.001). Our framework for optimizing the specificity of multilevel data improves upon more traditional single-level predictor structures and can readily be adapted to similar problems in healthcare and other domains.