Prediction of admission in pediatric emergency department with deep neural networks and triage textual data

Emergency Severity Index Version 4 and Triage of Pediatric Emergency Department Patients

Importance

Most emergency departments (EDs) across the US use the Emergency Severity Index (ESI) to predict acuity and resource needs. A comprehensive assessment of ESI accuracy among pediatric patients is lacking.

Objective

To assess the frequency of mistriage using ESI (version 4) among pediatric ED visits using automated measures of mistriage and identify characteristics associated with mistriage.

Design, Setting, and Participants

This cohort study used operational measures for each ESI level to classify encounters as undertriaged, overtriaged, or correctly triaged to assess the accuracy of the ESI and identify characteristics of mistriage. Participants were pediatric patients at 21 EDs within Kaiser Permanente Northern California from January 1, 2016, to December 31, 2020. During that time, version 4 of the ESI was in use by these EDs. Visits with missing ESI, incomplete ED time variables, patients transferred from another ED, and those who left against medical advice or without being seen were excluded. Data were analyzed between January 2022 and June 2023.

Exposures

Assigned ESI level.

Main Outcomes and Measures

Rates of undertriage and overtriage by assigned ESI level based on mistriage algorithm, patient and visit characteristics associated with undertriage and overtriage.

Results

This study included 1 016 816 pediatric ED visits; the mean (SD) age of patients was 7.3 (5.6) years, 479 610 (47.2%) were female, and 537 206 (52.8%) were male. Correct triage occurred in 346 918 visits (34.1%; 95% CI, 34.0%-34.2%), while overtriage and undertriage occurred in 594 485 visits (58.5%; 95% CI, 58.4%-58.6%) and 75 413 visits (7.4%; 95% CI, 7.4%-7.5%), respectively. In adjusted analyses, undertriage was more common among children at least 6 years old compared with those younger 6 years; male patients compared with female patients; patients with Asian, Black, or Hispanic or other races or ethnicities compared with White patients; patients with comorbid illnesses compared with those without; and patients who arrived by ambulance compared with nonambulance patients.

Conclusions and Relevance

This multicenter retrospective study found that mistriage with ESI version 4 was common in pediatric ED visits. There is an opportunity to improve pediatric ED triage, both in early identification of critically ill patients (limit undertriage) and in more accurate identification of low-acuity patients with low resource needs (limit overtriage). Future research should include assessments based on version 5 of the ESI, which was released after this study was completed.

The effects of applying artificial intelligence to triage in the emergency department: A systematic review of prospective studies

INTRODUCTION

Accurate and rapid triage can reduce undertriage and overtriage, which may improve emergency department flow. This study aimed to identify the effects of a prospective study applying artificial intelligence-based triage in the clinical field.

DESIGN

Systematic review of prospective studies.

METHODS

CINAHL, Cochrane, Embase, PubMed, ProQuest, KISS, and RISS were searched from March 9 to April 18, 2023. All the data were screened independently by three researchers. The review included prospective studies that measured outcomes related to AI-based triage. Three researchers extracted data and independently assessed the study's quality using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) protocol.

RESULTS

Of 1633 studies, seven met the inclusion criteria for this review. Most studies applied machine learning to triage, and only one was based on fuzzy logic. All studies, except one, utilized a five-level triage classification system. Regarding model performance, the feed-forward neural network achieved a precision of 33% in the level 1 classification, whereas the fuzzy clip model achieved a specificity and sensitivity of 99%. The accuracy of the model's triage prediction ranged from 80.5% to 99.1%. Other outcomes included time reduction, overtriage and undertriage checks, mistriage factors, and patient care and prognosis outcomes.

CONCLUSION

Triage nurses in the emergency department can use artificial intelligence as a supportive means for triage. Ultimately, we hope to be a resource that can reduce undertriage and positively affect patient health.

PROTOCOL REGISTRATION

We have registered our review in PROSPERO (registration number: CRD 42023415232).

Using a clinical narrative-aware pre-trained language model for predicting emergency department patient disposition and unscheduled return visits

The increasing prevalence of overcrowding in Emergency Departments (EDs) threatens the effective delivery of urgent healthcare. Mitigation strategies include the deployment of monitoring systems capable of tracking and managing patient disposition to facilitate appropriate and timely care, which subsequently reduces patient revisits, optimizes resource allocation, and enhances patient outcomes. This study used ∼ 250,000 emergency department visit records from Taipei Medical University-Shuang Ho Hospital to develop a natural language processing model using BlueBERT, a biomedical domain-specific pre-trained language model, to predict patient disposition status and unplanned readmissions. Data preprocessing and the integration of both structured and unstructured data were central to our approach. Compared to other models, BlueBERT outperformed due to its pre-training on a diverse range of medical literature, enabling it to better comprehend the specialized terminology, relationships, and context present in ED data. We found that translating Chinese-English clinical narratives into English and textualizing numerical data into categorical representations significantly improved the prediction of patient disposition (AUROC = 0.9014) and 72-hour unscheduled return visits (AUROC = 0.6475). The study concludes that the BlueBERT-based model demonstrated superior prediction capabilities, surpassing the performance of prior patient disposition predictive models, thus offering promising applications in the realm of ED clinical practice.

Drinking from the Holy Grail-Does a Perfect Triage System Exist? And Where to Look for It?

The Emergency Department (ED) is a facility meant to treat patients in need of medical assistance. The choice of triage system hugely impactsed the organization of any given ED and it is important to analyze them for their effectiveness. The goal of this review is to briefly describe selected triage systems in an attempt to find the perfect one. Papers published in PubMed from 1990 to 2022 were reviewed. The following terms were used for comparison: "ED" and "triage system". The papers contained data on the design and function of the triage system, its validation, and its performance. After studies comparing the distinct means of patient selection were reviewed, they were meant to be classified as either flawed or non-ideal. The validity of all the comparable segregation systems was similar. A possible solution would be to search for a new, measurable parameter for a more accurate risk estimation, which could be a game changer in terms of triage assessment. The dynamic development of artificial intelligence (AI) technologies has recently been observed. The authors of this study believe that the future segregation system should be a combination of the experience and intuition of trained healthcare professionals and modern technology (artificial intelligence).

An ensemble model for predicting dispositions of emergency department patients

OBJECTIVE

The healthcare challenge driven by an aging population and rising demand is one of the most pressing issues leading to emergency department (ED) overcrowding. An emerging solution lies in machine learning's potential to predict ED dispositions, thus leading to promising substantial benefits. This study's objective is to create a predictive model for ED patient dispositions by employing ensemble learning. It harnesses diverse data types, including structured and unstructured information gathered during ED visits to address the evolving needs of localized healthcare systems.

METHODS

In this cross-sectional study, 80,073 ED patient records were amassed from a major southern Taiwan hospital in 2018-2019. An ensemble model incorporated structured (demographics, vital signs) and pre-processed unstructured data (chief complaints, preliminary diagnoses) using bag-of-words (BOW) and term frequency-inverse document frequency (TF-IDF). Two random forest base-learners for structured and unstructured data were employed and then complemented by a multi-layer perceptron meta-learner.

RESULTS

The ensemble model demonstrates strong predictive performance for ED dispositions, achieving an area under the receiver operating characteristic curve of 0.94. The models based on unstructured data encoded with BOW and TF-IDF yield similar performance results. Among the structured features, the top five most crucial factors are age, pulse rate, systolic blood pressure, temperature, and acuity level. In contrast, the top five most important unstructured features are pneumonia, fracture, failure, suspect, and sepsis.

CONCLUSIONS

Findings indicate that utilizing ensemble learning with a blend of structured and unstructured data proves to be a predictive method for determining ED dispositions.

Prediction of hospitalization and waiting time within 24 hours of emergency department patients with unstructured text data

Overcrowding of emergency departments is a global concern, leading to numerous negative consequences. This study aimed to develop a useful and inexpensive tool derived from electronic medical records that supports clinical decision-making and can be easily utilized by emergency department physicians. We presented machine learning models that predicted the likelihood of hospitalizations within 24 hours and estimated waiting times. Moreover, we revealed the enhanced performance of these machine learning models compared to existing models by incorporating unstructured text data. Among several evaluated models, the extreme gradient boosting model that incorporated text data yielded the best performance. This model achieved an area under the receiver operating characteristic curve score of 0.922 and an area under the precision-recall curve score of 0.687. The mean absolute error revealed a difference of approximately 3 hours. Using this model, we classified the probability of patients not being admitted within 24 hours as Low, Medium, or High and identified important variables influencing this classification through explainable artificial intelligence. The model results are readily displayed on an electronic dashboard to support the decision-making of emergency department physicians and alleviate overcrowding, thereby resulting in socioeconomic benefits for medical facilities.

Applications of natural language processing at emergency department triage: A narrative review

INTRODUCTION

Natural language processing (NLP) uses various computational methods to analyse and understand human language, and has been applied to data acquired at Emergency Department (ED) triage to predict various outcomes. The objective of this scoping review is to evaluate how NLP has been applied to data acquired at ED triage, assess if NLP based models outperform humans or current risk stratification techniques when predicting outcomes, and assess if incorporating free-text improve predictive performance of models when compared to predictive models that use only structured data.

METHODS

All English language peer-reviewed research that applied an NLP technique to free-text obtained at ED triage was eligible for inclusion. We excluded studies focusing solely on disease surveillance, and studies that used information obtained after triage. We searched the electronic databases MEDLINE, Embase, Cochrane Database of Systematic Reviews, Web of Science, and Scopus for medical subject headings and text keywords related to NLP and triage. Databases were last searched on 01/01/2022. Risk of bias in studies was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST). Due to the high level of heterogeneity between studies and high risk of bias, a metanalysis was not conducted. Instead, a narrative synthesis is provided.

RESULTS

In total, 3730 studies were screened, and 20 studies were included. The population size varied greatly between studies ranging from 1.8 million patients to 598 triage notes. The most common outcomes assessed were prediction of triage score, prediction of admission, and prediction of critical illness. NLP models achieved high accuracy in predicting need for admission, triage score, critical illness, and mapping free-text chief complaints to structured fields. Incorporating both structured data and free-text data improved results when compared to models that used only structured data. However, the majority of studies (80%) were assessed to have a high risk of bias, and only one study reported the deployment of an NLP model into clinical practice.

CONCLUSION

Unstructured free-text triage notes have been used by NLP models to predict clinically relevant outcomes. However, the majority of studies have a high risk of bias, most research is retrospective, and there are few examples of implementation into clinical practice. Future work is needed to prospectively assess if applying NLP to data acquired at ED triage improves ED outcomes when compared to usual clinical practice.

Intelligent Framework for Early Detection of Severe Pediatric Diseases from Mild Symptoms

Children's health is one of the most significant fields in medicine. Most diseases that result in children's death or long-term morbidity are caused by preventable and treatable etiologies, and they appear in the child at the early stages as mild symptoms. This research aims to develop a machine learning (ML) framework to detect the severity of disease in children. The proposed framework helps in discriminating children's urgent/severe conditions and notifying parents whether a child needs to visit the emergency room immediately or not. The model considers several variables to detect the severity of cases, which are the symptoms, risk factors (e.g., age), and the child's medical history. The framework is implemented by using nine ML methods. The results achieved show the high performance of the proposed framework in identifying serious pediatric diseases, where decision tree and random forest outperformed the other methods with an accuracy rate of 94%. This shows the reliability of the proposed framework to be used as a pediatric decision-making system for detecting serious pediatric illnesses. The results are promising when compared to recent state-of-the-art studies. The main contribution of this research is to propose a framework that is viable for use by parents when their child suffers from any commonly developed symptoms.

Automated analysis of unstructured clinical assessments improves emergency department triage performance: A retrospective deep learning analysis

Objectives

Efficient and accurate emergency department (ED) triage is critical to prioritize the sickest patients and manage department flow. We explored the use of electronic health record data and advanced predictive analytics to improve triage performance.

Methods

Using a data set of over 5 million ED encounters of patients 18 years and older across 21 EDs from 2016 to 2020, we derived triage models using deep learning to predict 2 outcomes: hospitalization (primary outcome) and fast-track eligibility (exploratory outcome), defined as ED discharge with <2 resource types used (eg, laboratory or imaging studies) and no critical events (eg, resuscitative medications use or intensive care unit [ICU] admission). We report area under the receiver operator characteristic curve (AUC) and 95% confidence intervals (CI) for models using (1) triage variables alone (demographics and vital signs), (2) triage nurse clinical assessment alone (unstructured notes), and (3) triage variables plus clinical assessment for each prediction target.

Results

We found 12.7% of patients were hospitalized (n = 673,659) and 37.0% were fast-track eligible (n = 1,966,615). The AUC was lowest for models using triage variables alone: AUC 0.77 (95% CI 0.77-0.78) and 0.70 (95% CI 0.70-0.71) for hospitalization and fast-track eligibility, respectively, and highest for models incorporating clinical assessment with triage variables for both hospitalization and fast-track eligibility: AUC 0.87 (95% CI 0.87-0.87) for both prediction targets.

Conclusion

Our findings highlight the potential to use advanced predictive analytics to accurately predict key ED triage outcomes. Predictive accuracy was optimized when clinical assessments were added to models using simple structured variables alone.

Modern Paediatric Emergency Department: Potential Improvements in Light of New Evidence

The increasing attendance of paediatric emergency departments has become a serious health issue. To reduce an elevated burden of medical errors, inevitably caused by a high level of stress exerted on emergency physicians, we propose potential areas for improvement in regular paediatric emergency departments. In an effort to guarantee the demanded quality of care to all incoming patients, the workflow in paediatric emergency departments should be sufficiently optimised. The key component remains to implement one of the validated paediatric triage systems upon the patient's arrival at the emergency department and fast-tracking patients with a low level of risk according to the triage system. To ensure the patient's safety, emergency physicians should follow issued guidelines. Cognitive aids, such as well-designed checklists, posters or flow charts, generally improve physicians' adherence to guidelines and should be available in every paediatric emergency department. To sharpen diagnostic accuracy, the use of ultrasound in a paediatric emergency department, according to ultrasound protocols, should be targeted to answer specific clinical questions. Combining all mentioned improvements might reduce the number of errors linked to overcrowding. The review serves not only as a blueprint for modernising paediatric emergency departments but also as a bin of useful literature which can be suitable in the paediatric emergency field.

Machine Learning-Based Prediction of Hospital Admission Among Children in an Emergency Care Center

OBJECTIVES

Machine learning-based prediction of hospital admissions may have the potential to optimize patient disposition and improve clinical outcomes by minimizing both undertriage and overtriage in crowded emergency care. We developed and validated the predictive abilities of machine learning-based predictions of hospital admissions in a pediatric emergency care center.

METHODS

A prognostic study was performed using retrospectively collected data of children younger than 16 years who visited a single pediatric emergency care center in Osaka, Japan, between August 1, 2016, and October 15, 2019. Generally, the center treated walk-in children and did not treat trauma injuries. The main outcome was hospital admission as determined by the physician. The 83 potential predictors available at presentation were selected from the following categories: demographic characteristics, triage level, physiological parameters, and symptoms. To identify predictive abilities for hospital admission, maximize the area under the precision-recall curve, and address imbalanced outcome classes, we developed the following models for the preperiod training cohort (67% of the samples) and also used them in the 1-year postperiod validation cohort (33% of the samples): (1) logistic regression, (2) support vector machine, (3) random forest, and (4) extreme gradient boosting.

RESULTS

Among 88,283 children who were enrolled, the median age was 3.9 years, with 47,931 (54.3%) boys and 1985 (2.2%) requiring hospital admission. Among the models, extreme gradient boosting achieved the highest predictive abilities (eg, area under the precision-recall curve, 0.26; 95% confidence interval, 0.25-0.27; area under the receiver operating characteristic curve, 0.86; 95% confidence interval, 0.84-0.88; sensitivity, 0.77; and specificity, 0.82). With an optimal threshold, the positive and negative likelihood ratios were 4.22, and 0.28, respectively.

CONCLUSIONS

Machine learning-based prediction of hospital admissions may support physicians' decision-making for hospital admissions. However, further improvements are required before implementing these models in real clinical settings.

The Use and Structure of Emergency Nurses' Triage Narrative Data: Scoping Review

BACKGROUND

Emergency departments use triage to ensure that patients with the highest level of acuity receive care quickly and safely. Triage is typically a nursing process that is documented as structured and unstructured (free text) data. Free-text triage narratives have been studied for specific conditions but never reviewed in a comprehensive manner.

OBJECTIVE

The objective of this paper was to identify and map the academic literature that examines triage narratives. The paper described the types of research conducted, identified gaps in the research, and determined where additional review may be warranted.

METHODS

We conducted a scoping review of unstructured triage narratives. We mapped the literature, described the use of triage narrative data, examined the information available on the form and structure of narratives, highlighted similarities among publications, and identified opportunities for future research.

RESULTS

We screened 18,074 studies published between 1990 and 2022 in CINAHL, MEDLINE, Embase, Cochrane, and ProQuest Central. We identified 0.53% (96/18,074) of studies that directly examined the use of triage nurses' narratives. More than 12 million visits were made to 2438 emergency departments included in the review. In total, 82% (79/96) of these studies were conducted in the United States (43/96, 45%), Australia (31/96, 32%), or Canada (5/96, 5%). Triage narratives were used for research and case identification, as input variables for predictive modeling, and for quality improvement. Overall, 31% (30/96) of the studies offered a description of the triage narrative, including a list of the keywords used (27/96, 28%) or more fulsome descriptions (such as word counts, character counts, abbreviation, etc; 7/96, 7%). We found limited use of reporting guidelines (8/96, 8%).

CONCLUSIONS

The breadth of the identified studies suggests that there is widespread routine collection and research use of triage narrative data. Despite the use of triage narratives as a source of data in studies, the narratives and nurses who generate them are poorly described in the literature, and data reporting is inconsistent. Additional research is needed to describe the structure of triage narratives, determine the best use of triage narratives, and improve the consistent use of triage-specific data reporting guidelines.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1136/bmjopen-2021-055132.

Using the Delphi method to establish pediatric emergency triage criteria in a grade A tertiary women's and children's hospital in China

BACKGROUND

We aimed to establish simplified and quantifiable triage criteria in pediatric emergency care, improving the efficiency of pediatric emergency triage and ensuring patient safety.

METHODS

We preliminarily determined the pediatric emergency triage criteria with references to pediatric emergency department characteristics and internationally recognized triage tools after literature review and discussion. The final determination of the triage criteria was reached after two rounds of Delphi surveys completed by18 experts from 3 hospitals in China.

RESULTS

Both round 1 and round 2 surveys had a 100% response rate. The overall expert authority coefficient in the two rounds of surveys was 0.872. The experts had 100% enthusiasm for participating in the surveys. Kendall's coefficients of concordance for conditions/symptoms in patients triaged to level 1, 2, 3, and 4 were 0.149, 0.193, 0.102, and 0.266, respectively. All p-values were less than 0.05. The coefficients of variation in conditions/symptoms, vital signs, and the Pediatric Early Warning Score (PEWS) ranged between 0.00 and 0.205, meeting the inclusion criteria. The pediatric emergency triage criteria containing conditions/symptoms, vital signs, PEWS scores, and other 4 level 1 indicators, 51 level 2 indicators and 23 level 3 indicators were built. The maximum waiting time to treatment for the patients triaged to level 1, 2, 3, and 4 was immediate, within 10 min, within 30 min, and within 240 min, respectively.

CONCLUSION

The pediatric emergency triage criteria established in this study was scientific and reliable. It can be used to quickly identify the patients requiring urgent and immediate care, thereby ensuring the priorities for the care of critically ill patients.

COVID-19 pneumonia level detection using deep learning algorithm and transfer learning

The first COVID-19 confirmed case was reported in Wuhan, China, and spread across the globe with an unprecedented impact on humanity. Since this pandemic requires pervasive diagnosis, developing smart, fast, and efficient detection techniques is significant. To this end, we have developed an Artificial Intelligence engine to classify the lung inflammation level (mild, progressive, severe stage) of the COVID-19 confirmed patient. In particular, the developed model consists of two phases; in the first phase, we calculate the volume and density of lesions and opacities of the CT scan images of the confirmed COVID-19 patient using Morphological approaches. The second phase classifies the pneumonia level of the confirmed COVID-19 patient. We use a modified Convolution Neural Network (CNN) and k-Nearest Neighbor; we also compared the results of both models to the other classification algorithms to precisely classify lung inflammation. The experiments show that the CNN model can provide testing accuracy up to 95.65% compared with exiting classification techniques. The proposed system in this work can be applied efficiently to CT scan and X-ray image datasets. Also, in this work, the Transfer Learning technique has been used to train the pre-trained modified CNN model on a smaller dataset than the original dataset; the modified CNN achieved 92.80% of testing accuracy for detecting pneumonia on chest X-ray images for the relatively extensive dataset.

Effect of Applying a Real-Time Medical Record Input Assistance System With Voice Artificial Intelligence on Triage Task Performance in the Emergency Department: Prospective Interventional Study

BACKGROUND

Natural language processing has been established as an important tool when using unstructured text data; however, most studies in the medical field have been limited to a retrospective analysis of text entered manually by humans. Little research has focused on applying natural language processing to the conversion of raw voice data generated in the clinical field into text using speech-to-text algorithms.

OBJECTIVE

In this study, we investigated the promptness and reliability of a real-time medical record input assistance system with voice artificial intelligence (RMIS-AI) and compared it to the manual method for triage tasks in the emergency department.

METHODS

From June 4, 2021, to September 12, 2021, RMIS-AI, using a machine learning engine trained with 1717 triage cases over 6 months, was prospectively applied in clinical practice in a triage unit. We analyzed a total of 1063 triage tasks performed by 19 triage nurses who agreed to participate. The primary outcome was the time for participants to perform the triage task.

RESULTS

The median time for participants to perform the triage task was 204 (IQR 155, 277) seconds by RMIS-AI and 231 (IQR 180, 313) seconds using manual method; this difference was statistically significant (P<.001). Most variables required for entry in the triage note showed a higher record completion rate by the manual method, but in the recording of additional chief concerns and past medical history, RMIS-AI showed a higher record completion rate than the manual method. Categorical variables entered by RMIS-AI showed less accuracy compared with continuous variables, such as vital signs.

CONCLUSIONS

RMIS-AI improves the promptness in performing triage tasks as compared to using the manual input method. However, to make it a reliable alternative to the conventional method, technical supplementation and additional research should be pursued.

Imbalanced prediction of emergency department admission using natural language processing and deep neural network

The emergency department (ED) plays a very significant role in the hospital. Owing to the rising number of ED visits, medical service points, and ED market, overcrowding of EDs has become serious worldwide. Overcrowding has long been recognized as a vital issue that increases the risk to patients and negative emotions of medical personnel and impacts hospital cost management. For the past years, many researchers have been applying artificial intelligence to reduce crowding situations in the ED. Nevertheless, the datasets in ED hospital admission are naturally inherent with the high-class imbalance in the real world. Previous studies have not considered the imbalance of the datasets, particularly addressing the imbalance. This study purposes to develop a natural language processing model of a deep neural network with an attention mechanism to solve the imbalanced problem in ED admission. The proposed framework is used for predicting hospital admission so that the hospitals can arrange beds early and solve the problem of congestion in the ED. Furthermore, the study compares a variety of methods and obtains the best composition that has the best performance for forecasting hospitalization in ED. The study used the data from a specific hospital in Taiwan as an empirical study. The experimental result demonstrates that almost all imbalanced methods can improve the model's performance. In addition, the natural language processing model of Bi-directional Long Short-Term Memory with attention mechanism has the best results in all-natural language processing methods.

Assessing the Generalizability of a Clinical Machine Learning Model Across Multiple Emergency Departments

Objective

To assess the generalizability of a clinical machine learning algorithm across multiple emergency departments (EDs).

Patients and Methods

We obtained data on all ED visits at our health care system's largest ED from May 5, 2018, to December 31, 2019. We also obtained data from 3 satellite EDs and 1 distant-hub ED from May 1, 2018, to December 31, 2018. A gradient-boosted machine model was trained on pooled data from the included EDs. To prevent the effect of differing training set sizes, the data were randomly downsampled to match those of our smallest ED. A second model was trained on this downsampled, pooled data. The model's performance was compared using area under the receiver operating characteristic (AUC). Finally, site-specific models were trained and tested across all the sites, and the importance of features was examined to understand the reasons for differing generalizability.

Results

The training data sets contained 1918-64,161 ED visits. The AUC for the pooled model ranged from 0.84 to 0.94 across the sites; the performance decreased slightly when Ns were downsampled to match those of our smallest ED site. When site-specific models were trained and tested across all the sites, the AUCs ranged more widely from 0.71 to 0.93. Within a single ED site, the performance of the 5 site-specific models was most variable for our largest and smallest EDs. Finally, when the importance of features was examined, several features were common to all site-specific models; however, the weight of these features differed.

Conclusion

A machine learning model for predicting hospital admission from the ED will generalize fairly well within the health care system but will still have significant differences in AUC performance across sites because of site-specific factors.

SymptomID: A Framework for Rapid Symptom Identification in Pandemics Using News Reports

The ability to quickly learn fundamentals about a new infectious disease, such as how it is transmitted, the incubation period, and related symptoms, is crucial in any novel pandemic. For instance, rapid identification of symptoms can enable interventions for dampening the spread of the disease. Traditionally, symptoms are learned from research publications associated with clinical studies. However, clinical studies are often slow and time intensive, and hence delays can have dire consequences in a rapidly spreading pandemic like we have seen with COVID-19. In this article, we introduce SymptomID, a modular artificial intelligence–based framework for rapid identification of symptoms associated with novel pandemics using publicly available news reports. SymptomID is built using the state-of-the-art natural language processing model (Bidirectional Encoder Representations for Transformers) to extract symptoms from publicly available news reports and cluster-related symptoms together to remove redundancy. Our proposed framework requires minimal training data, because it builds on a pre-trained language model. In this study, we present a case study of SymptomID using news articles about the current COVID-19 pandemic. Our COVID-19 symptom extraction module, trained on 225 articles, achieves an F1 score of over 0.8. SymptomID can correctly identify well-established symptoms (e.g., “fever” and “cough”) and less-prevalent symptoms (e.g., “rashes,” “hair loss,” “brain fog”) associated with the novel coronavirus. We believe this framework can be extended and easily adapted in future pandemics to quickly learn relevant insights that are fundamental for understanding and combating a new infectious disease.

Machine learning methods applied to triage in emergency services: A systematic review

BACKGROUND

In emergency services is important to accurately assess and classify symptoms, which may be improved with the help of technology. One mechanism that could help and improve predictions from health records or patient flow is machine learning (ML).

AIM

To analyse the effectiveness of ML systems in triage for making predictions at the emergency department in comparison with other triage scales/scores.

METHODS

Following the PRISMA recommendations, a systematic review was conducted using CINAHL, Cochrane, Cuiden, Medline and Scopus databases with the search equation "Machine learning AND triage AND emergency".

RESULTS

Eleven studies were identified. The studies show that the use of ML methods consistently predict important outcomes like mortality, critical care outcomes and admission, and the need for hospitalization in comparison with scales like Emergency Severity Index or others. Among the ML models considered, XGBoost and Deep Neural Networks obtained the highest levels of prediction accuracy, while Logistic Regression performed obtained the worst values.

CONCLUSIONS

Machine learning methods can be a good instrument for helping triage process with the prediction of important emergency variables like mortality or the need for critical care or hospitalization.

The prediction of hospital length of stay using unstructured data

Objective

This study aimed to assess the performance improvement for machine learning-based hospital length of stay (LOS) predictions when clinical signs written in text are accounted for and compared to the traditional approach of solely considering structured information such as age, gender and major ICD diagnosis.

Methods

This study was an observational retrospective cohort study and analyzed patient stays admitted between 1 January to 24 September 2019. For each stay, a patient was admitted through the Emergency Department (ED) and stayed for more than two days in the subsequent service. LOS was predicted using two random forest models. The first included unstructured text extracted from electronic health records (EHRs). A word-embedding algorithm based on UMLS terminology with exact matching restricted to patient-centric affirmation sentences was used to assess the EHR data. The second model was primarily based on structured data in the form of diagnoses coded from the International Classification of Disease 10th Edition (ICD-10) and triage codes (CCMU/GEMSA classifications). Variables common to both models were: age, gender, zip/postal code, LOS in the ED, recent visit flag, assigned patient ward after the ED stay and short-term ED activity. Models were trained on 80% of data and performance was evaluated by accuracy on the remaining 20% test data.

Results

The model using unstructured data had a 75.0% accuracy compared to 74.1% for the model containing structured data. The two models produced a similar prediction in 86.6% of cases. In a secondary analysis restricted to intensive care patients, the accuracy of both models was also similar (76.3% vs 75.0%).

Conclusions

LOS prediction using unstructured data had similar accuracy to using structured data and can be considered of use to accurately model LOS.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-021-01722-4.

A Proposed Framework for Machine Learning-Aided Triage in Public Specialty Ophthalmology Clinics in Hong Kong

The public specialty ophthalmic clinics in Hong Kong, under the Hospital Authority, receive tens of thousands of referrals each year. Triaging these referrals incurs a significant workload for practitioners and the other clinical duties. It is well-established that Hong Kong is currently facing a shortage of healthcare workers. Thus a more efficient system in triaging will not only free up resources for better use but also improve the satisfaction of both practitioners and patients. Machine learning (ML) has been shown to improve the efficiency of various medical workflows, including triaging, by both reducing the workload and increasing accuracy in some cases. Despite a myriad of studies on medical artificial intelligence, there is no specific framework for a triaging algorithm in ophthalmology clinics. This study proposes a general framework for developing, deploying and evaluating an ML-based triaging algorithm in a clinical setting. Through literature review, this study identifies good practices in various facets of developing such a network and protocols for maintenance and evaluation of the impact concerning clinical utility and external validity out of the laboratory. We hope this framework, albeit not exhaustive, can act as a foundation to accelerate future pilot studies and deployments.

Advanced natural language processing technique to predict patient disposition based on emergency triage notes

OBJECTIVE

To demonstrate the potential of machine learning and capability of natural language processing (NLP) to predict disposition of patients based on triage notes in the ED.

METHODS

A retrospective cohort of ED triage notes from St Vincent's Hospital (Melbourne) was used to develop a deep-learning algorithm that predicts patient disposition. Bidirectional Encoder Representations from Transformers, a recent language representation model developed by Google, was utilised for NLP. Eighty percent of the dataset was used for training the model and 20% was used to test the algorithm performance. Ktrain library, a wrapper for TensorFlow Keras, was employed to develop the model.

RESULTS

The accuracy of the algorithm was 83% and the area under the curve was 0.88. Sensitivity, specificity, precision and F1-score of the algorithm were 72%, 86%, 56% and 63%, respectively.

CONCLUSION

Machine learning and NLP can be together applied to the ED triage note to predict patient disposition with a high level of accuracy. The algorithm can potentially assist ED clinicians in early identification of patients requiring admission by mitigating the cognitive load, thus optimises resource allocation in EDs.

Development and Validation of Machine Learning Models to Predict Admission From Emergency Department to Inpatient and Intensive Care Units

STUDY OBJECTIVE

This study aimed to develop and validate 2 machine learning models that use historical and current-visit patient data from electronic health records to predict the probability of patient admission to either an inpatient unit or ICU at each hour (up to 24 hours) of an emergency department (ED) encounter. The secondary goal was to provide a framework for the operational implementation of these machine learning models.

METHODS

Data were curated from 468,167 adult patient encounters in 3 EDs (1 academic and 2 community-based EDs) of a large academic health system from August 1, 2015, to October 31, 2018. The models were validated using encounter data from January 1, 2019, to December 31, 2019. An operational user dashboard was developed, and the models were run on real-time encounter data.

RESULTS

For the intermediate admission model, the area under the receiver operating characteristic curve was 0.873 and the area under the precision-recall curve was 0.636. For the ICU admission model, the area under the receiver operating characteristic curve was 0.951 and the area under the precision-recall curve was 0.461. The models had similar performance in both the academic- and community-based settings as well as across the 2019 and real-time encounter data.

CONCLUSION

Machine learning models were developed to accurately make predictions regarding the probability of inpatient or ICU admission throughout the entire duration of a patient's encounter in ED and not just at the time of triage. These models remained accurate for a patient cohort beyond the time period of the initial training data and were integrated to run on live electronic health record data, with similar performance.

Criticality index conducted in pediatric emergency department triage

OBJECTIVE

To develop and analyze the performance of a machine learning model capable of predicting the disposition of patients presenting to a pediatric emergency department (ED) based on triage assessment and historical information mined from electronic health records.

METHODS

We retrospectively reviewed data from 585,142 ED visits at a pediatric quaternary care institution between 2013 and 2020. An extreme gradient boosting machine learning model was trained on a randomly selected training data set (50%) to stratify patients into 3 classes: (1) high criticality (patients requiring intensive care unit [ICU] care within 4 h of hospital admission, patients who died within 4 h of admission, and patients who died in the ED); (2) moderate criticality (patients requiring hospitalization without the need for ICU care); and (3) low criticality (patients discharged home). Variables considered during model development included triage vital signs, aspects of triage nursing assessment, demographics, and historical information (diagnoses, medication use, and healthcare utilization). Historical factors were limited to the 6 months preceding the index ED visit. The model was tested on a previously withheld test data set (40%), and its performance analyzed.

RESULTS

The distribution of criticality among high, moderate, and low was 1.5%, 7.1%, and 91.4%, respectively. The one-versus-all area under the receiver operating characteristic (AUROC) curve for high and moderate criticality was 0.982 (95% CI 0.980, 0.983) and 0.968 (0.967, 0.969). The multi-class macro average AUROC and area under the receiver operating characteristic curve were 0.976 and 0.754. The features most integral to model performance included history of intravenous medications, capillary refill, emergency severity index level, history of hospitalization, use of a supplemental oxygen device, age, and history of admission to the ICU.

CONCLUSION

Pediatric ED disposition can be accurately predicted using information available at triage, providing an opportunity to improve quality of care and patient outcomes.

Machine learning in patient flow: a review

This work is a review of the ways in which machine learning has been used in order to plan, improve or aid the problem of moving patients through healthcare services. We decompose the patient flow problem into four subcategories: prediction of demand on a healthcare institution, prediction of the demand and resource required to transfer patients from the emergency department to the hospital, prediction of potential resource required for the treatment and movement of inpatients and prediction of length-of-stay and discharge timing. We argue that there are benefits to both approaches of considering the healthcare institution as a whole as well as the patient by patient case and that ideally a combination of these would be best for improving patient flow through hospitals. We also argue that it is essential for there to be a shared dataset that will allow researchers to benchmark their algorithms on and thereby allow future researchers to build on that which has already been done. We conclude that machine learning for the improvement of patient flow is still a young field with very few papers tailor-making machine learning methods for the problem being considered. Future works should consider the need to transfer algorithms trained on a dataset to multiple hospitals and allowing for dynamic algorithms which will allow real-time decision-making to help clinical staff on the shop floor.

Leveraging Data Science to Combat COVID-19: A Comprehensive Review

COVID-19, an infectious disease caused by the SARS-CoV-2 virus, was declared a pandemic by the World Health Organisation (WHO) in March 2020. By mid-August 2020, more than 21 million people have tested positive worldwide. Infections have been growing rapidly and tremendous efforts are being made to fight the disease. In this paper, we attempt to systematise the various COVID-19 research activities leveraging data science, where we define data science broadly to encompass the various methods and tools-including those from artificial intelligence (AI), machine learning (ML), statistics, modeling, simulation, and data visualization-that can be used to store, process, and extract insights from data. In addition to reviewing the rapidly growing body of recent research, we survey public datasets and repositories that can be used for further work to track COVID-19 spread and mitigation strategies. As part of this, we present a bibliometric analysis of the papers produced in this short span of time. Finally, building on these insights, we highlight common challenges and pitfalls observed across the surveyed works. We also created a live resource repository at https://github.com/Data-Science-and-COVID-19/Leveraging-Data-Science-To-Combat-COVID-19-A-Comprehensive-Review that we intend to keep updated with the latest resources including new papers and datasets.