Machine learning for real-time aggregated prediction of hospital admission for emergency patients

A pre-trained language model for emergency department intervention prediction using routine physiological data and clinical narratives

INTRODUCTION

The urgency and complexity of emergency room (ER) settings require precise and swift decision-making processes for patient care. Ensuring the timely execution of critical examinations and interventions is vital for reducing diagnostic errors, but the literature highlights a need for innovative approaches to optimize diagnostic accuracy and patient outcomes. In response, our study endeavors to create predictive models for timely examinations and interventions by leveraging the patient's symptoms and vital signs recorded during triage, and in so doing, augment traditional diagnostic methodologies.

METHODS

Focusing on four key areas-medication dispensing, vital interventions, laboratory testing, and emergency radiology exams, the study employed Natural Language Processing (NLP) and seven advanced machine learning techniques. The research was centered around the innovative use of BioClinicalBERT, a state-of-the-art NLP framework.

RESULTS

BioClinicalBERT emerged as the superior model, outperforming others in predictive accuracy. The integration of physiological data with patient narrative symptoms demonstrated greater effectiveness compared to models based solely on textual data. The robustness of our approach was confirmed by an Area Under the Receiver Operating Characteristic curve (AUROC) score of 0.9.

CONCLUSION

The findings of our study underscore the feasibility of establishing a decision support system for emergency patients, targeting timely interventions and examinations based on a nuanced analysis of symptoms. By using an advanced natural language processing technique, our approach shows promise for enhancing diagnostic accuracy. However, the current model is not yet fully mature for direct implementation into daily clinical practice. Recognizing the imperative nature of precision in the ER environment, future research endeavors must focus on refining and expanding predictive models to include detailed timely examinations and interventions. Although the progress achieved in this study represents an encouraging step towards a more innovative and technology-driven paradigm in emergency care, full clinical integration warrants further exploration and validation.

Improving Triage Accuracy in Prehospital Emergency Telemedicine: Scoping Review of Machine Learning-Enhanced Approaches

BACKGROUND

Prehospital telemedicine triage systems combined with machine learning (ML) methods have the potential to improve triage accuracy and safely redirect low-acuity patients from attending the emergency department. However, research in prehospital settings is limited but needed; emergency department overcrowding and adverse patient outcomes are increasingly common.

OBJECTIVE

In this scoping review, we sought to characterize the existing methods for ML-enhanced telemedicine emergency triage. In order to support future research, we aimed to delineate what data sources, predictors, labels, ML models, and performance metrics were used, and in which telemedicine triage systems these methods were applied.

METHODS

A scoping review was conducted, querying multiple databases (MEDLINE, PubMed, Scopus, and IEEE Xplore) through February 24, 2023, to identify potential ML-enhanced methods, and for those eligible, relevant study characteristics were extracted, including prehospital triage setting, types of predictors, ground truth labeling method, ML models used, and performance metrics. Inclusion criteria were restricted to the triage of emergency telemedicine services using ML methods on an undifferentiated (disease nonspecific) population. Only primary research studies in English were considered. Furthermore, only those studies using data collected remotely (as opposed to derived from physical assessments) were included. In order to limit bias, we exclusively included articles identified through our predefined search criteria and had 3 researchers (DR, JS, and KS) independently screen the resulting studies. We conducted a narrative synthesis of findings to establish a knowledge base in this domain and identify potential gaps to be addressed in forthcoming ML-enhanced methods.

RESULTS

A total of 165 unique records were screened for eligibility and 15 were included in the review. Most studies applied ML methods during emergency medical dispatch (7/15, 47%) or used chatbot applications (5/15, 33%). Patient demographics and health status variables were the most common predictors, with a notable absence of social variables. Frequently used ML models included support vector machines and tree-based methods. ML-enhanced models typically outperformed conventional triage algorithms, and we found a wide range of methods used to establish ground truth labels.

CONCLUSIONS

This scoping review observed heterogeneity in dataset size, predictors, clinical setting (triage process), and reported performance metrics. Standard structured predictors, including age, sex, and comorbidities, across articles suggest the importance of these inputs; however, there was a notable absence of other potentially useful data, including medications, social variables, and health system exposure. Ground truth labeling practices should be reported in a standard fashion as the true model performance hinges on these labels. This review calls for future work to form a standardized framework, thereby supporting consistent reporting and performance comparisons across ML-enhanced prehospital triage systems.

Reimagining Core Entrustable Professional Activities for Undergraduate Medical Education in the Era of Artificial Intelligence

The proliferation of generative artificial intelligence (AI) and its extensive potential for integration into many aspects of health care signal a transformational shift within the health care environment. In this context, medical education must evolve to ensure that medical trainees are adequately prepared to navigate the rapidly changing health care landscape. Medical education has moved toward a competency-based education paradigm, leading the Association of American Medical Colleges (AAMC) to define a set of Entrustable Professional Activities (EPAs) as its practical operational framework in undergraduate medical education. The AAMC's 13 core EPAs for entering residencies have been implemented with varying levels of success across medical schools. In this paper, we critically assess the existing core EPAs in the context of rapid AI integration in medicine. We identify EPAs that require refinement, redefinition, or comprehensive change to align with the emerging trends in health care. Moreover, this perspective proposes a set of "emerging" EPAs, informed by the changing landscape and capabilities presented by generative AI technologies. We provide a practical evaluation of the EPAs, alongside actionable recommendations on how medical education, viewed through the lens of the AAMC EPAs, can adapt and remain relevant amid rapid technological advancements. By leveraging the transformative potential of AI, we can reshape medical education to align with an AI-integrated future of medicine. This approach will help equip future health care professionals with technological competence and adaptive skills to meet the dynamic and evolving demands in health care.

Explainable hierarchical clustering for patient subtyping and risk prediction

We present a pipeline in which machine learning techniques are used to automatically identify and evaluate subtypes of hospital patients admitted between 2017 and 2021 in a large UK teaching hospital. Patient clusters are determined using routinely collected hospital data, such as those used in the UK's National Early Warning Score 2 (NEWS2). An iterative, hierarchical clustering process was used to identify the minimum set of relevant features for cluster separation. With the use of state-of-the-art explainability techniques, the identified subtypes are interpreted and assigned clinical meaning, illustrating their robustness. In parallel, clinicians assessed intracluster similarities and intercluster differences of the identified patient subtypes within the context of their clinical knowledge. For each cluster, outcome prediction models were trained and their forecasting ability was illustrated against the NEWS2 of the unclustered patient cohort. These preliminary results suggest that subtype models can outperform the established NEWS2 method, providing improved prediction of patient deterioration. By considering both the computational outputs and clinician-based explanations in patient subtyping, we aim to highlight the mutual benefit of combining machine learning techniques with clinical expertise.

Multi-modal deep learning for automated assembly of periapical radiographs

OBJECTIVES

Periapical radiographs are oftentimes taken in series to display all teeth present in the oral cavity. Our aim was to automatically assemble such a series of periapical radiographs into an anatomically correct status using a multi-modal deep learning model.

METHODS

4,707 periapical images from 387 patients (on average, 12 images per patient) were used. Radiographs were labeled according to their field of view and the dataset split into a training, validation, and test set, stratified by patient. In addition to the radiograph the timestamp of image generation was extracted and abstracted as follows: A matrix, containing the normalized timestamps of all images of a patient was constructed, representing the order in which images were taken, providing temporal context information to the deep learning model. Using the image data together with the time sequence data a multi-modal deep learning model consisting of two residual convolutional neural networks (ResNet-152 for image data, ResNet-50 for time data) was trained. Additionally, two uni-modal models were trained on image data and time data, respectively. A custom scoring technique was used to measure model performance.

RESULTS

Multi-modal deep learning outperformed both uni-modal image-based learning (p<0.001) and time-based learning (p<0.05). The multi-modal deep learning model predicted tooth labels with an F1-score, sensitivity and precision of 0.79, respectively, and an accuracy of 0.99. 37 out of 77 patient datasets were fully correctly assembled by multi-modal learning; in the remaining ones, usually only one image was incorrectly labeled.

CONCLUSIONS

Multi-modal modeling allowed automated assembly of periapical radiographs and outperformed both uni-modal models. Dental machine learning models can benefit from additional data modalities.

CLINICAL SIGNIFICANCE

Like humans, deep learning models may profit from multiple data sources for decision-making. We demonstrate how multi-modal learning can assist assembling periapical radiographs into an anatomically correct status. Multi-modal learning should be considered for more complex tasks, as clinically a wealth of data is usually available and could be leveraged.

Classification based on event in survival machine learning analysis of cardiovascular disease cohort

The aim of this study is to assess the effectiveness of supervised learning classification models in predicting patient outcomes in a survival analysis problem involving cardiovascular patients with a significant cured fraction. The sample comprised 919 patients (365 females and 554 males) who were referred to Sulaymaniyah Cardiac Hospital and followed up for a maximum of 650 days between 2021 and 2023. During the research period, 162 patients (17.6%) died, and the cure fraction in this cohort was confirmed using the Mahler and Zhu test (P < 0.01). To determine the best patient status prediction procedure, several machine learning classifications were applied. The patients were classified into alive and dead using various machine learning algorithms, with almost similar results based on several indicators. However, random forest was identified as the best method in most indicators, with an Area under ROC of 0.934. The only weakness of this method was its relatively poor performance in correctly diagnosing deceased patients, whereas SVM with FP Rate of 0.263 performed better in this regard. Logistic and simple regression also showed better performance than other methods, with an Area under ROC of 0.911 and 0.909 respectively.

Breaking Bias: The Role of Artificial Intelligence in Improving Clinical Decision-Making

This case report reflects on a delayed diagnosis for a 27-year-old woman who reported chest pain and shortness of breath to the emergency department. The treating clinician reflects upon how cognitive biases influenced their diagnostic process and how multiple missed opportunities resulted in missteps. Using artificial intelligence (AI) tools for clinical decision-making, we suggest how AI could augment the clinician, and in this case, delayed diagnosis avoided. Incorporating AI tools into clinical decision-making brings potential benefits, including improved diagnostic accuracy and addressing human factors contributing to medical errors. For example, they may support a real-time interpretation of medical imaging and assist clinicians in generating a differential diagnosis in ensuring that critical diagnoses are considered. However, it is vital to be aware of the potential pitfalls associated with the use of AI, such as automation bias, input data quality issues, limited clinician training in interpreting AI methods, and the legal and ethical considerations associated with their use. The report draws attention to the utility of AI clinical decision-support tools in overcoming human cognitive biases. It also emphasizes the importance of clinicians developing skills needed to steward the adoption of AI tools in healthcare and serve as patient advocates, ensuring safe and effective use of health data.

Prospective External Validation of a Commercial Model Predicting the Likelihood of Inpatient Admission From the Emergency Department

STUDY OBJECTIVE

Early notification of admissions from the emergency department (ED) may allow hospitals to plan for inpatient bed demand. This study aimed to assess Epic's ED Likelihood to Occupy an Inpatient Bed predictive model and its application in improving hospital bed planning workflows.

METHODS

All ED adult (18 years and older) visits from September 2021 to August 2022 at a large regional health care system were included. The primary outcome was inpatient admission. The predictive model is a random forest algorithm that uses demographic and clinical features. The model was implemented prospectively, with scores generated every 15 minutes. The area under the receiver operator curves (AUROC) and precision-recall curves (AUPRC) were calculated using the maximum score prior to the outcome and for each prediction independently. Test characteristics and lead time were calculated over a range of model score thresholds.

RESULTS

Over 11 months, 329,194 encounters were evaluated, with an incidence of inpatient admission of 25.4%. The encounter-level AUROC was 0.849 (95% confidence interval [CI], 0.848 to 0.851), and the AUPRC was 0.643 (95% CI, 0.640 to 0.647). With a prediction horizon of 6 hours, the AUROC was 0.758 (95% CI, 0.758 to 0.759,) and the AUPRC was 0.470 (95% CI, 0.469 to 0.471). At a predictive model threshold of 40, the sensitivity was 0.49, the positive predictive value was 0.65, and the median lead-time warning was 127 minutes before the inpatient bed request.

CONCLUSION

The Epic ED Likelihood to Occupy an Inpatient Bed model may improve hospital bed planning workflows. Further study is needed to determine its operational effect.

Emergency department use and Artificial Intelligence in Pelotas: design and baseline results

OBJETIVO

To describe the initial baseline results of a population-based study, as well as a protocol in order to evaluate the performance of different machine learning algorithms with the objective of predicting the demand for urgent and emergency services in a representative sample of adults from the urban area of Pelotas, Southern Brazil.

METHODS

The study is entitled "Emergency department use and Artificial Intelligence in PELOTAS (RS) (EAI PELOTAS)" (https://wp.ufpel.edu.br/eaipelotas/). Between September and December 2021, a baseline was carried out with participants. A follow-up was planned to be conducted after 12 months in order to assess the use of urgent and emergency services in the last year. Afterwards, machine learning algorithms will be tested to predict the use of urgent and emergency services over one year.

RESULTS

In total, 5,722 participants answered the survey, mostly females (66.8%), with an average age of 50.3 years. The mean number of household people was 2.6. Most of the sample has white skin color and incomplete elementary school or less. Around 30% of the sample has obesity, 14% diabetes, and 39% hypertension.

CONCLUSION

The present paper presented a protocol describing the steps that were and will be taken to produce a model capable of predicting the demand for urgent and emergency services in one year among residents of Pelotas, in Rio Grande do Sul state.

Predicting Adult Hospital Admission from Emergency Department Using Machine Learning: An Inclusive Gradient Boosting Model

BACKGROUND AND AIM

We analyzed an inclusive gradient boosting model to predict hospital admission from the emergency department (ED) at different time points. We compared its results to multiple models built exclusively at each time point.

METHODS

This retrospective multisite study utilized ED data from the Mount Sinai Health System, NY, during 2015-2019. Data included tabular clinical features and free-text triage notes represented using bag-of-words. A full gradient boosting model, trained on data available at different time points (30, 60, 90, 120, and 150 min), was compared to single models trained exclusively at data available at each time point. This was conducted by concatenating the rows of data available at each time point to one data matrix for the full model, where each row is considered a separate case.

RESULTS

The cohort included 1,043,345 ED visits. The full model showed comparable results to the single models at all time points (AUCs 0.84-0.88 for different time points for both the full and single models).

CONCLUSION

A full model trained on data concatenated from different time points showed similar results to single models trained at each time point. An ML-based prediction model can use used for identifying hospital admission.

An external validation study of the Score for Emergency Risk Prediction (SERP), an interpretable machine learning-based triage score for the emergency department

Emergency departments (EDs) are experiencing complex demands. An ED triage tool, the Score for Emergency Risk Prediction (SERP), was previously developed using an interpretable machine learning framework. It achieved a good performance in the Singapore population. We aimed to externally validate the SERP in a Korean cohort for all ED patients and compare its performance with Korean triage acuity scale (KTAS). This retrospective cohort study included all adult ED patients of Samsung Medical Center from 2016 to 2020. The outcomes were 30-day and in-hospital mortality after the patients' ED visit. We used the area under the receiver operating characteristic curve (AUROC) to assess the performance of the SERP and other conventional scores, including KTAS. The study population included 285,523 ED visits, of which 53,541 were after the COVID-19 outbreak (2020). The whole cohort, in-hospital, and 30 days mortality rates were 1.60%, and 3.80%. The SERP achieved an AUROC of 0.821 and 0.803, outperforming KTAS of 0.679 and 0.729 for in-hospital and 30-day mortality, respectively. SERP was superior to other scores for in-hospital and 30-day mortality prediction in an external validation cohort. SERP is a generic, intuitive, and effective triage tool to stratify general patients who present to the emergency department.