A validation of machine learning-based risk scores in the prehospital setting

Use of artificial intelligence to support prehospital traumatic injury care: A scoping review

Background

Artificial intelligence (AI) has transformative potential to support prehospital clinicians, emergency physicians, and trauma surgeons in acute traumatic injury care. This scoping review examines the literature evaluating AI models using prehospital features to support early traumatic injury care.

Methods

We conducted a systematic search in August 2023 of PubMed, Embase, and Web of Science. Two independent reviewers screened titles/abstracts, with a third reviewer for adjudication, followed by a full-text analysis. We included original research and conference presentations evaluating AI models-machine learning (ML), deep learning (DL), and natural language processing (NLP)-that used prehospital features or features available immediately upon emergency department arrival. Review articles were excluded. The same investigators extracted data and systematically categorized outcomes to ensure consistency and transparency. We calculated kappa for interrater reliability and descriptive statistics.

Results

We identified 1050 unique publications, with 49 meeting inclusion criteria after title and abstract review (kappa 0.58) and full-text review. Publications increased annually from 2 in 2007 to 10 in 2022. Geographic analysis revealed a 61% focus on data from the United States. Studies were predominantly retrospective (88%), used local (45%) or national level (41%) data, focused on adults only (59%) or did not specify adults or pediatrics (27%), and 57% encompassed both blunt and penetrating injury mechanisms. The majority used machine learning (88%) alone or in conjunction with DL or NLP, and the top three algorithms used were support vector machine, logistic regression, and random forest. The most common study objectives were to predict the need for critical care and life-saving interventions (29%), assist in triage (22%), and predict survival (20%).

Conclusions

A small but growing body of literature described AI models based on prehospital features that may support decisions made by dispatchers, Emergency Medical Services clinicians, and trauma teams in early traumatic injury care.

Improving Triage Accuracy in Prehospital Emergency Telemedicine: Scoping Review of Machine Learning-Enhanced Approaches

BACKGROUND

Prehospital telemedicine triage systems combined with machine learning (ML) methods have the potential to improve triage accuracy and safely redirect low-acuity patients from attending the emergency department. However, research in prehospital settings is limited but needed; emergency department overcrowding and adverse patient outcomes are increasingly common.

OBJECTIVE

In this scoping review, we sought to characterize the existing methods for ML-enhanced telemedicine emergency triage. In order to support future research, we aimed to delineate what data sources, predictors, labels, ML models, and performance metrics were used, and in which telemedicine triage systems these methods were applied.

METHODS

A scoping review was conducted, querying multiple databases (MEDLINE, PubMed, Scopus, and IEEE Xplore) through February 24, 2023, to identify potential ML-enhanced methods, and for those eligible, relevant study characteristics were extracted, including prehospital triage setting, types of predictors, ground truth labeling method, ML models used, and performance metrics. Inclusion criteria were restricted to the triage of emergency telemedicine services using ML methods on an undifferentiated (disease nonspecific) population. Only primary research studies in English were considered. Furthermore, only those studies using data collected remotely (as opposed to derived from physical assessments) were included. In order to limit bias, we exclusively included articles identified through our predefined search criteria and had 3 researchers (DR, JS, and KS) independently screen the resulting studies. We conducted a narrative synthesis of findings to establish a knowledge base in this domain and identify potential gaps to be addressed in forthcoming ML-enhanced methods.

RESULTS

A total of 165 unique records were screened for eligibility and 15 were included in the review. Most studies applied ML methods during emergency medical dispatch (7/15, 47%) or used chatbot applications (5/15, 33%). Patient demographics and health status variables were the most common predictors, with a notable absence of social variables. Frequently used ML models included support vector machines and tree-based methods. ML-enhanced models typically outperformed conventional triage algorithms, and we found a wide range of methods used to establish ground truth labels.

CONCLUSIONS

This scoping review observed heterogeneity in dataset size, predictors, clinical setting (triage process), and reported performance metrics. Standard structured predictors, including age, sex, and comorbidities, across articles suggest the importance of these inputs; however, there was a notable absence of other potentially useful data, including medications, social variables, and health system exposure. Ground truth labeling practices should be reported in a standard fashion as the true model performance hinges on these labels. This review calls for future work to form a standardized framework, thereby supporting consistent reporting and performance comparisons across ML-enhanced prehospital triage systems.

Using machine learning methods to predict all-cause somatic hospitalizations in adults: A systematic review

AIM

In this review, we investigated how Machine Learning (ML) was utilized to predict all-cause somatic hospital admissions and readmissions in adults.

METHODS

We searched eight databases (PubMed, Embase, Web of Science, CINAHL, ProQuest, OpenGrey, WorldCat, and MedNar) from their inception date to October 2023, and included records that predicted all-cause somatic hospital admissions and readmissions of adults using ML methodology. We used the CHARMS checklist for data extraction, PROBAST for bias and applicability assessment, and TRIPOD for reporting quality.

RESULTS

We screened 7,543 studies of which 163 full-text records were read and 116 met the review inclusion criteria. Among these, 45 predicted admission, 70 predicted readmission, and one study predicted both. There was a substantial variety in the types of datasets, algorithms, features, data preprocessing steps, evaluation, and validation methods. The most used types of features were demographics, diagnoses, vital signs, and laboratory tests. Area Under the ROC curve (AUC) was the most used evaluation metric. Models trained using boosting tree-based algorithms often performed better compared to others. ML algorithms commonly outperformed traditional regression techniques. Sixteen studies used Natural language processing (NLP) of clinical notes for prediction, all studies yielded good results. The overall adherence to reporting quality was poor in the review studies. Only five percent of models were implemented in clinical practice. The most frequently inadequately addressed methodological aspects were: providing model interpretations on the individual patient level, full code availability, performing external validation, calibrating models, and handling class imbalance.

CONCLUSION

This review has identified considerable concerns regarding methodological issues and reporting quality in studies investigating ML to predict hospitalizations. To ensure the acceptability of these models in clinical settings, it is crucial to improve the quality of future studies.

The AI Future of Emergency Medicine

In the coming years, artificial intelligence (AI) and machine learning will likely give rise to profound changes in the field of emergency medicine, and medicine more broadly. This article discusses these anticipated changes in terms of 3 overlapping yet distinct stages of AI development. It reviews some fundamental concepts in AI and explores their relation to clinical practice, with a focus on emergency medicine. In addition, it describes some of the applications of AI in disease diagnosis, prognosis, and treatment, as well as some of the practical issues that they raise, the barriers to their implementation, and some of the legal and regulatory challenges they create.

Injury severity bias in missing prehospital vital signs: Prevalence and implications for trauma registries

BACKGROUND

Vital signs are important factors in assessing injury severity and guiding trauma resuscitation, especially among severely injured patients. Despite this, physiological data are frequently missing from trauma registries. This study aimed to evaluate the extent of missing prehospital data in a hospital-based trauma registry and to assess the associations between prehospital physiological data completeness and indicators of injury severity.

METHODS

A retrospective review was conducted on all adult trauma patients brought directly to a level 1 trauma center in Toronto, Ontario by paramedics from January 1, 2015, to December 31, 2019. The proportion of missing data was evaluated for each variable and patterns of missingness were assessed. To investigate the associations between prehospital data completeness and injury severity factors, descriptive and unadjusted logistic regression analyses were performed.

RESULTS

A total of 3,528 patients were included. We considered prehospital data missing if any of heart rate, systolic blood pressure, respiratory rate or oxygen saturation were incomplete. Each individual variable was missing from the registry in approximately 20 % of patients, with oxygen saturation missing most frequently (n = 831; 23.6 %). Over 25 % (n = 909) of patients were missing at least one prehospital vital sign, of which 69.1 % (n = 628) were missing all four of these variables. Patients with incomplete data were more severely injured, had higher mortality, and more frequently received lifesaving interventions such as blood transfusion and intubation. Patients were most likely to have missing prehospital physiological data if they died in the trauma bay (unadjusted OR: 9.79; 95 % CI: 6.35-15.10), did not survive to discharge (unadjusted OR: 3.55; 95 % CI: 2.76-4.55), or had a prehospital GCS less than 9 (OR: 3.24; 95 % CI: 2.59-4.06).

CONCLUSION

In this single center trauma registry, key prehospital variables were frequently missing, particularly among more severely injured patients. Patients with missing data had higher mortality, more severe injury characteristics and received more life-saving interventions in the trauma bay, suggesting an injury severity bias in prehospital vital sign missingness. To ensure the validity of research based on trauma registry data, patterns of missingness must be carefully considered to ensure missing data is appropriately addressed.

Deep continual learning for medical call incidents text classification under the presence of dataset shifts

The aim of this work is to develop and evaluate a deep classifier that can effectively prioritize Emergency Medical Call Incidents (EMCI) according to their life-threatening level under the presence of dataset shifts. We utilized a dataset consisting of 1982746 independent EMCI instances obtained from the Health Services Department of the Region of Valencia (Spain), with a time span from 2009 to 2019 (excluding 2013). The dataset includes free text dispatcher observations recorded during the call, as well as a binary variable indicating whether the event was life-threatening. To evaluate the presence of dataset shifts, we examined prior probability shifts, covariate shifts, and concept shifts. Subsequently, we designed and implemented four deep Continual Learning (CL) strategies-cumulative learning, continual fine-tuning, experience replay, and synaptic intelligence-alongside three deep CL baselines-joint training, static approach, and single fine-tuning-based on DistilBERT models. Our results demonstrated evidence of prior probability shifts, covariate shifts, and concept shifts in the data. Applying CL techniques had a statistically significant (α=0.05) positive impact on both backward and forward knowledge transfer, as measured by the F1-score, compared to non-continual approaches. We can argue that the utilization of CL techniques in the context of EMCI is effective in adapting deep learning classifiers to changes in data distributions, thereby maintaining the stability of model performance over time. To our knowledge, this study represents the first exploration of a CL approach using real EMCI data.

Dispatch Categories as Indicators of Out-of-Hospital Time Critical Interventions and Associated Emergency Department Outcomes

OBJECTIVES

Emergency medical services (EMS) systems increasingly grapple with rising call volumes and workforce shortages, forcing systems to decide which responses may be delayed. Limited research has linked dispatch codes, on-scene findings, and emergency department (ED) outcomes. This study evaluated the association between dispatch categorizations and time-critical EMS responses defined by prehospital interventions and ED outcomes. Secondarily, we proposed a framework for identifying dispatch categorizations that are safe or unsafe to hold in queue.

METHODS

This retrospective, multi-center analysis encompassed all 9-1-1 responses from 8 accredited EMS systems between 1/1/2021 and 06/30/2023, utilizing the Medical Priority Dispatch System (MPDS). Independent variables included MPDS Protocol numbers and Determinant levels. EMS treatments and ED diagnoses/dispositions were categorized as time-critical using a multi-round consensus survey. The primary outcome was the proportion of EMS responses categorized as time-critical. A non-parametric test for trend was used to assess the proportion of time-critical responses Determinant levels. Based on group consensus, Protocol/Determinant level combinations with at least 120 responses (∼1 per week) were further categorized as safe to hold in queue (<1% time-critical intervention by EMS and <5% time-critical ED outcome) or unsafe to hold in queue (>10% time-critical intervention by EMS or >10% time-critical ED outcome).

RESULTS

Of 1,715,612 EMS incidents, 6% (109,250) involved a time-critical EMS intervention. Among EMS transports with linked outcome data (543,883), 12% had time-critical ED outcomes. The proportion of time-critical EMS interventions increased with Determinant level (OMEGA: 1%, ECHO: 38%, p-trend < 0.01) as did time-critical ED outcomes (OMEGA: 3%, ECHO: 31%, p-trend < 0.01). Of 162 unique Protocols/Determinants with at least 120 uses, 30 met criteria for safe to hold in queue, accounting for 8% (142,067) of incidents. Meanwhile, 72 Protocols/Determinants met criteria for unsafe to hold, accounting for 52% (883,683) of incidents. Seven of 32 ALPHA level Protocols and 3/17 OMEGA level Protocols met the proposed criteria for unsafe to hold in queue.

CONCLUSIONS

In general, Determinant levels aligned with time-critical responses; however, a notable minority of lower acuity Determinant level Protocols met criteria for unsafe to hold. This suggests a more nuanced approach to dispatch prioritization, considering both Protocol and Determinant level factors.

AI-based approach for transcribing and classifying unstructured emergency call data: A methodological proposal

Emergency care-sensitive conditions (ECSCs) require rapid identification and treatment and are responsible for over half of all deaths worldwide. Prehospital emergency care (PEC) can provide rapid treatment and access to definitive care for many ECSCs and can reduce mortality in several different settings. The objective of this study is to propose a method for using artificial intelligence (AI) and machine learning (ML) to transcribe audio, extract, and classify unstructured emergency call data in the Serviço de Atendimento Móvel de Urgência (SAMU) system in southern Brazil. The study used all "1-9-2" calls received in 2019 by the SAMU Novo Norte Emergency Regulation Center (ERC) call center in Maringá, in the Brazilian state of Paraná. The calls were processed through a pipeline using machine learning algorithms, including Automatic Speech Recognition (ASR) models for transcription of audio calls in Portuguese, and a Natural Language Understanding (NLU) classification model. The pipeline was trained and validated using a dataset of labeled calls, which were manually classified by medical students using LabelStudio. The results showed that the AI model was able to accurately transcribe the audio with a Word Error Rate of 42.12% using Wav2Vec 2.0 for ASR transcription of audio calls in Portuguese. Additionally, the NLU classification model had an accuracy of 73.9% in classifying the calls into different categories in a validation subset. The study found that using AI to categorize emergency calls in low- and middle-income countries is largely unexplored, and the applicability of conventional open-source ML models trained on English language datasets is unclear for non-English speaking countries. The study concludes that AI can be used to transcribe audio and extract and classify unstructured emergency call data in an emergency system in southern Brazil as an initial step towards developing a decision-making support tool.

AI Algorithm to Predict Acute Coronary Syndrome in Prehospital Cardiac Care: Retrospective Cohort Study

BACKGROUND

Overcrowding of hospitals and emergency departments (EDs) is a growing problem. However, not all ED consultations are necessary. For example, 80% of patients in the ED with chest pain do not have an acute coronary syndrome (ACS). Artificial intelligence (AI) is useful in analyzing (medical) data, and might aid health care workers in prehospital clinical decision-making before patients are presented to the hospital.

OBJECTIVE

The aim of this study was to develop an AI model which would be able to predict ACS before patients visit the ED. The model retrospectively analyzed prehospital data acquired by emergency medical services' nurse paramedics.

METHODS

Patients presenting to the emergency medical services with symptoms suggestive of ACS between September 2018 and September 2020 were included. An AI model using a supervised text classification algorithm was developed to analyze data. Data were analyzed for all 7458 patients (mean 68, SD 15 years, 54% men). Specificity, sensitivity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for control and intervention groups. At first, a machine learning (ML) algorithm (or model) was chosen; afterward, the features needed were selected and then the model was tested and improved using iterative evaluation and in a further step through hyperparameter tuning. Finally, a method was selected to explain the final AI model.

RESULTS

The AI model had a specificity of 11% and a sensitivity of 99.5% whereas usual care had a specificity of 1% and a sensitivity of 99.5%. The PPV of the AI model was 15% and the NPV was 99%. The PPV of usual care was 13% and the NPV was 94%.

CONCLUSIONS

The AI model was able to predict ACS based on retrospective data from the prehospital setting. It led to an increase in specificity (from 1% to 11%) and NPV (from 94% to 99%) when compared to usual care, with a similar sensitivity. Due to the retrospective nature of this study and the singular focus on ACS it should be seen as a proof-of-concept. Other (possibly life-threatening) diagnoses were not analyzed. Future prospective validation is necessary before implementation.

On Scene Injury Severity Prediction (OSISP) model for trauma developed using the Swedish Trauma Registry

BACKGROUND

Providing optimal care for trauma, the leading cause of death for young adults, remains a challenge e.g., due to field triage limitations in assessing a patient's condition and deciding on transport destination. Data-driven On Scene Injury Severity Prediction (OSISP) models for motor vehicle crashes have shown potential for providing real-time decision support. The objective of this study is therefore to evaluate if an Artificial Intelligence (AI) based clinical decision support system can identify severely injured trauma patients in the prehospital setting.

METHODS

The Swedish Trauma Registry was used to train and validate five models - Logistic Regression, Random Forest, XGBoost, Support Vector Machine and Artificial Neural Network - in a stratified 10-fold cross validation setting and hold-out analysis. The models performed binary classification of the New Injury Severity Score and were evaluated using accuracy metrics, area under the receiver operating characteristic curve (AUC) and Precision-Recall curve (AUCPR), and under- and overtriage rates.

RESULTS

There were 75,602 registrations between 2013-2020 and 47,357 (62.6%) remained after eligibility criteria were applied. Models were based on 21 predictors, including injury location. From the clinical outcome, about 40% of patients were undertriaged and 46% were overtriaged. Models demonstrated potential for improved triaging and yielded AUC between 0.80-0.89 and AUCPR between 0.43-0.62.

CONCLUSIONS

AI based OSISP models have potential to provide support during assessment of injury severity. The findings may be used for developing tools to complement field triage protocols, with potential to improve prehospital trauma care and thereby reduce morbidity and mortality for a large patient population.

Artificial intelligence and machine learning in prehospital emergency care: A scoping review

Our scoping review provides a comprehensive analysis of the landscape of artificial intelligence (AI) applications in prehospital emergency care (PEC). It contributes to the field by highlighting the most studied AI applications and identifying the most common methodological approaches across 106 included studies. The findings indicate a promising future for AI in PEC, with many unique use cases, such as prognostication, demand prediction, resource optimization, and the Internet of Things continuous monitoring systems. Comparisons with other approaches showed AI outperforming clinicians and non-AI algorithms in most cases. However, most studies were internally validated and retrospective, highlighting the need for rigorous prospective validation of AI applications before implementation in clinical settings. We identified knowledge and methodological gaps using an evidence map, offering a roadmap for future investigators. We also discussed the significance of explainable AI for establishing trust in AI systems among clinicians and facilitating real-world validation of AI models.

Machine Learning-Based Models for Prediction of Critical Illness at Community, Paramedic, and Hospital Stages

Overcrowding of emergency department (ED) has put a strain on national healthcare systems and adversely affected the clinical outcomes of critically ill patients. Early identification of critically ill patients prior to ED visits can help induce optimal patient flow and allocate medical resources effectively. This study aims to develop ML-based models for predicting critical illness in the community, paramedic, and hospital stages using Korean National Emergency Department Information System (NEDIS) data. Random forest and light gradient boosting machine (LightGBM) were applied to develop predictive models. The predictive model performance based on AUROC in community stage, paramedic stage, and hospital stage was estimated to be 0.870 (95% CI: 0.869-0.871), 0.897 (95% CI: 0.896-0.898), and 0.950 (95% CI: 0.949-0.950) in random forest and 0.877 (95% CI: 0.876-0.878), 0.899 (95% CI: 0.898-0.900), and 0.950 (95% CI: 0.950-0.951) in LightGBM, respectively. The ML models showed high performance in predicting critical illness using variables available at each stage, which can be helpful in guiding patients to appropriate hospitals according to their severity of illness. Furthermore, a simulation model can be developed for proper allocation of limited medical resources.

Information governance as a socio-technical process in the development of trustworthy healthcare AI

In this paper we describe our experiences of managing information governance (IG) processes for the assurance of healthcare AI, using the example of an out-of-hospital-cardiac-arrest recognition software within the context of the Welsh Ambulance Service. We frame IG as a socio-technical process. IG processes for the development of trustworthy healthcare AI rely on information governance work, which entails dialogue, negotiation, and trade-offs around the legal basis for data sharing, data requirements and data control. Information governance work should start early in the design life cycle and will likely continue throughout. This includes a focus on establishing and building relationships, as well as a focus on organizational readiness and deeper understanding of both AI technologies as well as their safety assurance requirements.

Development of outcomes for evaluating emergency care triage: a Delphi approach

BACKGROUND

Triage is used as standard of care for prioritization and identification of time-critical patients in the emergency department (ED) globally, but it is unclear what outcomes should be used to evaluate triage. Currently used outcomes do not include important time-critical diagnoses and conditions.

METHOD

We used 18 Swedish triage experts to collect and assess outcomes for the evaluation of 5-level triage systems. The experts suggested 68 outcomes which were then tested through a modified Delphi approach in three rounds. The outcomes aimed to identify correctly prioritized red patients (in need of a resuscitation team), and orange patients (other time critical conditions). Consensus was pre-defined as 70% dichotomized (positive/negative) concordance.

RESULTS

Diagnoses, interventions, mortality, level of care and lab results were included in the outcomes. Positive consensus was reached for 49 outcomes and negative consensus for 7 outcomes, with an 83% response rate. The five most approved outcomes were the interventions Percutaneous coronary intervention, Surgical airway and Massive transfusion together with the diagnoses Tension pneumothorax and Intracerebral hemorrhage that received specific interventions. The outcomes with the clearest disapproval included Admittance to a ward, Treatment with antihistamines and The ordering of a head computed tomography scan. The outcomes were considered valid only if occurring in or from the ED.

CONCLUSION

This study proposes a standard of 49 outcomes divided into two sets tied to red and orange priority respectively, to be used when evaluating 5-level priority triage systems; Lund Outcome Set for Evaluation of Triage (LOSET). The proposed outcomes include diagnoses, interventions and laboratory results. Before widespread implementation of LOSET, prospective testing is needed, preferably at multiple sites.

Exploring the effectiveness of artificial intelligence, machine learning and deep learning in trauma triage: A systematic review and meta-analysis

Background

The development of artificial intelligence (AI), machine learning (ML) and deep learning (DL) has advanced rapidly in the medical field, notably in trauma medicine. We aimed to systematically appraise the efficacy of AI, ML and DL models for predicting outcomes in trauma triage compared to conventional triage tools.

Methods

We searched PubMed, MEDLINE, ProQuest, Embase and reference lists for studies published from 1 January 2010 to 9 June 2022. We included studies which analysed the use of AI, ML and DL models for trauma triage in human subjects. Reviews and AI/ML/DL models used for other purposes such as teaching, or diagnosis were excluded. Data was extracted on AI/ML/DL model type, comparison tools, primary outcomes and secondary outcomes. We performed meta-analysis on studies reporting our main outcomes of mortality, hospitalisation and critical care admission.

Results

One hundred and fourteen studies were identified in our search, of which 14 studies were included in the systematic review and 10 were included in the meta-analysis. All studies performed external validation. The best-performing AI/ML/DL models outperformed conventional trauma triage tools for all outcomes in all studies except two. For mortality, the mean area under the receiver operating characteristic (AUROC) score difference between AI/ML/DL models and conventional trauma triage was 0.09, 95% CI (0.02, 0.15), favouring AI/ML/DL models (p = 0.008). The mean AUROC score difference for hospitalisation was 0.11, 95% CI (0.10, 0.13), favouring AI/ML/DL models (p = 0.0001). For critical care admission, the mean AUROC score difference was 0.09, 95% CI (0.08, 0.10) favouring AI/ML/DL models (p = 0.00001).

Conclusions

This review demonstrates that the predictive ability of AI/ML/DL models is significantly better than conventional trauma triage tools for outcomes of mortality, hospitalisation and critical care admission. However, further research and in particular randomised controlled trials are required to evaluate the clinical and economic impacts of using AI/ML/DL models in trauma medicine.

Development of a machine-learning algorithm to predict in-hospital cardiac arrest for emergency department patients using a nationwide database

In this retrospective observational study, we aimed to develop a machine-learning model using data obtained at the prehospital stage to predict in-hospital cardiac arrest in the emergency department (ED) of patients transferred via emergency medical services. The dataset was constructed by attaching the prehospital information from the National Fire Agency and hospital factors to data from the National Emergency Department Information System. Machine-learning models were developed using patient variables, with and without hospital factors. We validated model performance and used the SHapley Additive exPlanation model interpretation. In-hospital cardiac arrest occurred in 5431 of the 1,350,693 patients (0.4%). The extreme gradient boosting model showed the best performance with area under receiver operating curve of 0.9267 when incorporating the hospital factor. Oxygen supply, age, oxygen saturation, systolic blood pressure, the number of ED beds, ED occupancy, and pulse rate were the most influential variables, in that order. ED occupancy and in-hospital cardiac arrest occurrence were positively correlated, and the impact of ED occupancy appeared greater in small hospitals. The machine-learning predictive model using the integrated information acquired in the prehospital stage effectively predicted in-hospital cardiac arrest in the ED and can contribute to the efficient operation of emergency medical systems.

An external validation study of the Score for Emergency Risk Prediction (SERP), an interpretable machine learning-based triage score for the emergency department

Emergency departments (EDs) are experiencing complex demands. An ED triage tool, the Score for Emergency Risk Prediction (SERP), was previously developed using an interpretable machine learning framework. It achieved a good performance in the Singapore population. We aimed to externally validate the SERP in a Korean cohort for all ED patients and compare its performance with Korean triage acuity scale (KTAS). This retrospective cohort study included all adult ED patients of Samsung Medical Center from 2016 to 2020. The outcomes were 30-day and in-hospital mortality after the patients' ED visit. We used the area under the receiver operating characteristic curve (AUROC) to assess the performance of the SERP and other conventional scores, including KTAS. The study population included 285,523 ED visits, of which 53,541 were after the COVID-19 outbreak (2020). The whole cohort, in-hospital, and 30 days mortality rates were 1.60%, and 3.80%. The SERP achieved an AUROC of 0.821 and 0.803, outperforming KTAS of 0.679 and 0.729 for in-hospital and 30-day mortality, respectively. SERP was superior to other scores for in-hospital and 30-day mortality prediction in an external validation cohort. SERP is a generic, intuitive, and effective triage tool to stratify general patients who present to the emergency department.

Using machine learning to predict subsequent events after EMS non-conveyance decisions

Background

Predictors of subsequent events after Emergency Medical Services (EMS) non-conveyance decisions are still unclear, though patient safety is the priority in prehospital emergency care. The aim of this study was to find out whether machine learning can be used in this context and to identify the predictors of subsequent events based on narrative texts of electronic patient care records (ePCR).

Methods

This was a prospective cohort study of EMS patients in Finland. The data was collected from three different regions between June 1 and November 30, 2018. Machine learning, in form of text classification, and manual evaluation were used to predict subsequent events from the clinical notes after a non-conveyance mission.

Results

FastText-model (AUC 0.654) performed best in prediction of subsequent events after EMS non-conveyance missions (n = 11,846). The model and manual analyses showed that many of the subsequent events were planned before, EMS guided the patients to visit primary health care facilities or ED next or following days after non-conveyance. The most frequent signs and symptoms as subsequent event predictors were musculoskeletal-, infection-related and non-specific complaints. 1 in 5 the EMS documentation was inadequate and many of these led to a subsequent event.

Conclusion

Machine learning can be used to predict subsequent events after EMS non-conveyance missions. From the patient safety perspective, it is notable that subsequent event does not necessarily mean that patient safety is compromised. There were a number of subsequent visits to primary health care or EDs, which were planned before by EMS. This demonstrates the appropriate use of limited resources to avoid unnecessary conveyance to the ED. However, further studies are needed without planned subsequent events to find out the harmful subsequent events, where EMS non-conveyance puts patient safety at risk.

Assuring safe artificial intelligence in critical ambulance service response: study protocol

Introduction: Early recognition of out-of-hospital cardiac arrest (OHCA) by ambulance service call centre operators is important so that cardiopulmonary resuscitation can be delivered immediately, but around 25% of OHCAs are not picked up by call centre operators. An artificial intelligence (AI) system has been developed to support call centre operators in the detection of OHCA. The study aims to (1) explore ambulance service stakeholder perceptions on the safety of OHCA AI decision support in call centres, and (2) develop a clinical safety case for the OHCA AI decision-support system.Methods and analysis: The study will be undertaken within the Welsh Ambulance Service. The study is part research and part service evaluation. The research utilises a qualitative study design based on thematic analysis of interview data. The service evaluation consists of the development of a clinical safety case based on document analysis, analysis of the AI model and its development process and informal interviews with the technology developer.Conclusions: AI presents many opportunities for ambulance services, but safety assurance requirements need to be understood. The ASSIST project will continue to explore and build the body of knowledge in this area.

Suboptimal prehospital decision- making for referral to alternative levels of care - frequency, measurement, acceptance rate and room for improvement

Background

The emergency medical services (EMS) have undergone dramatic changes during the past few decades. Increased utilisation, changes in care-seeking behaviour and competence among EMS clinicians have given rise to a shift in EMS strategies in many countries. From transport to the emergency department to at the scene deciding on the most appropriate level of care and mode of transport. Among the non-conveyed patients some may suffer from “time-sensitive conditions” delaying diagnosis and treatment. Thus, four questions arise:

How often are time-sensitive cases referred to primary care or self-care advice?

How can we measure and define the level of inappropriate clinical decision-making?

What is acceptable?

How to increase patient safety?

Main text

To what extent time-sensitive cases are non-conveyed varies. About 5–25% of referred patients visit the emergency department within 72 hours, 5% are hospitalised, 1–3% are reported to have a time-sensitive condition and seven-day mortality rates range from 0.3 to 6%.

The level of inappropriate clinical decision-making can be measured using surrogate measures such as emergency department attendances, hospitalisation and short-term mortality. These measures do not reveal time-sensitive conditions. Defining a scoring system may be one alternative, where misclassifications of time-sensitive cases are rated based on how severely they affected patient outcome.

In terms of what is acceptable there is no general agreement. Although a zero-vision approach does not seem to be realistic unless under-triage is split into different levels of severity with zero-vision in the most severe categories.

There are several ways to reduce the risk of misclassifications. Implementation of support systems for decision-making using machine learning to improve the initial assessment is one approach. Using a trigger tool to identify adverse events is another.

Conclusion

A substantial number of patients are non-conveyed, including a small portion with time-sensitive conditions. This poses a threat to patient safety. No general agreement on how to define and measure the extent of such EMS referrals and no agreement of what is acceptable exists, but we conclude an overall zero-vision is not realistic. Developing specific tools supporting decision making regarding EMS referral may be one way to reduce misclassification rates.

Artificial intelligence in Emergency Medical Services dispatching: assessing the potential impact of an automatic speech recognition software on stroke detection taking the Capital Region of Denmark as case in point

BACKGROUND AND PURPOSE

Stroke recognition at the Emergency Medical Services (EMS) impacts the stroke treatment and thus the related health outcome. At the EMS Copenhagen 66.2% of strokes are detected by the Emergency Medical Dispatcher (EMD) and in Denmark approximately 50% of stroke patients arrive at the hospital within the time-to-treatment. An automatic speech recognition software (ASR) can increase the recognition of Out-of-Hospital cardiac arrest (OHCA) at the EMS by 16%. This research aims to analyse the potential impact an ASR could have on stroke recognition at the EMS Copenhagen and the related treatment.

METHODS

Stroke patient data (n = 9049) from the years 2016-2018 were analysed retrospectively, regarding correlations between stroke detection at the EMS and stroke specific, as well as personal characteristics such as stroke type, sex, age, weekday, time of day, year, EMS number contacted, and treatment. The possible increase in stroke detection through an ASR and the effect on stroke treatment was calculated based on the impact of an existing ASR to detect OHCA from CORTI AI.

RESULTS

The Chi-Square test with the respective post-hoc test identified a negative correlation between stroke detection and females, the 1813-Medical Helpline, as well as weekends, and a positive correlation between stroke detection and treatment and thrombolysis. While the association analysis showed a moderate correlation between stroke detection and treatment the correlation to the other treatment options was weak or very weak. A potential increase in stroke detection to 61.19% with an ASR and hence an increase of thrombolysis by 5% in stroke patients calling within time-to-treatment was predicted.

CONCLUSIONS

An ASR can potentially improve stroke recognition by EMDs and subsequent stroke treatment at the EMS Copenhagen. Based on the analysis results improvement of stroke recognition is particularly relevant for females, younger stroke patients, calls received through the 1813-Medical Helpline, and on weekends.

TRIAL REGISTRATION

This study was registered at the Danish Data Protection Agency (PVH-2014-002) and the Danish Patient Safety Authority (R-21013122).

Can Prehospital Data Improve Early Identification of Sepsis in Emergency Department? An Integrative Review of Machine Learning Approaches

BACKGROUND

Sepsis is associated with high mortality, especially during the novel coronavirus disease 2019 (COVID-19) pandemic. Along with high monetary health care costs for sepsis treatment, there is a lasting impact on lives of sepsis survivors and their caregivers. Early identification is necessary to reduce the negative impact of sepsis and to improve patient outcomes. Prehospital data are among the earliest information collected by health care systems. Using these untapped sources of data in machine learning (ML)-based approaches can identify patients with sepsis earlier in emergency department (ED).

OBJECTIVES

This integrative literature review aims to discuss the importance of utilizing prehospital data elements in ED, summarize their current use in developing ML-based prediction models, and specifically identify those data elements that can potentially contribute to early identification of sepsis in ED when used in ML-based approaches.

METHOD

Literature search strategy includes following two separate searches: (1) use of prehospital data in ML models in ED; and (2) ML models that are developed specifically to predict/detect sepsis in ED. In total, 24 articles are used in this review.

RESULTS

A summary of prehospital data used to identify time-sensitive conditions earlier in ED is provided. Literature related to use of ML models for early identification of sepsis in ED is limited and no studies were found related to ML models using prehospital data in prediction/early identification of sepsis in ED. Among those using ED data, ML models outperform traditional statistical models. In addition, the use of the free-text elements and natural language processing (NLP) methods could result in better prediction of sepsis in ED.

CONCLUSION

This study reviews the use of prehospital data in early decision-making in ED and suggests that researchers utilize such data elements for prediction/early identification of sepsis in ML-based approaches.

Correlation between cerebrospinal fluid and plasma neurofilament light protein in treated HIV infection: results from the COBRA study

Cerebrospinal fluid (CSF) neurofilament light protein (NfL) is a marker of central nervous system neuro-axonal injury. A novel, ultra-sensitive assay can determine plasma NfL. In untreated people-with-HIV (PWH), CSF and plasma NfL are strongly correlated. We aimed to assess this correlation in PWH on suppressive antiretroviral treatment (ART) and lifestyle-similar HIV-negative individuals enrolled into the COmorBidity in Relation to AIDS (COBRA) study. Differences in paired CSF (sandwich ELISA, UmanDiagnostics) and plasma (Simoa digital immunoassay, Quanterix™) NfL between PWH and HIV-negative participants were tested using Wilcoxon's test; associations were assessed using Pearson's correlation. CSF and plasma NfL, standardised to Z-scores, were included as dependent variables in linear regression models to identify factors independently associated with values in PWH and HIV-negative participants. Overall, 132 PWH (all with plasma HIV RNA < 50 copies/mL) and 79 HIV-negative participants were included. Neither CSF (median 570 vs 568 pg/mL, p = 0.37) nor plasma (median 10.7 vs 9.9 pg/mL, p = 0.15) NfL differed significantly between PWH and HIV-negative participants, respectively. CSF and plasma NfL correlated moderately, with no significant difference by HIV status (PWH: rho = 0.52; HIV-negative participants: rho = 0.47, p (interaction) = 0.63). In multivariable regression analysis, higher CSF NfL Z-score was statistically significantly associated with older age and higher CSF protein, and higher plasma NfL Z-score with older age, higher serum creatinine and lower bodyweight. In conclusion, in PWH on ART, the correlation between CSF and plasma NfL is moderate and similar to that observed in lifestyle-similar HIV-negative individuals. Consideration of renal function and bodyweight may be required when utilising plasma NfL.

Structural brain morphometry as classifier and predictor of ADHD and reward-related comorbidities

Attention deficit/hyperactivity disorder (ADHD) is one of the most common neurodevelopmental disorders, and around two-thirds of affected children report persisting problems in adulthood. This negative trajectory is associated with high comorbidity with disorders like obesity, depression, or substance use disorder (SUD). Decreases in cortical volume and thickness have also been reported in depression, SUD, and obesity, but it is unclear whether structural brain alterations represent unique disorder-specific profiles. A transdiagnostic exploration of ADHD and typical comorbid disorders could help to understand whether specific morphometric brain changes are due to ADHD or, alternatively, to the comorbid disorders. In the current study, we studied the brain morphometry of 136 subjects with ADHD with and without comorbid depression, SUD, and obesity to test whether there are unique or common brain alterations. We employed a machine-learning-algorithm trained to classify subjects with ADHD in the large ENIGMA-ADHD dataset and used it to predict the diagnostic status of subjects with ADHD and/or comorbidities. The parcellation analysis demonstrated decreased cortical thickness in medial prefrontal areas that was associated with presence of any comorbidity. However, these results did not survive correction for multiple comparisons. Similarly, the machine learning analysis indicated that the predictive algorithm grouped most of our ADHD participants as belonging to the ADHD-group, but no systematic differences between comorbidity status came up. In sum, neither a classical comparison of segmented structural brain metrics nor an ML model based on the ADHD ENIGMA data differentiate between ADHD with and without comorbidities. As the ML model is based in part on adolescent brains, this might indicate that comorbid disorders and their brain changes are not captured by the ML model because it represents a different developmental brain trajectory.

Effect of early nutritional initiation on post-cerebral infarction discharge destination: A propensity-matched analysis using machine learning

AIM

Malnutrition is associated with poor outcomes in cerebral infarction patients, with research indicating that early nutritional initiation may improve the short-term prognosis of patients. However, evidence supported by big data is lacking. Here, to determine the effect of nutritional initiation during the first 3 days after hospital admission on home discharge rates, propensity score matching was conducted in patients with acute cerebral infarction.

METHODS

This retrospective observational study, using the Diagnosis Procedure Combination anonymised database in Japan, included 41 477 ischaemic cerebral infarction patients hospitalised between 2016 and 2019. The patients were divided into two groups: those who received oral or enteral nutrition during the first 3 days of hospital admission (early nutrition group, n = 37 318) and those who did not (control group, n = 4159). One-to-one pair-matching was performed using propensity scores calculated via extreme gradient boosting to limit the confounding variables of the two groups.

RESULTS

After propensity score matching, 3541 pairs of patients were selected. The dependence of home discharge rates on early nutrition was significant (p < 0.05), and the effectiveness of early nutrition for home discharge showed an odds ratio of 1.79 (95% confidence interval of 1.59-2.03 in Fisher's exact test).

CONCLUSIONS

Our findings revealed that early nutritional initiation during the first 3 days of admission resulted in higher home discharge rates.

XGBoost Algorithm Prediction of Critical Care Outcome for Adult Patients Presenting to Emergency Department Using Initial Triage Information

Background

The emergency department (ED) triage system to classify and prioritize patients from high risk to less urgent continues to be a challenge.

Objective

This study, comprising 80,433 patients, aims to develop a machine learning algorithm prediction model of critical care outcomes for adult patients using information collected during ED triage and compare the performance with that of the baseline model using the Korean Triage and Acuity Scale (KTAS).

Methods

To predict the need for critical care, we used 13 predictors from triage information: age, gender, mode of ED arrival, the time interval between onset and ED arrival, reason of ED visit, chief complaints, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, oxygen saturation, and level of consciousness. The baseline model with KTAS was developed using logistic regression, and the machine learning model with 13 variables was generated using extreme gradient boosting (XGB) and deep neural network (DNN) algorithms. The discrimination was measured by the area under the receiver operating characteristic (AUROC) curve. The ability of calibration with Hosmer–Lemeshow test and reclassification with net reclassification index were evaluated. The calibration plot and partial dependence plot were used in the analysis.

Results

The AUROC of the model with the full set of variables (0.833-0.861) was better than that of the baseline model (0.796). The XGB model of AUROC 0.861 (95% CI 0.848-0.874) showed a higher discriminative performance than the DNN model of 0.833 (95% CI 0.819-0.848). The XGB and DNN models proved better reclassification than the baseline model with a positive net reclassification index. The XGB models were well-calibrated (Hosmer-Lemeshow test; P>.05); however, the DNN showed poor calibration power (Hosmer-Lemeshow test; P<.001). We further interpreted the nonlinear association between variables and critical care prediction.

Conclusions

Our study demonstrated that the performance of the XGB model using initial information at ED triage for predicting patients in need of critical care outperformed the conventional model with KTAS.

Registered nurses' understandings of emergency medical dispatch center work: A qualitative phenomenographic interview study

Non-urgent and urgent telephone nursing services are increasing globally, and phenomenographic research has shown that how work is understood may influence work performance. This descriptive study makes a qualitative inductive investigation of understandings of emergency medical dispatch center work among registered nurses. Twenty-four registered nurses at three mid Swedish emergency medical dispatch centers were interviewed. Analysis based on phenomenographic principles identified five categories in the interviews: (i) Assess, prioritize, direct, or refer; (ii) Facilitate ambulance nursing work; (iii) Perform nursing care; (iv) Always be available for the public; and (v) Have the person behind the patient in mind. The first constitutes the basis of the work. The second emphasizes cooperation with and support for the ambulance staff. The third entails remotely providing nursing care, whilst the fourth stresses serving the entire population. The fifth and most comprehensive way of understanding work involves having a holistic view of the person in need, including person-centered care. Provision of high-quality emergency medical dispatch center work involves all categories. Combined, they constitute a "work map," valuable for reflection, competence development, and introduction of new staff.

Machine learning model predicts short-term mortality among prehospital patients: A prospective development study from Finland

Aim

To show whether adding blood glucose to the National Early Warning Score (NEWS) parameters in a machine learning model predicts 30-day mortality more precisely than the standard NEWS in a prehospital setting.

Methods

In this study, vital sign data prospectively collected from 3632 unselected prehospital patients in June 2015 were used to compare the standard NEWS to random forest models for predicting 30-day mortality. The NEWS parameters and blood glucose levels were used to develop the random forest models. Predictive performance on an unknown patient population was estimated with a ten-fold stratified cross-validation method.

Results

All NEWS parameters and blood glucose levels were reported in 2853 (79%) eligible patients. Within 30 days after contact with ambulance staff, 97 (3.4%) of the analysed patients had died. The area under the receiver operating characteristic curve for the 30-day mortality of the evaluated models was 0.682 (95% confidence interval [CI], 0.619-0.744) for the standard NEWS, 0.735 (95% CI, 0.679-0.787) for the random forest-trained NEWS parameters only and 0.758 (95% CI, 0.705-0.807) for the random forest-trained NEWS parameters and blood glucose. The models predicted secondary outcomes similarly, but adding blood glucose into the random forest model slightly improved its performance in predicting short-term mortality.

Conclusions

Among unselected prehospital patients, a machine learning model including blood glucose and NEWS parameters had a fair performance in predicting 30-day mortality.

Prehospital identification of Covid-19: an observational study

Background

The novel coronavirus disease 2019 (Covid-19) pandemic has affected prehospital care systems across the world, but the prehospital presentation of affected patients and the extent to which prehospital care providers are able to identify them is not well characterized. In this study, we describe the presentation of Covid-19 patients in a Swedish prehospital care system, and asses the predictive value of Covid-19 suspicion as documented by dispatch and ambulance nurses.

Methods

Data for all patients with dispatch, ambulance, and hospital records between January 1–August 31, 2020 were extracted. A descriptive statistical analysis of patients with and without hospital-confirmed Covid-19 was performed. In a subset of records beginning from April 14, we assessed the sensitivity and specificity of documented Covid-19 suspicion in dispatch and ambulance patient care records.

Results

A total of 11,894 prehospital records were included, of which 481 had a primary hospital diagnosis code related to-, or positive test results for Covid-19. Covid-19-positive patients had considerably worse outcomes than patients with negative test results, with 30-day mortality rates of 24% vs 11%, but lower levels of prehospital acuity (e.g. emergent transport rates of 14% vs 22%). About half (46%) of Covid-19-positive patients presented to dispatchers with primary complaints typically associated with Covid-19. Six thousand seven hundred seventy-six records were included in the assessment of predictive value. Sensitivity was 76% (95% CI 71–80) and 82% (78–86) for dispatch and ambulance suspicion respectively, while specificities were 86% (85–87) and 78% (77–79).

Conclusions

While prehospital suspicion was strongly indicative of hospital-confirmed Covid-19, based on the sensitivity identified in this study, prehospital suspicion should not be relied upon as a single factor to rule out the need for isolation precautions. The data provided may be used to develop improved guidelines for identifying Covid-19 patients in the prehospital setting.

Random forest machine learning method outperforms prehospital National Early Warning Score for predicting one-day mortality: A retrospective study

Aim of the study

The National Early Warning Score (NEWS) is a validated method for predicting clinical deterioration in hospital wards, but its performance in prehospital settings remains controversial. Modern machine learning models may outperform traditional statistical analyses for predicting short-term mortality. Thus, we aimed to compare the mortality prediction accuracy of NEWS and random forest machine learning using prehospital vital signs.

Methods

In this retrospective study, all electronic ambulance mission reports between 2008 and 2015 in a single EMS system were collected. Adult patients (≥ 18 years) were included in the analysis. Random forest models with and without blood glucose were compared to the traditional NEWS for predicting one-day mortality. A ten-fold cross-validation method was applied to train and validate the random forest models.

Results

A total of 26,458 patients were included in the study of whom 278 (1.0%) died within one day of ambulance mission. The area under the receiver operating characteristic curve for one-day mortality was 0.836 (95% CI, 0.810−0.860) for NEWS, 0.858 (95% CI, 0.832−0.883) for a random forest trained with NEWS variables only and 0.868 (0.843−0.892) for a random forest trained with NEWS variables and blood glucose.

Conclusion

A random forest algorithm trained with NEWS variables was superior to traditional NEWS for predicting one-day mortality in adult prehospital patients, although the risk of selection bias must be acknowledged. The inclusion of blood glucose in the model further improved its predictive performance.

Artificial intelligence in emergency medicine: A scoping review

INTRODUCTION

Despite the growing investment in and adoption of artificial intelligence (AI) in medicine, the applications of AI in an emergency setting remain unclear. This scoping review seeks to identify available literature regarding the applications of AI in emergency medicine.

METHODS

The scoping review was conducted according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for scoping reviews using Medline-OVID, EMBASE, CINAHL, and IEEE, with a double screening and extraction process. The search included articles published until February 28, 2020. Articles were excluded if they did not self-classify as studying an AI intervention, were not relevant to the emergency department (ED), or did not report outcomes or evaluation.

RESULTS

Of the 1483 original database citations, 395 were eligible for full-text evaluation. Of these articles, a total of 150 were included in the scoping review. The majority of included studies were retrospective in nature (n = 124, 82.7%), with only 3 (2.0%) prospective controlled trials. We found 37 (24.7%) interventions aimed at improving diagnosis within the ED. Among the 150 studies, 19 (12.7%) focused on diagnostic imaging within the ED. A total of 16 (10.7%) studies were conducted in the out-of-hospital environment (eg, emergency medical services, paramedics) with the remainder occurring either in the ED or the trauma bay. Of the 24 (16%) studies that had human comparators, there were 12 (8%) studies in which AI interventions outperformed clinicians in at least 1 measured outcome.

CONCLUSION

AI-related research is rapidly increasing in emergency medicine. There are several promising AI interventions that can improve emergency care, particularly for acute radiographic imaging and prediction-based diagnoses. Higher quality evidence is needed to further assess both short- and long-term clinical outcomes.

Institution-Specific Machine Learning Models for Prehospital Assessment to Predict Hospital Admission: Prediction Model Development Study

Background

Although multiple prediction models have been developed to predict hospital admission to emergency departments (EDs) to address overcrowding and patient safety, only a few studies have examined prediction models for prehospital use. Development of institution-specific prediction models is feasible in this age of data science, provided that predictor-related information is readily collectable.

Objective

We aimed to develop a hospital admission prediction model based on patient information that is commonly available during ambulance transport before hospitalization.

Methods

Patients transported by ambulance to our ED from April 2018 through March 2019 were enrolled. Candidate predictors were age, sex, chief complaint, vital signs, and patient medical history, all of which were recorded by emergency medical teams during ambulance transport. Patients were divided into two cohorts for derivation (3601/5145, 70.0%) and validation (1544/5145, 30.0%). For statistical models, logistic regression, logistic lasso, random forest, and gradient boosting machine were used. Prediction models were developed in the derivation cohort. Model performance was assessed by area under the receiver operating characteristic curve (AUROC) and association measures in the validation cohort.

Results

Of 5145 patients transported by ambulance, including deaths in the ED and hospital transfers, 2699 (52.5%) required hospital admission. Prediction performance was higher with the addition of predictive factors, attaining the best performance with an AUROC of 0.818 (95% CI 0.792-0.839) with a machine learning model and predictive factors of age, sex, chief complaint, and vital signs. Sensitivity and specificity of this model were 0.744 (95% CI 0.716-0.773) and 0.745 (95% CI 0.709-0.776), respectively.

Conclusions

For patients transferred to EDs, we developed a well-performing hospital admission prediction model based on routinely collected prehospital information including chief complaints.

AutoScore: A Machine Learning-Based Automatic Clinical Score Generator and Its Application to Mortality Prediction Using Electronic Health Records

BACKGROUND

Risk scores can be useful in clinical risk stratification and accurate allocations of medical resources, helping health providers improve patient care. Point-based scores are more understandable and explainable than other complex models and are now widely used in clinical decision making. However, the development of the risk scoring model is nontrivial and has not yet been systematically presented, with few studies investigating methods of clinical score generation using electronic health records.

OBJECTIVE

This study aims to propose AutoScore, a machine learning-based automatic clinical score generator consisting of 6 modules for developing interpretable point-based scores. Future users can employ the AutoScore framework to create clinical scores effortlessly in various clinical applications.

METHODS

We proposed the AutoScore framework comprising 6 modules that included variable ranking, variable transformation, score derivation, model selection, score fine-tuning, and model evaluation. To demonstrate the performance of AutoScore, we used data from the Beth Israel Deaconess Medical Center to build a scoring model for mortality prediction and then compared the data with other baseline models using the receiver operating characteristic analysis. A software package in R 3.5.3 (R Foundation) was also developed to demonstrate the implementation of AutoScore.

RESULTS

Implemented on the data set with 44,918 individual admission episodes of intensive care, the AutoScore-created scoring models performed comparably well as other standard methods (ie, logistic regression, stepwise regression, least absolute shrinkage and selection operator, and random forest) in terms of predictive accuracy and model calibration but required fewer predictors and presented high interpretability and accessibility. The nine-variable, AutoScore-created, point-based scoring model achieved an area under the curve (AUC) of 0.780 (95% CI 0.764-0.798), whereas the model of logistic regression with 24 variables had an AUC of 0.778 (95% CI 0.760-0.795). Moreover, the AutoScore framework also drives the clinical research continuum and automation with its integration of all necessary modules.

CONCLUSIONS

We developed an easy-to-use, machine learning-based automatic clinical score generator, AutoScore; systematically presented its structure; and demonstrated its superiority (predictive performance and interpretability) over other conventional methods using a benchmark database. AutoScore will emerge as a potential scoring tool in various medical applications.

Enhancing predictions of patient conveyance using emergency call handler free text notes for unconscious and fainting incidents reported to the London Ambulance Service

OBJECTIVE

Pre-hospital emergency medical services use clinical decision support systems (CDSS) to triage calls. Call handlers often supplement this by making free text notes covering key incident information. We investigate whether machine learning approaches using features from such free text notes can improve prediction of unconscious patients who require conveyance.

MATERIALS AND METHODS

We analysed a subset of all London Ambulance Service calls that were triaged through the Medical Priority Dispatch System (MPDS) as involving an unconscious or fainting patient in 2018. We use and compare two machine learning algorithms: random forest (RF) and gradient boosting machine (GBM). For each incident, we predict whether the patient will be conveyed to a hospital emergency department or equivalent using as features 1) the MPDS code, 2) the free text notes and 3) the two together. We evaluate model performance using the area under the curve (AUC) metric. Given the imbalance of outcomes (patient conveyed 71 %, not conveyed 29 %), we also consider sensitivity and specificity.

RESULTS

Using only the MPDS code resulted in an AUC of 0.57. Using the text notes gave an improved AUC score of 0.63 and combining the two gave an AUC score of 0.64 (scores were similar for RF and GBM). GBM models scored better on sensitivity (0.93 vs 0.62 for RF in the combined model), but specificity was lower (0.17 vs. 0.56 for RF in the combined model).

CONCLUSIONS

Using information contained in the free text notes made by call handlers in combination with MPDS improves prediction of unconscious and fainting patients requiring conveyance to a hospital emergency department (or equivalent) when compared with machine learning models using MPDS codes only. This suggests there is some useful information in unstructured data captured by emergency call handlers that complements MPDS codes. Quantifying this gain can help inform emergency medical service policy when evaluating the decision to expand or augment existing CDSS.