Training machine learning models to predict 30-day mortality in patients discharged from the emergency department: a retrospective, population-based registry study

Automating detection of diagnostic error of infectious diseases using machine learning

Diagnostic error, a cause of substantial morbidity and mortality, is largely discovered and evaluated through self-report and manual review, which is costly and not suitable to real-time intervention. Opportunities exist to leverage electronic health record data for automated detection of potential misdiagnosis, executed at scale and generalized across diseases. We propose a novel automated approach to identifying diagnostic divergence considering both diagnosis and risk of mortality. Our objective was to identify cases of emergency department infectious disease misdiagnoses by measuring the deviation between predicted diagnosis and documented diagnosis, weighted by mortality. Two machine learning models were trained for prediction of infectious disease and mortality using the first 24h of data. Charts were manually reviewed by clinicians to determine whether there could have been a more correct or timely diagnosis. The proposed approach was validated against manual reviews and compared using the Spearman rank correlation. We analyzed 6.5 million ED visits and over 700 million associated clinical features from over one hundred emergency departments. The testing set performances of the infectious disease (Macro F1 = 86.7, AUROC 90.6 to 94.7) and mortality model (Macro F1 = 97.6, AUROC 89.1 to 89.1) were in expected ranges. Human reviews and the proposed automated metric demonstrated positive correlations ranging from 0.231 to 0.358. The proposed approach for diagnostic deviation shows promise as a potential tool for clinicians to find diagnostic errors. Given the vast number of clinical features used in this analysis, further improvements likely need to either take greater account of data structure (what occurs before when) or involve natural language processing. Further work is needed to explain the potential reasons for divergence and to refine and validate the approach for implementation in real-world settings.

Development and validation of prognostic machine learning models for short- and long-term mortality among acutely admitted patients based on blood tests

Several scores predicting mortality at the emergency department have been developed. However, all with shortcomings either simple and applicable in a clinical setting, with poor performance, or advanced, with high performance, but clinically difficult to implement. This study aimed to explore if machine learning algorithms could predict all-cause short- and long-term mortality based on the routine blood test collected at admission.

METHODS

We analyzed data from a retrospective cohort study, including patients > 18 years admitted to the Emergency Department (ED) of Copenhagen University Hospital Hvidovre, Denmark between November 2013 and March 2017. The primary outcomes were 3-, 10-, 30-, and 365-day mortality after admission. PyCaret, an automated machine learning library, was used to evaluate the predictive performance of fifteen machine learning algorithms using the area under the receiver operating characteristic curve (AUC).

RESULTS

Data from 48,841 admissions were analyzed, of these 34,190 (70%) were randomly divided into training data, and 14,651 (30%) were in test data. Eight machine learning algorithms achieved very good to excellent results of AUC on test data in a of range 0.85-0.93. In prediction of short-term mortality, lactate dehydrogenase (LDH), leukocyte counts and differentials, Blood urea nitrogen (BUN) and mean corpuscular hemoglobin concentration (MCHC) were the best predictors, whereas prediction of long-term mortality was favored by age, LDH, soluble urokinase plasminogen activator receptor (suPAR), albumin, and blood urea nitrogen (BUN).

CONCLUSION

The findings suggest that measures of biomarkers taken from one blood sample during admission to the ED can identify patients at high risk of short-and long-term mortality following emergency admissions.

The prediction of 24-h mortality by the respiratory rate and oxygenation index compared with National Early Warning Score in emergency department patients: an observational study

BACKGROUND

The ROX index combines respiratory rate and oxygenation to predict the response to oxygen therapy in pneumonia. It is calculated by dividing the patient's oxygen saturation, by the inspired oxygen concentration, and then by the respiratory rate (e.g. 95%/0.21/16 = 28). Since this index includes the most essential physiological variables to detect deterioration, it may be a helpful risk tool in the emergency department (ED). Although small studies suggest it can predict early mortality, no large study has compared it with the National Early Warning Score (NEWS), the most widely validated risk score for death within 24 h.

AIM

The aim of this study was to compare the ability of the ROX index with the NEWS to predict mortality within 24 h of arrival at the hospital.

METHODS

This was a retrospective observational multicentre analysis of data in the Netherlands Emergency Department Evaluation Database (NEED) on 270 665 patients attending four participating Dutch EDs. The ROX index and NEWS were determined on ED arrival and prior to ED treatment.

RESULTS

The risk of death within 24 h increased with falling ROX and rising NEWS values. The area under the receiving operating characteristic curves for 24-h mortality of NEWS was significantly higher than for the ROX index [0.92; 95% confidence interval (CI), 0.91-0.92 versus 0.87; 95% CI, 0.86-0.88; P  < 0.01]. However, the observed and predicted mortality by the ROX index was identical to mortality of 5%, after which mortality was underestimated. In contrast, up to a predicted 24-h mortality of 3% NEWS slightly underestimates mortality, and above this level over-estimates it. The standardized net benefit of ROX is slightly higher than NEWS up to a predicted 24-h mortality of 3%.

CONCLUSION

The prediction of 24-h mortality by the ROX index is more accurate than NEWS for most patients likely to be encountered in the ED. ROX may be used as a first screening tool in the ED.

Ethical considerations in implementing AI for mortality prediction in the emergency department: Linking theory and practice

Background

Artificial intelligence (AI) is predicted to be a solution for improving healthcare, increasing efficiency, and saving time and recourses. A lack of ethical principles for the use of AI in practice has been highlighted by several stakeholders due to the recent attention given to it. Research has shown an urgent need for more knowledge regarding the ethical implications of AI applications in healthcare. However, fundamental ethical principles may not be sufficient to describe ethical concerns associated with implementing AI applications.

Objective

The aim of this study is twofold, (1) to use the implementation of AI applications to predict patient mortality in emergency departments as a setting to explore healthcare professionals' perspectives on ethical issues in relation to ethical principles and (2) to develop a model to guide ethical considerations in AI implementation in healthcare based on ethical theory.

Methods

Semi-structured interviews were conducted with 18 participants. The abductive approach used to analyze the empirical data consisted of four steps alternating between inductive and deductive analyses.

Results

Our findings provide an ethical model demonstrating the need to address six ethical principles (autonomy, beneficence, non-maleficence, justice, explicability, and professional governance) in relation to ethical theories defined as virtue, deontology, and consequentialism when AI applications are to be implemented in clinical practice.

Conclusions

Ethical aspects of AI applications are broader than the prima facie principles of medical ethics and the principle of explicability. Ethical aspects thus need to be viewed from a broader perspective to cover different situations that healthcare professionals, in general, and physicians, in particular, may face when using AI applications in clinical practice.

Long-term exposure to particulate matter was associated with increased dementia risk using both traditional approaches and novel machine learning methods

Air pollution exposure has been linked to various diseases, including dementia. However, a novel method for investigating the associations between air pollution exposure and disease is lacking. The objective of this study was to investigate whether long-term exposure to ambient particulate air pollution increases dementia risk using both the traditional Cox model approach and a novel machine learning (ML) with random forest (RF) method. We used health data from a national population-based cohort in Taiwan from 2000 to 2017. We collected the following ambient air pollution data from the Taiwan Environmental Protection Administration (EPA): fine particulate matter (PM_2.5) and gaseous pollutants, including sulfur dioxide (SO₂), carbon monoxide (CO), ozone (O₃), nitrogen oxide (NO_x), nitric oxide (NO), and nitrogen dioxide (NO₂). Spatiotemporal-estimated air quality data calculated based on a geostatistical approach, namely, the Bayesian maximum entropy method, were collected. Each subject's residential county and township were reviewed monthly and linked to air quality data based on the corresponding township and month of the year for each subject. The Cox model approach and the ML with RF method were used. Increasing the concentration of PM_2.5 by one interquartile range (IQR) increased the risk of dementia by approximately 5% (HR = 1.05 with 95% CI = 1.04-1.05). The comparison of the performance of the extended Cox model approach with the RF method showed that the prediction accuracy was approximately 0.7 by the RF method, but the AUC was lower than that of the Cox model approach. This national cohort study over an 18-year period provides supporting evidence that long-term particulate air pollution exposure is associated with increased dementia risk in Taiwan. The ML with RF method appears to be an acceptable approach for exploring associations between air pollutant exposure and disease.

Combining machine learning and conventional statistical approaches for risk factor discovery in a large cohort study

We present a simple and efficient hypothesis-free machine learning pipeline for risk factor discovery that accounts for non-linearity and interaction in large biomedical databases with minimal variable pre-processing. In this study, mortality models were built using gradient boosting decision trees (GBDT) and important predictors were identified using a Shapley values-based feature attribution method, SHAP values. Cox models controlled for false discovery rate were used for confounder adjustment, interpretability, and further validation. The pipeline was tested using information from 502,506 UK Biobank participants, aged 37–73 years at recruitment and followed over seven years for mortality registrations. From the 11,639 predictors included in GBDT, 193 potential risk factors had SHAP values ≥ 0.05, passed the correlation test, and were selected for further modelling. Of the total variable importance summed up, 60% was directly health related, and baseline characteristics, sociodemographics, and lifestyle factors each contributed about 10%. Cox models adjusted for baseline characteristics, showed evidence for an association with mortality for 166 out of the 193 predictors. These included mostly well-known risk factors (e.g., age, sex, ethnicity, education, material deprivation, smoking, physical activity, self-rated health, BMI, and many disease outcomes). For 19 predictors we saw evidence for an association in the unadjusted but not adjusted analyses, suggesting bias by confounding. Our GBDT-SHAP pipeline was able to identify relevant predictors ‘hidden’ within thousands of variables, providing an efficient and pragmatic solution for the first stage of hypothesis free risk factor identification.

Improving Machine Learning 30-Day Mortality Prediction by Discounting Surprising Deaths

BACKGROUND

Machine learning (ML) is an emerging tool for predicting need of end-of-life discussion and palliative care, by using mortality as a proxy. But deaths, unforeseen by emergency physicians at time of the emergency department (ED) visit, might have a weaker association with the ED visit.

OBJECTIVES

To develop an ML algorithm that predicts unsurprising deaths within 30 days after ED discharge.

METHODS

In this retrospective registry study, we included all ED attendances within the Swedish region of Halland in 2015 and 2016. All registered deaths within 30 days after ED discharge were classified as either "surprising" or "unsurprising" by an adjudicating committee with three senior specialists in emergency medicine. ML algorithms were developed for the death subclasses by using Logistic Regression (LR), Random Forest (RF), and Support Vector Machine (SVM).

RESULTS

Of all 30-day deaths (n = 148), 76% (n = 113) were not surprising to the adjudicating committee. The most common diseases were advanced stage cancer, multidisease/frailty, and dementia. By using LR, RF, and SVM, mean area under the receiver operating characteristic curve (ROC-AUC) of unsurprising deaths in the test set were 0.950 (SD 0.008), 0.944 (SD 0.007), and 0.949 (SD 0.007), respectively. For all mortality, the ROC-AUCs for LR, RF, and SVM were 0.924 (SD 0.012), 0.922 (SD 0.009), and 0.931 (SD 0.008). The difference in prediction performance between all and unsurprising death was statistically significant (P < .001) for all three models.

CONCLUSION

In patients discharged to home from the ED, three-quarters of all 30-day deaths did not surprise an adjudicating committee with emergency medicine specialists. When only unsurprising deaths were included, ML mortality prediction improved significantly.

A predictive model for serious adverse events in adults with acute poisoning in prehospital and hospital care

OBJECTIVE

The objective of this study was to design a risk model with variables determined before hospital arrival to predict the risk of serious adverse events in patients with acute poisoning.

METHODS

A preliminary prospective, multicentre cohort study of adults with prehospital diagnosis of acute intoxication was conducted. The study was carried out in the Public Health System of the Community of Castilla-Leon (Spain), including seven advanced life support units and five hospitals, between April 1, 2018, and June 30, 2019. People aged >18 years with a main prehospital diagnosis of acute poisoning admitted to a referral hospital on advanced life support were included. The main outcome measure was prehospital and hospital serious adverse events in patients with acute poisoning.

RESULTS

We included 221 patients, with a median age of 47 years (interquartile range: 33-61). The most frequent cause of poisoning was psychopharmaceuticals (111 cases, 49.8%): 38 (17.2%) patients had a serious adverse event, with a hospital mortality of 4.1% (nine cases) in the 30 days after the index event. The final model included age ≥65 years (odds ratio [OR]: 9.59, 95% confidence interval [CI]: 3.48-26.45; p < 0.001), oxygen saturation/fraction of inspired oxygen index ≤300 (OR: 15.03, 95% CI: 5.74-39.33; p < 0.001), and point-of-care lactate ≥4 mmol/L (OR: 7.68, 95% CI: 2.88-20.45; p < 0.001). The poisoning Early Warning Score was constructed from these three variables, and 1 point was assigned to each variable. The area under the curve of the score was 0.896 (95% CI: 0.82-0.96; p < 0.001).

CONCLUSIONS

The poisoning Early Warning Score may help in decision-making and promote early identification of high-risk patients with acute poisoning in the prehospital context.

A simple prognostic score predicts one-year mortality of alert and calm emergency department patients: A prospective two-center observational study

STUDY OBJECTIVE

To derive and validate a prognostic score to predict 1-year mortality using vital signs, mobility and other variables that are readily available at the bedside at no additional cost.

METHODS

Post hoc analysis of two independent prospective observational studies in two emergency departments, one in Denmark and the other in Switzerland.

PARTICIPANTS

Alert and calm emergency department patients.

MEASUREMENTS

The prediction of mortality from presentation to 365 days by vital signs, mobility and other variables that are readily available at the bedside at no additional cost.

RESULTS

One thousand six hundred and eighteen alert and calm patients were in the Danish cohort and 1331 in the Swiss cohort. Logistic regression identified age >68 years, abnormal vital signs, impaired mobility and the decision to admit as significant predictors of 365-day mortality. A simple prognostic score awarded one point to each of these predictors. Less than two of these predictors were present in 45.6% of patients, and only 0.4% of these patients died within a year. If two or more of these predictors were present, 365-day mortality increased exponentially.

CONCLUSION

Age >68 years, the decision for hospital admission, any vital sign abnormality at presentation and impaired mobility at presentation are equally powerful predictors of 1-year mortality in alert and calm emergency department patients. If validated by others these predictors could be used to discharge patients with confidence since nearly half of these patients had less than two predictors and none of them died within 30 days. However, when two or more predictors were present 365-day mortality increased exponentially.