Assessment of the Feasibility of automated, real-time clinical decision support in the emergency department using electronic health record data

Limitations in the use of automated mental status detection for clinical decision support

BACKGROUND

Clinical decision support (CDS) tools improve adherence to evidence-based practices but are dependent upon data quality in the electronic health record (EHR). Mental status is an integral component of many risk stratification scores, but it is not known whether EHR-measures of altered mental status are reliable. The Glasgow Coma Scale (GCS) is a measure of altered mentation that is widely adopted and entered in the EHR in structured format. We sought to determine the accuracy GCS < 15 as an EHR-measure of altered mentation compared to ED provider documentation.

METHODS

In patients presenting to an academic Emergency Department (ED) with pneumonia we abstracted GCS values entered by nurses during routine care and in a randomly selected subset manually reviewed provider documentation for evidence of altered mental status. We defined eConfusion as present if GCS < 15 at any point during the ED encounter. We then calculated the CURB-65 score and corresponding suggested disposition using each method. Performance of eConfusion and corresponding CURB-65 compared to manual versions was measured using agreement (Cohen's K), sensitivity, and specificity.

RESULTS

Among 300 randomly selected encounters, 47 (16 %) had eConfusion present and 46 (15 %) had evidence of altered mental status in provider documentation with Cohen's K 0.73. eConfusion had 78 % sensitivity and 96 % specificity for provider documented altered mental status. When input into CURB-65 to recommend inpatient disposition, eConfusion had 95 % sensitivity, and recommended discordant disposition for 3 %.

CONCLUSIONS

There was modest agreement between eConfusion and provider documentation of altered mental status. eConfusion had good specificity but low sensitivity which resulted in under-estimation of the CURB-65 score and occasional inappropriate disposition recommendations compared to provider documentation. These data do not support the use of GCS as a measure for altered mentation for use in CDS tools in the ED.

Clinical risk calculators informing the decision to admit: A methodologic evaluation and assessment of applicability

INTRODUCTION

Clinical prediction and decision tools that generate outcome-based risk stratification and/or intervention recommendations are prevalent. Appropriate use and validity of these tools, especially those that inform complex clinical decisions, remains unclear. The objective of this study was to assess the methodologic quality and applicability of clinical risk scoring tools used to guide hospitalization decision-making.

METHODS

In February 2021, a comprehensive search was performed of a clinical calculator online database (mdcalc.com) that is publicly available and well-known to clinicians. The primary reference for any calculator tool informing outpatient versus inpatient disposition was considered for inclusion. Studies were restricted to the adult, acute care population. Those focused on obstetrics/gynecology or critical care admission were excluded. The Wasson-Laupacis framework of methodologic standards for clinical prediction rules was applied to each study.

RESULTS

A total of 22 calculators provided hospital admission recommendations for 9 discrete medical conditions using adverse events (14/22), mortality (6/22), or confirmatory diagnosis (2/22) as outcomes of interest. The most commonly met methodologic standards included mathematical technique description (22/22) and clinical sensibility (22/22) and least commonly met included reproducibility of the rule (1/22) and measurement of effect on clinical use (1/22). Description of the studied population was often lacking, especially patient race/ethnicity (2/22) and mental or behavioral health (0/22). Only one study reported any item related to social determinants of health.

CONCLUSION

Studies commonly do not meet rigorous methodologic standards and often fail to report pertinent details that would guide applicability. These clinical tools focus primarily on specific disease entities and clinical variables, missing the breadth of information necessary to make a disposition determination and raise significant validation and generalizability concerns.

Integration of Risk Scores and Integration Capability in Electronic Patient Records

BACKGROUND

Digital availability of patient data is continuously improving with the increasing implementation of electronic patient records in physician practices. The emergence of digital health data defines new fields of application for data analytics applications, which in turn offer extensive options of using data. Common areas of data analytics applications include decision support, administration, and fraud detection. Risk scores play an important role in compiling algorithms that underlay tools for decision support.

OBJECTIVES

This study aims to identify the current state of risk score integration and integration capability in electronic patient records for cardiovascular disease and diabetes in German primary care practices.

METHODS

We developed an evaluation framework to determine the current state of risk score integration and future integration options for four cardiovascular disease risk scores (arriba, Pooled Cohort Equations, QRISK3, and Systematic Coronary Risk Evaluation) and two diabetes risk scores (Finnish Diabetes Risk Score and German Diabetes Risk Score). We then used this framework to evaluate the integration of risk scores in common practice software solutions by examining the software and inquiring the respective software contact person.

RESULTS

Our evaluation showed that the most widely integrated risk score is arriba, as recommended by German medical guidelines. Every software version in our sample provided either an interface to arriba or the option to implement one. Our assessment of integration capability revealed a more nuanced picture. Results on data availability were mixed. Each score contains at least one variable, which requires laboratory diagnostics. Our analysis of data standardization showed that only one score documented all variables in a standardized way.

CONCLUSION

Our assessment revealed that the current state of risk score integration in physician practice software is rather low. Integration capability currently faces some obstacles. Future research should develop a comprehensive framework that considers the reasonable integration of risk scores into practice workflows, disease prevention programs, and the awareness of physicians and patients.

Discovering Composite Lifestyle Biomarkers With Artificial Intelligence From Clinical Studies to Enable Smart eHealth and Digital Therapeutic Services

Discovery of biomarkers is a continuous activity of the research community in the clinical domain that recently shifted its focus toward digital, non-traditional biomarkers that often use physiological, psychological, social, and environmental data to derive an intermediate biomarker. Such biomarkers, by triggering smart services, can be used in a clinical trial framework and eHealth or digital therapeutic services. In this work, we discuss the APACHE trial for determining the quality of life (QoL) of cervical cancer patients and demonstrate how we are discovering a biomarker for this therapeutic area that predicts significant QoL variations. To this extent, we present how real-world data can unfold a big potential for detecting the cervical cancer QoL biomarker and how it can be used for novel treatments. The presented methodology, derived in APACHE, is introduced by Healthentia eClinical solution, and it is beginning to be used in several clinical studies.

Factors predicting failure of internal fixations of fractures of the lower limbs: a prospective cohort study

Background

We assessed predictive factors of patients with fractures of the lower extremities caused by trauma. We examined which factors are associated with an increased risk of failure. Furthermore, the predictive factors were set into context with other long-term outcomes, concrete pain and physical functioning.

Methods

We performed a prospective cohort study at a single level I trauma center. We enrolled patients with traumatic fractures of the lower extremities treated with internal fixation from April 2017 to July 2018. We evaluated the following predictive factors: age, gender, diabetes, smoking status, obesity, open fractures and peripheral arterial diseases. The primary outcome was time to failure (nonunion, implant failure or reposition). Secondary outcomes were pain and physical functioning measured 6 months after initial surgery. For the analysis of the primary outcome, we used a stratified (according fracture location) Cox proportional hazard regression model.

Results

We included 204 patients. Overall, we observed failure in 33 patients (16.2 %). Most of the failures occurred within the first 3 months. Obesity and open fractures were associated with an increased risk of failure and decreased physical functioning. None of the predictors showed an association with pain. Age, female gender and smoking of more than ≥ 10 package years increased failure risk numerically but statistical uncertainty was high.

Conclusions

We found that obesity and open fractures were strongly associated with an increased risk of failure. These predictors seem promising candidates to be included in a risk prediction model and can be considered as a good start for clinical decision making across different types of fractures at the lower limbs. However, large heterogeneity regarding the other analyzed predictors suggests that “simple” models might not be adequate for a precise personalized risk estimation and that computer-based models incorporating a variety of detailed information (e.g. pattern of injury, x-ray and clinical data) and their interrelation may be required to significantly increase prediction precision.

Trial registration

NCT03091114.

Risk prediction of delirium in hospitalized patients using machine learning: An implementation and prospective evaluation study

OBJECTIVE

Machine learning models trained on electronic health records have achieved high prognostic accuracy in test datasets, but little is known about their embedding into clinical workflows. We implemented a random forest-based algorithm to identify hospitalized patients at high risk for delirium, and evaluated its performance in a clinical setting.

MATERIALS AND METHODS

Delirium was predicted at admission and recalculated on the evening of admission. The defined prediction outcome was a delirium coded for the recent hospital stay. During 7 months of prospective evaluation, 5530 predictions were analyzed. In addition, 119 predictions for internal medicine patients were compared with ratings of clinical experts in a blinded and nonblinded setting.

RESULTS

During clinical application, the algorithm achieved a sensitivity of 74.1% and a specificity of 82.2%. Discrimination on prospective data (area under the receiver-operating characteristic curve = 0.86) was as good as in the test dataset, but calibration was poor. The predictions correlated strongly with delirium risk perceived by experts in the blinded (r = 0.81) and nonblinded (r = 0.62) settings. A major advantage of our setting was the timely prediction without additional data entry.

DISCUSSION

The implemented machine learning algorithm achieved a stable performance predicting delirium in high agreement with expert ratings, but improvement of calibration is needed. Future research should evaluate the acceptance of implemented machine learning algorithms by health professionals.

CONCLUSIONS

Our study provides new insights into the implementation process of a machine learning algorithm into a clinical workflow and demonstrates its predictive power for delirium.

Real-World Evidence, Causal Inference, and Machine Learning

The current focus on real world evidence (RWE) is occurring at a time when at least two major trends are converging. First, is the progress made in observational research design and methods over the past decade. Second, the development of numerous large observational healthcare databases around the world is creating repositories of improved data assets to support observational research. OBJECTIVE: This paper examines the implications of the improvements in observational methods and research design, as well as the growing availability of real world data for the quality of RWE. These developments have been very positive. On the other hand, unstructured data, such as medical notes, and the sparcity of data created by merging multiple data assets are not easily handled by traditional health services research statistical methods. In response, machine learning methods are gaining increased traction as potential tools for analyzing massive, complex datasets. CONCLUSIONS: Machine learning methods have traditionally been used for classification and prediction, rather than causal inference. The prediction capabilities of machine learning are valuable by themselves. However, using machine learning for causal inference is still evolving. Machine learning can be used for hypothesis generation, followed by the application of traditional causal methods. But relatively recent developments, such as targeted maximum likelihood methods, are directly integrating machine learning with causal inference.

Signs and Symptoms That Rule out Community-Acquired Pneumonia in Outpatient Adults: A Systematic Review and Meta-Analysis

BACKGROUND

A systematic review of clinical decision rules to identify patients at low risk for community-acquired pneumonia (CAP) has not been previously presented in the literature.

METHODS

A systematic review of MEDLINE for prospective studies that used at least 2 signs, symptoms, or point-of-care tests to determine the likelihood of CAP. We included studies that enrolled adults and adolescents in the outpatient setting where all or a random sample of patients received a chest radiograph as the reference standard. We excluded retrospective studies and studies that recruited primarily patients with hospital-acquired CAP.

RESULTS

Our search identified 974 articles, 12 of which were included in the final analysis. The simple heuristic of normal vital signs (temperature, respiratory rate, and heart rate) to identify patients at low risk for CAP was reported by 4 studies and had a summary estimate of the negative likelihood ratio (LR-) of 0.24 (95% CI, 0.17 to 0.34) and a sensitivity of 0.89 (95% CI, 0.79 to 0.94). The simple heuristic of normal vital signs combined with a normal pulmonary examination to identify patients at low risk for CAP was reported by 3 studies, and had a summary estimate of LR- of 0.10 (95% CI, 0.07 to 0.13) with an area under the receiver operating characteristic curve of 0.92.

CONCLUSIONS

Adults with an acute respiratory infection who have normal vital signs and a normal pulmonary examination are very unlikely to have CAP. Given a baseline CAP risk of 4%, these patients have only a 0.4% likelihood of CAP.

Man vs machine in emergency medicine - a study on the effects of manual and automatic vital sign documentation on data quality and perceived workload, using observational paired sample data and questionnaires

BACKGROUND

Emergency medicine is characterized by a high patient flow where timely decisions are essential. Clinical decision support systems have the potential to assist in such decisions but will be dependent on the data quality in electronic health records which often is inadequate. This study explores the effect of automated documentation of vital signs on data quality and workload.

METHODS

An observational study of 200 vital sign measurements was performed to evaluate the effects of manual vs automatic documentation on data quality. Data collection using questionnaires was performed to compare the workload on wards using manual or automatic documentation.

RESULTS

In the automated documentation time to documentation was reduced by 6.1 min (0.6 min vs 7.7 min, p <  0.05) and completeness increased (98% vs 95%, p <  0.05). Regarding workflow temporal demands were lower in the automatic documentation workflow compared to the manual group (50 vs 23, p <  0.05). The same was true for frustration level (64 vs 33, p <  0.05). The experienced reduction in temporal demands was in line with the anticipated, whereas the experienced reduction in frustration was lower than the anticipated (27 vs 54, p < 0.05).

DISCUSSION

The study shows that automatic documentation will improve the currency and the completeness of vital sign data in the Electronic Health Record while reducing workload regarding temporal demands and experienced frustration. The study also shows that these findings are in line with staff anticipations but indicates that the anticipations on the reduction of frustration may be exaggerated among the staff. The open-ended answers indicate that frustration focus will change from double documentation of vital signs to technical aspects of the automatic documentation system.