The Effect of Patient-Specific Drug-Drug Interaction Alerting on the Frequency of Alerts: A Pilot Study

Leveraging explainable artificial intelligence to optimize clinical decision support

OBJECTIVE

To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches.

METHODS

We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful.

RESULTS

The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues.

CONCLUSION

We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.

Barriers to Adoption of Tailored Drug-Drug Interaction Clinical Decision Support

OBJECTIVE

Despite the benefits of the tailored drug-drug interaction (DDI) alerts and the broad dissemination strategy, the uptake of our tailored DDI alert algorithms that are enhanced with patient-specific and context-specific factors has been limited. The goal of the study was to examine barriers and health care system dynamics related to implementing tailored DDI alerts and identify the factors that would drive optimization and improvement of DDI alerts.

METHODS

We employed a qualitative research approach, conducting interviews with a participant interview guide framed based on Proctor's taxonomy of implementation outcomes and informed by the Theoretical Domains Framework. Participants included pharmacists with informatics roles within hospitals, chief medical informatics officers, and associate medical informatics directors/officers. Our data analysis was informed by the technique used in grounded theory analysis, and the reporting of open coding results was based on a modified version of the Safety-Related Electronic Health Record Research Reporting Framework.

RESULTS

Our analysis generated 15 barriers, and we mapped the interconnections of these barriers, which clustered around three entities (i.e., users, organizations, and technical stakeholders). Our findings revealed that misaligned interests regarding DDI alert performance and misaligned expectations regarding DDI alert optimizations among these entities within health care organizations could result in system inertia in implementing tailored DDI alerts.

CONCLUSION

Health care organizations primarily determine the implementation and optimization of DDI alerts, and it is essential to identify and demonstrate value metrics that health care organizations prioritize to enable tailored DDI alert implementation. This could be achieved via a multifaceted approach, such as partnering with health care organizations that have the capacity to adopt tailored DDI alerts and identifying specialists who know users' needs, liaise with organizations and vendors, and facilitate technical stakeholders' work. In the future, researchers can adopt the systematic approach to study tailored DDI implementation problems from other system perspectives (e.g., the vendors' system).

Meaningful time-related aspects of alerts in Clinical Decision Support Systems. A unified framework

Alerts are a common functionality of clinical decision support systems (CDSSs). Although they have proven to be useful in clinical practice, the alert burden can lead to alert fatigue and significantly reduce their usability and acceptance. Based on a literature review, we propose a unified framework consisting of a set of meaningful timestamps that allows the use of state-of-the-art measures for alert burden, such as alert dwell time, alert think time, and response time. In addition, it can be used to investigate other measures that could be relevant as regards dealing with this problem. Furthermore, we provide a case study concerning three different types of alerts to which the framework was successfully applied. We consider that our framework can easily be adapted to other CDSSs and that it could be useful for dealing with alert burden measurement thus contributing to its appropriate management.

Using AI-generated suggestions from ChatGPT to optimize clinical decision support

OBJECTIVE

To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions.

METHODS

We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy.

RESULTS

Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy.

CONCLUSION

AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.

Assessing the Value of ChatGPT for Clinical Decision Support Optimization

Objective

To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions.

Methods

We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy.

Results

Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy.

Conclusion

AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.

Potential drug-drug interactions in drug therapy for older adults with chronic coronary syndrome at hospital discharge: A real-world study

Introduction: Polypharmacy are commonly observed among older adults with cardiovascular disease. However, multiple medications lead to increased risk of drug-drug interactions (DDIs). Therefore, identification and prevention actions related to harmful DDIs are expected in older adults. The study aimed to describe the prevalence of potential DDIs (pDDIs) in discharge prescriptions among older adults with chronic coronary syndrome (CCS).

Methods: A single-center cross-sectional study was performed in a tertiary public hospital in Beijing, China. CCS patients aged 65 years and above who were admitted to cardiology wards over a 3-month period and alive at discharge were included. Electronic medical records and discharge prescriptions were reviewed. pDDIs were evaluated through the Lexi-Interact online.

Results: pDDIs were identified in 72.9% of the 402 individuals (n = 293). A total of 864 pDDIs were obtained. 72.1% of patients were found with C DDIs (n = 290) and 20.3% were categorized in D and X DDIs (n = 82). The only X DDI was between cyclosporine and atorvastatin. Under category D, glycemia alterations within antidiabetics and increased chances of bleeding with antithrombotic were the most common. Concomitant use of clopidogrel and calcium channel blockers was a frequent situation within category C, followed by synergic blood pressure lowering agents and increased rosuvastatin concentration induced by clopidogrel.

Conclusion: DDIs exposure was common in older CCS. DDIs screening tools should be introduced to alert potential adverse effects. Prescribers need to rigorously review or modulate therapies to prevent DDI-related adverse outcomes. Clinical pharmacists should be more involved in complex drug regimen management.

Prescribers’ reported acceptance and use of drug-drug interaction alerts: An Australian survey

Drug-drug interaction (DDI) alerts are frequently included in electronic medical record (eMR) systems to provide users with relevant information and guidance at the point of care. In this study, we aimed to examine views of DDI alerts among prescribers, including junior doctors, registrars and senior doctors, across Australia. A validated survey for assessing prescribers’ reported acceptance and use of DDI alerts was distributed among researcher networks and in newsletters. Fifty useable responses were received, more than half ( n = 28) from senior doctors. Prescribers at all levels expected DDI alerts to improve performance but junior doctors reported that this was at a high cost, with respect to time and effort. Senior doctors and registrars reported rarely reading alerts and rarely changing prescribing decisions based on alerts. Respondents identified a number of problems with current alerts including limited relevance, repetition, and poor design, highlighting some clear areas for alert improvement.

Overall performance of a drug-drug interaction clinical decision support system: quantitative evaluation and end-user survey

BACKGROUND

Clinical decision support systems are implemented in many hospitals to prevent medication errors and associated harm. They are however associated with a high burden of false positive alerts and alert fatigue. The aim of this study was to evaluate a drug-drug interaction (DDI) clinical decision support system in terms of its performance, uptake and user satisfaction and to identify barriers and opportunities for improvement.

METHODS

A quantitative evaluation and end-user survey were performed in a large teaching hospital. First, very severe DDI alerts generated between 2019 and 2021 were evaluated retrospectively. Data collection comprised alert burden, override rates, the number of alert overrides reviewed by pharmacists and the resulting pharmacist recommendations as well as their acceptance rate. Second, an e-survey was carried out among prescribers to assess satisfaction, usefulness and relevance of DDI alerts as well as reasons for overriding.

RESULTS

A total of 38,409 very severe DDI alerts were generated, of which 88.2% were overridden by the prescriber. In 3.2% of reviewed overrides, a recommendation by the pharmacist was provided, of which 79.2% was accepted. False positive alerts were caused by a too broad screening interval and lack of incorporation of patient-specific characteristics, such as QTc values. Co-prescribing of a non-vitamin K oral anticoagulant and a low molecular weight heparin accounted for 49.8% of alerts, of which 92.2% were overridden. In 88 (1.1%) of these overridden alerts, concurrent therapy was still present. Despite the high override rate, the e-survey revealed that the DDI clinical decision support system was found useful by prescribers.

CONCLUSIONS

Identified barriers were the lack of DDI-specific screening intervals and inclusion of patient-specific characteristics, both leading to a high number of false positive alerts and risk for alert fatigue. Despite these barriers, the added value of the DDI clinical decision support system was recognized by prescribers. Hence, integration of DDI-specific screening intervals and patient-specific characteristics is warranted to improve the performance of the DDI software.

Impact of an opioid prescribing alert system on patients with posttraumatic stress disorder

BACKGROUND AND OBJECTIVES

Posttraumatic stress disorder (PTSD) is associated with higher rates of chronic pain and increased risk of developing Opioid use disorder. This paper evaluates the impact of PRIMUM, an electronic health record-embedded (EHR) clinical decision support intervention on opioid prescribing patterns for patients with diagnosis of PTSD.

METHODS

Inpatient, emergency department (ED), urgent care, and outpatient encounters with ICD-10 codes F43.1 (PTSD), F43.10 (PTSD, unspecified), F43.11 (PTSD, acute), and F43.12 (PTSD, chronic) at Atrium Health between 1/1/2016 and 12/29/2018 were included in the study.

RESULTS

A total of 3121 patients with a diagnosis of PTSD were seen in 37,443 encounters during the study period. Ten percent (n = 3761) of the encounters resulted in prescriptions for opioids and PRIMUM alerts were triggered in 1488 of these encounters. These alerts resulted in "decision influenced" for 17% of patients (n = 255) or no prescriptions for opioids or benzodiazepines for 5.8% (n = 86). The majority of the prescriptions were below 50 Morphine milligram equivalents (MME)/day, but there were 570 (15.5%) prescriptions for doses of 50-90 MME and 721 (19.6%) prescriptions for >90 MME/day.

DISCUSSION AND CONCLUSION

The PRIMUM alert system helps improve patient safety. PRIMUM affected clinician decisions 17% of the time, and the effect was greater in patients with opioid overdose history and those presenting for early refills.

SCIENTIFIC SIGNIFICANCE

The effectiveness of clinical support interventions for opioid prescribing for patients with PTSD has not been documented previously. Our findings provide novel evidence that the EHR can be used to improve patient safety among patients with PTSD in the context of substance use.

Cold-Start Problems in Data-Driven Prediction of Drug-Drug Interaction Effects

Combining drugs, a phenomenon often referred to as polypharmacy, can induce additional adverse effects. The identification of adverse combinations is a key task in pharmacovigilance. In this context, in silico approaches based on machine learning are promising as they can learn from a limited number of combinations to predict for all. In this work, we identify various subtasks in predicting effects caused by drug–drug interaction. Predicting drug–drug interaction effects for drugs that already exist is very different from predicting outcomes for newly developed drugs, commonly called a cold-start problem. We propose suitable validation schemes for the different subtasks that emerge. These validation schemes are critical to correctly assess the performance. We develop a new model that obtains AUC-ROC =0.843 for the hardest cold-start task up to AUC-ROC =0.957 for the easiest one on the benchmark dataset of Zitnik et al. Finally, we illustrate how our predictions can be used to improve post-market surveillance systems or detect drug–drug interaction effects earlier during drug development.

High-priority drug-drug interaction clinical decision support overrides in a newly implemented commercial computerized provider order-entry system: Override appropriateness and adverse drug events

OBJECTIVE

The study sought to determine frequency and appropriateness of overrides of high-priority drug-drug interaction (DDI) alerts and whether adverse drug events (ADEs) were associated with overrides in a newly implemented electronic health record.

MATERIALS AND METHODS

We conducted a retrospective study of overridden high-priority DDI alerts occurring from April 1, 2016, to March 31, 2017, from inpatient and outpatient settings at an academic health center. We studied highest-severity DDIs that were previously designated as "hard stops" and additional high-priority DDIs identified from clinical experience and literature review. All highest-severity alert overrides (n = 193) plus a stratified random sample of additional overrides (n = 371) were evaluated for override appropriateness, using predetermined criteria. Charts were reviewed to identify ADEs for overrides that resulted in medication administration. A chi-square test was used to compare ADE rate by override appropriateness.

RESULTS

Of 16 011 alerts presented to providers, 15 318 (95.7%) were overridden, including 193 (87.3%) of the highest-severity DDIs and 15 125 (95.8%) of additional DDIs. Override appropriateness was 45.4% overall, 0.5% for highest-severity DDIs and 68.7% for additional DDIs. For alerts that resulted in medication administration (n = 423, 75.0%), 29 ADEs were identified (6.9%, 5.1 per 100 overrides). The rate of ADEs was higher with inappropriate vs appropriate overrides (9.4% vs 4.3%; P = .038).

CONCLUSIONS

The override rate was nearly 90% for even the highest-severity DDI alerts, indicating that stronger suggestions should be made for these alerts, while other alerts should be evaluated for potential suppression.

Comparison of Medication Alerts from Two Commercial Applications in the USA

Introduction

Medication organizations across the USA have adopted electronic health records, and one of the most anticipated benefits of these was improved medication safety, but alert fatigue has been a major issue.

Objective

We compared the appropriateness of medication-related clinical decision support alerts triggered by two commercial applications: EPIC and Seegnal’s platform.

Methods

This was a retrospective comparison of two commercial applications. We provided Seegnal with deidentified inpatient, outpatient, and inpatient genetic electronic medical record (EMR)-extracted datasets for 657, 2731, and 413 patients, respectively. Seegnal then provided the alerts that would have triggered, which we compared with those triggered by EPIC in clinical care. A random sample of the alerts triggered were reviewed for appropriateness, and the positive predictive value (PPV) and negative predictive value (NPV) were calculated. We also reviewed all the inpatient and outpatient charts for patients within our cohort who were receiving ten or more concomitant medications with alerts we found to be appropriate to assess whether any adverse events had occurred and whether Seegnal’s platform could have prevented them.

Results

Results from EPIC and the Seegnal platform were compared based on alert load, PPV, NPV, and potential adverse events. Overall, compared with EPIC, the Seegnal platform triggered fewer alerts in the inpatient (1697 vs. 27,540), outpatient (2341 vs. 35,134), and inpatient genetic (1493 vs. 20,975) cohorts. The Seegnal platform had higher specificity in the inpatient (99 vs. 0.3%; p < 0.0001), outpatient (99 vs. 0.3%; p < 0.0001), and inpatient genetic (97.9 vs. 1.2%; p < 0.0001) groups and higher sensitivity in the inpatient (100 vs. 68.8%; p < 0.0001) and outpatient (88.6 vs.78.3%; p < 0.0001) groups but not in the inpatient genetic cohort (81 vs. 78.5%; p = 0.11). We identified 16 adverse events that occurred in the inpatient setting, 11 (69%) of which potentially could have been prevented with the Seegnal platform.

Conclusions

Overall, the Seegnal platform triggered 94% fewer alerts than EPIC in the inpatient setting and 93% fewer in the outpatient setting, with much higher sensitivity and specificity. This application could substantially reduce alert fatigue and improve medication safety at the same time.

Supplementary Information

The online version of this article (10.1007/s40264-021-01048-0) contains supplementary material, which is available to authorized users.

Clinical Decision Support and Implications for the Clinician Burnout Crisis

OBJECTIVES

This survey aimed to review aspects of clinical decision support (CDS) that contribute to burnout and identify key themes for improving the acceptability of CDS to clinicians, with the goal of decreasing said burnout.

METHODS

We performed a survey of relevant articles from 2018-2019 addressing CDS and aspects of clinician burnout from PubMed and Web of Science™. Themes were manually extracted from publications that met inclusion criteria.

RESULTS

Eighty-nine articles met inclusion criteria, including 12 review articles. Review articles were either prescriptive, describing how CDS should work, or analytic, describing how current CDS tools are deployed. The non-review articles largely demonstrated poor relevance and acceptability of current tools, and few studies showed benefits in terms of efficiency or patient outcomes from implemented CDS. Encouragingly, multiple studies highlighted steps that succeeded in improving both acceptability and relevance of CDS.

CONCLUSIONS

CDS can contribute to clinician frustration and burnout. Using the techniques of improving relevance, soliciting feedback, customization, measurement of outcomes and metrics, and iteration, the effects of CDS on burnout can be ameliorated.