Optimizing Drug-Drug Interaction Alerts Using a Multidimensional Approach

Multicomponent processes to identify and prioritise low-value care in hospital settings: a scoping review

OBJECTIVES

This scoping review mapped and synthesised original research that identified low-value care in hospital settings as part of multicomponent processes.

DESIGN

Scoping review.

DATA SOURCES

Electronic databases (EMBASE, PubMed, CINAHL, PsycINFO and Cochrane CENTRAL) and grey literature were last searched 11 July and 3 June 2022, respectively, with no language or date restrictions.

ELIGIBILITY CRITERIA

We included original research targeting the identification and prioritisation of low-value care as part of a multicomponent process in hospital settings.

DATA EXTRACTION AND SYNTHESIS

Screening was conducted in duplicate. Data were extracted by one of six authors and checked by another author. A framework synthesis was conducted using seven areas of focus for the review and an overuse framework.

RESULTS

Twenty-seven records were included (21 original studies, 4 abstracts and 2 reviews), originating from high-income countries. Benefit or value (11 records), risk or harm (10 records) were common concepts referred to in records that explicitly defined low-value care (25 records). Evidence of contextualisation including barriers and enablers of low-value care identification processes were identified (25 records). Common components of these processes included initial consensus, consultation, ranking exercise or list development (16 records), and reviews of evidence (16 records). Two records involved engagement of patients and three evaluated the outcomes of multicomponent processes. Five records referenced a theory, model or framework.

CONCLUSIONS

Gaps identified included applying systematic efforts to contextualise the identification of low-value care, involving people with lived experience of hospital care and initiatives in resource poor contexts. Insights were obtained regarding the theories, models and frameworks used to guide initiatives and ways in which the concept 'low-value care' had been used and reported. A priority for further research is evaluating the effect of initiatives that identify low-value care using contextualisation as part of multicomponent processes.

Leveraging explainable artificial intelligence to optimize clinical decision support

OBJECTIVE

To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches.

METHODS

We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful.

RESULTS

The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues.

CONCLUSION

We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.

Using AI-generated suggestions from ChatGPT to optimize clinical decision support

OBJECTIVE

To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions.

METHODS

We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy.

RESULTS

Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy.

CONCLUSION

AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.

Assessing the Value of ChatGPT for Clinical Decision Support Optimization

Objective

To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions.

Methods

We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy.

Results

Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy.

Conclusion

AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.

Clinician adherence to pharmacogenomics prescribing recommendations in clinical decision support alerts

Thoughtful integration of interruptive clinical decision support (CDS) alerts within the electronic health record is essential to guide clinicians on the application of pharmacogenomic results at point of care. St. Jude Children's Research Hospital implemented a preemptive pharmacogenomic testing program in 2011 in a multidisciplinary effort involving extensive education to clinicians about pharmacogenomic implications. We conducted a retrospective analysis of clinicians' adherence to 4783 pharmacogenomically guided CDS alerts that triggered for 12 genes and 60 drugs. Clinicians adhered to the therapeutic recommendations provided in 4392 alerts (92%). In our population of pediatric patients with catastrophic illnesses, the most frequently presented gene/drug CDS alerts were TPMT/NUDT15 and thiopurines (n = 3850), CYP2D6 and ondansetron (n = 667), CYP2D6 and oxycodone (n = 99), G6PD and G6PD high-risk medications (n  = 51), and CYP2C19 and proton pump inhibitors (omeprazole and pantoprazole; n = 50). The high adherence rate was facilitated by our team approach to prescribing and our collaborative CDS design and delivery.

Prescribers’ reported acceptance and use of drug-drug interaction alerts: An Australian survey

Drug-drug interaction (DDI) alerts are frequently included in electronic medical record (eMR) systems to provide users with relevant information and guidance at the point of care. In this study, we aimed to examine views of DDI alerts among prescribers, including junior doctors, registrars and senior doctors, across Australia. A validated survey for assessing prescribers’ reported acceptance and use of DDI alerts was distributed among researcher networks and in newsletters. Fifty useable responses were received, more than half ( n = 28) from senior doctors. Prescribers at all levels expected DDI alerts to improve performance but junior doctors reported that this was at a high cost, with respect to time and effort. Senior doctors and registrars reported rarely reading alerts and rarely changing prescribing decisions based on alerts. Respondents identified a number of problems with current alerts including limited relevance, repetition, and poor design, highlighting some clear areas for alert improvement.

Contextualized Drug-Drug Interaction Management Improves Clinical Utility Compared With Basic Drug-Drug Interaction Management in Hospitalized Patients

Drug–drug interactions (DDIs) frequently trigger adverse drug events or reduced efficacy. Most DDI alerts, however, are overridden because of irrelevance for the specific patient. Basic DDI clinical decision support (CDS) systems offer limited possibilities for decreasing the number of irrelevant DDI alerts without missing relevant ones. Computerized decision tree rules were designed to context‐dependently suppress irrelevant DDI alerts. A crossover study was performed to compare the clinical utility of contextualized and basic DDI management in hospitalized patients. First, a basic DDI‐CDS system was used in clinical practice while contextualized DDI alerts were collected in the background. Next, this process was reversed. All medication orders (MOs) from hospitalized patients with at least one DDI alert were included. The following outcome measures were used to assess clinical utility: positive predictive value (PPV), negative predictive value (NPV), number of pharmacy interventions (PIs)/1,000 MOs, and the median time spent on DDI management/1,000 MOs. During the basic DDI management phase 1,919 MOs/day were included, triggering 220 DDI alerts/1,000 MOs; showing 57 basic DDI alerts/1,000 MOs to pharmacy staff; PPV was 2.8% with 1.6 PIs/1,000 MOs costing 37.2 minutes/1,000 MOs. No DDIs were missed by the contextualized CDS system (NPV 100%). During the contextualized DDI management phase 1,853 MOs/day were included, triggering 244 basic DDI alerts/1,000 MOs, showing 9.6 contextualized DDIs/1,000 MOs to pharmacy staff; PPV was 41.4% (P < 0.01), with 4.0 PIs/1,000 MOs (P < 0.01) and 13.7 minutes/1,000 MOs. The clinical utility of contextualized DDI management exceeds that of basic DDI management.

Optimizing Hospital Electronic Prescribing Systems: A Systematic Scoping Review

OBJECTIVE

Considerable international investment in hospital electronic prescribing (ePrescribing) systems has been made, but despite this, it is proving difficult for most organizations to realize safety, quality, and efficiency gains in prescribing. The objective of this work was to develop policy-relevant insights into the optimization of hospital ePrescribing systems to maximize the benefits and minimize the risks of these expensive digital health infrastructures.

METHODS

We undertook a systematic scoping review of the literature by searching MEDLINE, Embase, and CINAHL databases. We searched for primary studies reporting on ePrescribing optimization strategies and independently screened and abstracted data until saturation was achieved. Findings were theoretically and thematically synthesized taking a medicine life-cycle perspective, incorporating consultative phases with domain experts.

RESULTS

We identified 23,609 potentially eligible studies from which 1367 satisfied our inclusion criteria. Thematic synthesis was conducted on a data set of 76 studies, of which 48 were based in the United States. Key approaches to optimization included the following: stakeholder engagement, system or process redesign, technological innovations, and education and training packages. Single-component interventions (n = 26) described technological optimization strategies focusing on a single, specific step in the prescribing process. Multicomponent interventions (n = 50) used a combination of optimization strategies, typically targeting multiple steps in the medicines management process.

DISCUSSION

We identified numerous optimization strategies for enhancing the performance of ePrescribing systems. Key considerations for ePrescribing optimization include meaningful stakeholder engagement to reconceptualize the service delivery model and implementing technological innovations with supporting training packages to simultaneously impact on different facets of the medicines management process.

The potential for leveraging machine learning to filter medication alerts

OBJECTIVE

To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user's view.

MATERIALS AND METHODS

We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate <0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1-there is no difference in precision among prediction models; and H2-the removal of any feature category does not change precision.

RESULTS

A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P < 0.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (-0.147, P = 0.001).

CONCLUSIONS

Machine learning potentially enables the intelligent filtering of medication alerts.

Designing and evaluating contextualized drug-drug interaction algorithms

Objective

Alert fatigue is a common issue with off-the-shelf clinical decision support. Most warnings for drug-drug interactions (DDIs) are overridden or ignored, likely because they lack relevance to the patient's clinical situation. Existing alerting systems for DDIs are often simplistic in nature or do not take the specific patient context into consideration, leading to overly sensitive alerts. The objective of this study is to develop, validate, and test DDI alert algorithms that take advantage of patient context available in electronic health records (EHRs) data.

Methods

Data on the rate at which DDI alerts were triggered but for which no action was taken over a 3-month period (override rates) from a single tertiary care facility were used to identify DDIs that were considered a high-priority for contextualized alerting. A panel of DDI experts developed algorithms that incorporate drug and patient characteristics that affect the relevance of such warnings. The algorithms were then implemented as computable artifacts, validated using a synthetic health records data, and tested over retrospective data from a single urban hospital.

Results

Algorithms and computable knowledge artifacts were developed and validated for a total of 8 high priority DDIs. Testing on retrospective real-world data showed the potential for the algorithms to reduce alerts that interrupt clinician workflow by more than 50%. Two algorithms (citalopram/QT interval prolonging agents, and fluconazole/opioid) showed potential to filter nearly all interruptive alerts for these combinations.

Conclusion

The 8 DDI algorithms are a step toward addressing a critical need for DDI alerts that are more specific to patient context than current commercial alerting systems. Data commonly available in EHRs can improve DDI alert specificity.

High-priority drug-drug interaction clinical decision support overrides in a newly implemented commercial computerized provider order-entry system: Override appropriateness and adverse drug events

OBJECTIVE

The study sought to determine frequency and appropriateness of overrides of high-priority drug-drug interaction (DDI) alerts and whether adverse drug events (ADEs) were associated with overrides in a newly implemented electronic health record.

MATERIALS AND METHODS

We conducted a retrospective study of overridden high-priority DDI alerts occurring from April 1, 2016, to March 31, 2017, from inpatient and outpatient settings at an academic health center. We studied highest-severity DDIs that were previously designated as "hard stops" and additional high-priority DDIs identified from clinical experience and literature review. All highest-severity alert overrides (n = 193) plus a stratified random sample of additional overrides (n = 371) were evaluated for override appropriateness, using predetermined criteria. Charts were reviewed to identify ADEs for overrides that resulted in medication administration. A chi-square test was used to compare ADE rate by override appropriateness.

RESULTS

Of 16 011 alerts presented to providers, 15 318 (95.7%) were overridden, including 193 (87.3%) of the highest-severity DDIs and 15 125 (95.8%) of additional DDIs. Override appropriateness was 45.4% overall, 0.5% for highest-severity DDIs and 68.7% for additional DDIs. For alerts that resulted in medication administration (n = 423, 75.0%), 29 ADEs were identified (6.9%, 5.1 per 100 overrides). The rate of ADEs was higher with inappropriate vs appropriate overrides (9.4% vs 4.3%; P = .038).

CONCLUSIONS

The override rate was nearly 90% for even the highest-severity DDI alerts, indicating that stronger suggestions should be made for these alerts, while other alerts should be evaluated for potential suppression.

A Minimal Information Model for Potential Drug-Drug Interactions

Despite the significant health impacts of adverse events associated with drug-drug interactions, no standard models exist for managing and sharing evidence describing potential interactions between medications. Minimal information models have been used in other communities to establish community consensus around simple models capable of communicating useful information. This paper reports on a new minimal information model for describing potential drug-drug interactions. A task force of the Semantic Web in Health Care and Life Sciences Community Group of the World-Wide Web consortium engaged informaticians and drug-drug interaction experts in in-depth examination of recent literature and specific potential interactions. A consensus set of information items was identified, along with example descriptions of selected potential drug-drug interactions (PDDIs). User profiles and use cases were developed to demonstrate the applicability of the model. Ten core information items were identified: drugs involved, clinical consequences, seriousness, operational classification statement, recommended action, mechanism of interaction, contextual information/modifying factors, evidence about a suspected drug-drug interaction, frequency of exposure, and frequency of harm to exposed persons. Eight best practice recommendations suggest how PDDI knowledge artifact creators can best use the 10 information items when synthesizing drug interaction evidence into artifacts intended to aid clinicians. This model has been included in a proposed implementation guide developed by the HL7 Clinical Decision Support Workgroup and in PDDIs published in the CDS Connect repository. The complete description of the model can be found at https://w3id.org/hclscg/pddi.

Practice-enhancing publications about the medication-use process in 2019

PURPOSE

This article identifies, prioritizes, and summarizes published literature on the medication-use process (MUP) from calendar year 2019 that can impact health-system pharmacy daily practice. The MUP is the foundational system that provides the framework for safe medication utilization within the healthcare environment. The MUP is defined in this article as having the following components: prescribing/transcribing, dispensing, administration, and monitoring. Articles that evaluated one of the steps were gauged for their usefulness in promoting daily practice change.

SUMMARY

A PubMed search was conducted in January 2020 for calendar year 2019 using targeted Medical Subject Headings keywords; in addition, searches of the table of contents of selected pharmacy journals were conducted. A total of 4,317 articles were identified. A thorough review identified 66 potentially practice-enhancing articles: 17 for prescribing/transcribing, 17 for dispensing, 7 for administration, and 25 for monitoring. Ranking of the articles for importance by peers led to the selection of key articles from each category. The highest-ranked articles are briefly summarized, with a mention of why each article is important within health-system pharmacy. The other articles are listed for further review and evaluation.

CONCLUSION

It is important to routinely review the published literature and to incorporate significant findings into daily practice; this article assists in identifying and summarizing the most impactful recently published literature in this area. Health-system pharmacists have an active role in improving the MUP in their institution, and awareness of the significant published studies can assist in changing practice at the institutional level.

Biomedical Data Science and Informatics Challenges to Implementing Pharmacogenomics with Electronic Health Records

Pharmacogenomic information must be incorporated into electronic health records (EHRs) with clinical decision support in order to fully realize its potential to improve drug therapy. Supported by various clinical knowledge resources, pharmacogenomic workflows have been implemented in several healthcare systems. Little standardization exists across these efforts, however, which limits scalability both within and across clinical sites. Limitations in information standards, knowledge management, and the capabilities of modern EHRs remain challenges for the widespread use of pharmacogenomics in the clinic, but ongoing efforts are addressing these challenges. Although much work remains to use pharmacogenomic information more effectively within clinical systems, the experiences of pioneering sites and lessons learned from those programs may be instructive for other clinical areas beyond genomics. We present a vision of what can be achieved as informatics and data science converge to enable further adoption of pharmacogenomics in the clinic.