1
|
Liu S, McCoy AB, Peterson JF, Lasko TA, Sittig DF, Nelson SD, Andrews J, Patterson L, Cobb CM, Mulherin D, Morton CT, Wright A. Leveraging explainable artificial intelligence to optimize clinical decision support. J Am Med Inform Assoc 2024; 31:968-974. [PMID: 38383050 PMCID: PMC10990514 DOI: 10.1093/jamia/ocae019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 01/02/2024] [Accepted: 01/22/2024] [Indexed: 02/23/2024] Open
Abstract
OBJECTIVE To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches. METHODS We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful. RESULTS The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues. CONCLUSION We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.
Collapse
Affiliation(s)
- Siru Liu
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Computer Science, Vanderbilt University, Nashville, TN 37212, United States
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Josh F Peterson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Thomas A Lasko
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Computer Science, Vanderbilt University, Nashville, TN 37212, United States
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, TX 77030, United States
| | - Scott D Nelson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Jennifer Andrews
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Pathology, Microbiology and Immunology, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Lorraine Patterson
- HeathIT, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Cheryl M Cobb
- Department of Psychiatry and Behavioral Sciences, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - David Mulherin
- HeathIT, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Colleen T Morton
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| |
Collapse
|
2
|
Zhang T, Gephart SM, Subbian V, Boyce RD, Villa-Zapata L, Tan MS, Horn J, Gomez-Lumbreras A, Romero AV, Malone DC. Barriers to Adoption of Tailored Drug-Drug Interaction Clinical Decision Support. Appl Clin Inform 2023; 14:779-788. [PMID: 37793617 PMCID: PMC10550365 DOI: 10.1055/s-0043-1772686] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Accepted: 07/20/2023] [Indexed: 10/06/2023] Open
Abstract
OBJECTIVE Despite the benefits of the tailored drug-drug interaction (DDI) alerts and the broad dissemination strategy, the uptake of our tailored DDI alert algorithms that are enhanced with patient-specific and context-specific factors has been limited. The goal of the study was to examine barriers and health care system dynamics related to implementing tailored DDI alerts and identify the factors that would drive optimization and improvement of DDI alerts. METHODS We employed a qualitative research approach, conducting interviews with a participant interview guide framed based on Proctor's taxonomy of implementation outcomes and informed by the Theoretical Domains Framework. Participants included pharmacists with informatics roles within hospitals, chief medical informatics officers, and associate medical informatics directors/officers. Our data analysis was informed by the technique used in grounded theory analysis, and the reporting of open coding results was based on a modified version of the Safety-Related Electronic Health Record Research Reporting Framework. RESULTS Our analysis generated 15 barriers, and we mapped the interconnections of these barriers, which clustered around three entities (i.e., users, organizations, and technical stakeholders). Our findings revealed that misaligned interests regarding DDI alert performance and misaligned expectations regarding DDI alert optimizations among these entities within health care organizations could result in system inertia in implementing tailored DDI alerts. CONCLUSION Health care organizations primarily determine the implementation and optimization of DDI alerts, and it is essential to identify and demonstrate value metrics that health care organizations prioritize to enable tailored DDI alert implementation. This could be achieved via a multifaceted approach, such as partnering with health care organizations that have the capacity to adopt tailored DDI alerts and identifying specialists who know users' needs, liaise with organizations and vendors, and facilitate technical stakeholders' work. In the future, researchers can adopt the systematic approach to study tailored DDI implementation problems from other system perspectives (e.g., the vendors' system).
Collapse
Affiliation(s)
- Tianyi Zhang
- Department of Systems and Industrial Engineering, College of Engineering, University of Arizona, Tucson, Arizona
| | - Sheila M. Gephart
- Advanced Nursing Practice and Science Division, College of Nursing, University of Arizona, Tucson, Arizona
| | - Vignesh Subbian
- Department of Systems and Industrial Engineering, College of Engineering, University of Arizona, Tucson, Arizona
| | - Richard D. Boyce
- Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania
| | - Lorenzo Villa-Zapata
- Clinical and Administrative Pharmacy, College of Pharmacy, University of Georgia, Athens, Georgia
| | - Malinda S. Tan
- Department of Pharmacotherapy, College of Pharmacy, University of Utah, Salt Lake City, Utah
| | - John Horn
- Department of Pharmacy, School of Pharmacy, University of Washington, Seattle, Washington
| | - Ainhoa Gomez-Lumbreras
- Department of Pharmacotherapy, College of Pharmacy, University of Utah, Salt Lake City, Utah
| | | | - Daniel C. Malone
- Department of Pharmacotherapy, College of Pharmacy, University of Utah, Salt Lake City, Utah
| |
Collapse
|
3
|
Cánovas-Segura B, Morales A, Juarez JM, Campos M. Meaningful time-related aspects of alerts in Clinical Decision Support Systems. A unified framework. J Biomed Inform 2023:104397. [PMID: 37245656 DOI: 10.1016/j.jbi.2023.104397] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2022] [Revised: 03/11/2023] [Accepted: 05/15/2023] [Indexed: 05/30/2023]
Abstract
Alerts are a common functionality of clinical decision support systems (CDSSs). Although they have proven to be useful in clinical practice, the alert burden can lead to alert fatigue and significantly reduce their usability and acceptance. Based on a literature review, we propose a unified framework consisting of a set of meaningful timestamps that allows the use of state-of-the-art measures for alert burden, such as alert dwell time, alert think time, and response time. In addition, it can be used to investigate other measures that could be relevant as regards dealing with this problem. Furthermore, we provide a case study concerning three different types of alerts to which the framework was successfully applied. We consider that our framework can easily be adapted to other CDSSs and that it could be useful for dealing with alert burden measurement thus contributing to its appropriate management.
Collapse
Affiliation(s)
| | - Antonio Morales
- AIKE Research Group (INTICO), University of Murcia, Murcia, Spain.
| | - Jose M Juarez
- AIKE Research Group (INTICO), University of Murcia, Murcia, Spain.
| | - Manuel Campos
- AIKE Research Group (INTICO), University of Murcia, Murcia, Spain; Murcian Bio-Health Institute (IMIB-Arrixaca), Murcia, Spain.
| |
Collapse
|
4
|
Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD, McCoy AB, Sittig DF, Wright A. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc 2023:7136722. [PMID: 37087108 DOI: 10.1093/jamia/ocad072] [Citation(s) in RCA: 79] [Impact Index Per Article: 79.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 03/28/2023] [Accepted: 04/11/2023] [Indexed: 04/24/2023] Open
Abstract
OBJECTIVE To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. METHODS We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. RESULTS Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. CONCLUSION AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
Collapse
Affiliation(s)
- Siru Liu
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Aileen P Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
- Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Barron L Patterson
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Jonathan P Wanderer
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
- Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Robert W Turer
- Department of Emergency Medicine, University of Texas Southwestern Medical Center, Dallas, Texas, USA
- Clinical Informatics Center, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Scott D Nelson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
5
|
Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD, McCoy AB, Sittig DF, Wright A. Assessing the Value of ChatGPT for Clinical Decision Support Optimization. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.02.21.23286254. [PMID: 36865144 PMCID: PMC9980251 DOI: 10.1101/2023.02.21.23286254] [Citation(s) in RCA: 17] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/25/2023]
Abstract
Objective To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
Collapse
|
6
|
Zhao M, Liu CF, Feng YF, Chen H. Potential drug-drug interactions in drug therapy for older adults with chronic coronary syndrome at hospital discharge: A real-world study. Front Pharmacol 2022; 13:946415. [PMID: 36091832 PMCID: PMC9449411 DOI: 10.3389/fphar.2022.946415] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2022] [Accepted: 07/22/2022] [Indexed: 11/18/2022] Open
Abstract
Introduction: Polypharmacy are commonly observed among older adults with cardiovascular disease. However, multiple medications lead to increased risk of drug-drug interactions (DDIs). Therefore, identification and prevention actions related to harmful DDIs are expected in older adults. The study aimed to describe the prevalence of potential DDIs (pDDIs) in discharge prescriptions among older adults with chronic coronary syndrome (CCS). Methods: A single-center cross-sectional study was performed in a tertiary public hospital in Beijing, China. CCS patients aged 65 years and above who were admitted to cardiology wards over a 3-month period and alive at discharge were included. Electronic medical records and discharge prescriptions were reviewed. pDDIs were evaluated through the Lexi-Interact online. Results: pDDIs were identified in 72.9% of the 402 individuals (n = 293). A total of 864 pDDIs were obtained. 72.1% of patients were found with C DDIs (n = 290) and 20.3% were categorized in D and X DDIs (n = 82). The only X DDI was between cyclosporine and atorvastatin. Under category D, glycemia alterations within antidiabetics and increased chances of bleeding with antithrombotic were the most common. Concomitant use of clopidogrel and calcium channel blockers was a frequent situation within category C, followed by synergic blood pressure lowering agents and increased rosuvastatin concentration induced by clopidogrel. Conclusion: DDIs exposure was common in older CCS. DDIs screening tools should be introduced to alert potential adverse effects. Prescribers need to rigorously review or modulate therapies to prevent DDI-related adverse outcomes. Clinical pharmacists should be more involved in complex drug regimen management.
Collapse
Affiliation(s)
- Mei Zhao
- Department of Pharmacy, Peking University People’s Hospital, Beijing, China
| | - Chuan-Fen Liu
- Department of Cardiology, Peking University People’s Hospital, Beijing, China
- Beijing Key Laboratory of Early Prediction and Intervention of Acute Myocardial Infarction, Peking University People’s Hospital, Beijing, China
- Center for Cardiovascular Translational Research, Peking University People’s Hospital, Beijing, China
| | - Yu-Fei Feng
- Department of Pharmacy, Peking University People’s Hospital, Beijing, China
| | - Hong Chen
- Department of Cardiology, Peking University People’s Hospital, Beijing, China
- Beijing Key Laboratory of Early Prediction and Intervention of Acute Myocardial Infarction, Peking University People’s Hospital, Beijing, China
- Center for Cardiovascular Translational Research, Peking University People’s Hospital, Beijing, China
- *Correspondence: Hong Chen,
| |
Collapse
|
7
|
Baysari MT, Dort BAV, Zheng WY, Li L, Hilmer S, Westbrook J, Day R. Prescribers’ reported acceptance and use of drug-drug interaction alerts: An Australian survey. Health Informatics J 2022; 28:14604582221100678. [DOI: 10.1177/14604582221100678] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Drug-drug interaction (DDI) alerts are frequently included in electronic medical record (eMR) systems to provide users with relevant information and guidance at the point of care. In this study, we aimed to examine views of DDI alerts among prescribers, including junior doctors, registrars and senior doctors, across Australia. A validated survey for assessing prescribers’ reported acceptance and use of DDI alerts was distributed among researcher networks and in newsletters. Fifty useable responses were received, more than half ( n = 28) from senior doctors. Prescribers at all levels expected DDI alerts to improve performance but junior doctors reported that this was at a high cost, with respect to time and effort. Senior doctors and registrars reported rarely reading alerts and rarely changing prescribing decisions based on alerts. Respondents identified a number of problems with current alerts including limited relevance, repetition, and poor design, highlighting some clear areas for alert improvement.
Collapse
Affiliation(s)
- Melissa T Baysari
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, Charles Perkins Centre, The University of Sydney, NSW, Australia
| | - Bethany A Van Dort
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, Charles Perkins Centre, The University of Sydney, NSW, Australia
| | - Wu Yi Zheng
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, Charles Perkins Centre, The University of Sydney, NSW, Australia
- Black Dog Institute, NSW Australia
| | - Ling Li
- Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia
| | - Sarah Hilmer
- Kolling Institute of Medical Research, Faculty of Medicine and Health, The University of Sydney, NSW, Australia
- Departments of Clinical Pharmacology and Aged Care, Royal North Shore Hospital, Sydney, NSW, Australia
| | - Johanna Westbrook
- Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia
| | - Richard Day
- Department of Clinical Pharmacology and Toxicology, St Vincent’s Hospital, Sydney, NSW, Australia
- St Vincent’s Clinical School, Faculty of Medicine, UNSW, Sydney, NSW, Australia
| |
Collapse
|
8
|
Van De Sijpe G, Quintens C, Walgraeve K, Van Laer E, Penny J, De Vlieger G, Schrijvers R, De Munter P, Foulon V, Casteels M, Van der Linden L, Spriet I. Overall performance of a drug-drug interaction clinical decision support system: quantitative evaluation and end-user survey. BMC Med Inform Decis Mak 2022; 22:48. [PMID: 35193547 PMCID: PMC8864797 DOI: 10.1186/s12911-022-01783-z] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2021] [Accepted: 02/03/2022] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Clinical decision support systems are implemented in many hospitals to prevent medication errors and associated harm. They are however associated with a high burden of false positive alerts and alert fatigue. The aim of this study was to evaluate a drug-drug interaction (DDI) clinical decision support system in terms of its performance, uptake and user satisfaction and to identify barriers and opportunities for improvement. METHODS A quantitative evaluation and end-user survey were performed in a large teaching hospital. First, very severe DDI alerts generated between 2019 and 2021 were evaluated retrospectively. Data collection comprised alert burden, override rates, the number of alert overrides reviewed by pharmacists and the resulting pharmacist recommendations as well as their acceptance rate. Second, an e-survey was carried out among prescribers to assess satisfaction, usefulness and relevance of DDI alerts as well as reasons for overriding. RESULTS A total of 38,409 very severe DDI alerts were generated, of which 88.2% were overridden by the prescriber. In 3.2% of reviewed overrides, a recommendation by the pharmacist was provided, of which 79.2% was accepted. False positive alerts were caused by a too broad screening interval and lack of incorporation of patient-specific characteristics, such as QTc values. Co-prescribing of a non-vitamin K oral anticoagulant and a low molecular weight heparin accounted for 49.8% of alerts, of which 92.2% were overridden. In 88 (1.1%) of these overridden alerts, concurrent therapy was still present. Despite the high override rate, the e-survey revealed that the DDI clinical decision support system was found useful by prescribers. CONCLUSIONS Identified barriers were the lack of DDI-specific screening intervals and inclusion of patient-specific characteristics, both leading to a high number of false positive alerts and risk for alert fatigue. Despite these barriers, the added value of the DDI clinical decision support system was recognized by prescribers. Hence, integration of DDI-specific screening intervals and patient-specific characteristics is warranted to improve the performance of the DDI software.
Collapse
Affiliation(s)
- Greet Van De Sijpe
- Pharmacy Department, University Hospitals Leuven, Leuven, Belgium. .,Clinical Pharmacology and Pharmacotherapy, Department of Pharmaceutical and Pharmacological Sciences, KU Leuven, Leuven, Belgium.
| | - Charlotte Quintens
- Pharmacy Department, University Hospitals Leuven, Leuven, Belgium.,Clinical Pharmacology and Pharmacotherapy, Department of Pharmaceutical and Pharmacological Sciences, KU Leuven, Leuven, Belgium
| | | | - Eva Van Laer
- Pharmacy Department, University Hospitals Leuven, Leuven, Belgium
| | - Jens Penny
- Department of Information Technology, University Hospitals Leuven, Leuven, Belgium
| | - Greet De Vlieger
- Department of Intensive Care Medicine, University Hospitals Leuven, Leuven, Belgium
| | - Rik Schrijvers
- Department of General Internal Medicine, University Hospitals Leuven, Leuven, Belgium.,Department of Microbiology, Immunology and Transplantation, KU Leuven, Leuven, Belgium
| | - Paul De Munter
- Department of General Internal Medicine, University Hospitals Leuven, Leuven, Belgium.,Department of Microbiology, Immunology and Transplantation, KU Leuven, Leuven, Belgium
| | - Veerle Foulon
- Clinical Pharmacology and Pharmacotherapy, Department of Pharmaceutical and Pharmacological Sciences, KU Leuven, Leuven, Belgium
| | - Minne Casteels
- Clinical Pharmacology and Pharmacotherapy, Department of Pharmaceutical and Pharmacological Sciences, KU Leuven, Leuven, Belgium
| | - Lorenz Van der Linden
- Pharmacy Department, University Hospitals Leuven, Leuven, Belgium.,Clinical Pharmacology and Pharmacotherapy, Department of Pharmaceutical and Pharmacological Sciences, KU Leuven, Leuven, Belgium
| | - Isabel Spriet
- Pharmacy Department, University Hospitals Leuven, Leuven, Belgium.,Clinical Pharmacology and Pharmacotherapy, Department of Pharmaceutical and Pharmacological Sciences, KU Leuven, Leuven, Belgium
| |
Collapse
|
9
|
Bestha D, Tomatsu S, Hutcheson B, Blankenship K, Yu Z, Wally MK, Wyatt S, Seymour RB, Hsu JR, Rachal J. Impact of an opioid prescribing alert system on patients with posttraumatic stress disorder. Am J Addict 2022; 31:123-131. [PMID: 35112432 DOI: 10.1111/ajad.13261] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2021] [Revised: 12/17/2021] [Accepted: 01/15/2022] [Indexed: 11/30/2022] Open
Abstract
BACKGROUND AND OBJECTIVES Posttraumatic stress disorder (PTSD) is associated with higher rates of chronic pain and increased risk of developing Opioid use disorder. This paper evaluates the impact of PRIMUM, an electronic health record-embedded (EHR) clinical decision support intervention on opioid prescribing patterns for patients with diagnosis of PTSD. METHODS Inpatient, emergency department (ED), urgent care, and outpatient encounters with ICD-10 codes F43.1 (PTSD), F43.10 (PTSD, unspecified), F43.11 (PTSD, acute), and F43.12 (PTSD, chronic) at Atrium Health between 1/1/2016 and 12/29/2018 were included in the study. RESULTS A total of 3121 patients with a diagnosis of PTSD were seen in 37,443 encounters during the study period. Ten percent (n = 3761) of the encounters resulted in prescriptions for opioids and PRIMUM alerts were triggered in 1488 of these encounters. These alerts resulted in "decision influenced" for 17% of patients (n = 255) or no prescriptions for opioids or benzodiazepines for 5.8% (n = 86). The majority of the prescriptions were below 50 Morphine milligram equivalents (MME)/day, but there were 570 (15.5%) prescriptions for doses of 50-90 MME and 721 (19.6%) prescriptions for >90 MME/day. DISCUSSION AND CONCLUSION The PRIMUM alert system helps improve patient safety. PRIMUM affected clinician decisions 17% of the time, and the effect was greater in patients with opioid overdose history and those presenting for early refills. SCIENTIFIC SIGNIFICANCE The effectiveness of clinical support interventions for opioid prescribing for patients with PTSD has not been documented previously. Our findings provide novel evidence that the EHR can be used to improve patient safety among patients with PTSD in the context of substance use.
Collapse
Affiliation(s)
- Durga Bestha
- Department of Psychiatry, Atrium Health, Charlotte, North Carolina, USA
| | - Shizuka Tomatsu
- Department of Psychiatry, Atrium Health, Charlotte, North Carolina, USA
| | | | - Kelly Blankenship
- Department of Psychiatry, Dayton Children's Hospital, Dayton, Ohio, USA
| | - Ziqing Yu
- Department of Orthopaedic Surgery, Atrium Health Musculoskeletal Institute, Charlotte, North Carolina, USA
| | - Meghan K Wally
- Department of Orthopaedic Surgery, Atrium Health Musculoskeletal Institute, Charlotte, North Carolina, USA
| | - Stephen Wyatt
- Department of Psychiatry, Atrium Health, Charlotte, North Carolina, USA
| | - Rachel B Seymour
- Department of Orthopaedic Surgery, Atrium Health Musculoskeletal Institute, Charlotte, North Carolina, USA
| | - Joseph R Hsu
- Department of Orthopaedic Surgery, Atrium Health Musculoskeletal Institute, Charlotte, North Carolina, USA
| | - James Rachal
- Department of Psychiatry, Atrium Health, Charlotte, North Carolina, USA
| | | |
Collapse
|
10
|
Cold-Start Problems in Data-Driven Prediction of Drug-Drug Interaction Effects. Pharmaceuticals (Basel) 2021; 14:ph14050429. [PMID: 34063324 PMCID: PMC8147651 DOI: 10.3390/ph14050429] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2021] [Revised: 04/27/2021] [Accepted: 04/28/2021] [Indexed: 02/02/2023] Open
Abstract
Combining drugs, a phenomenon often referred to as polypharmacy, can induce additional adverse effects. The identification of adverse combinations is a key task in pharmacovigilance. In this context, in silico approaches based on machine learning are promising as they can learn from a limited number of combinations to predict for all. In this work, we identify various subtasks in predicting effects caused by drug–drug interaction. Predicting drug–drug interaction effects for drugs that already exist is very different from predicting outcomes for newly developed drugs, commonly called a cold-start problem. We propose suitable validation schemes for the different subtasks that emerge. These validation schemes are critical to correctly assess the performance. We develop a new model that obtains AUC-ROC =0.843 for the hardest cold-start task up to AUC-ROC =0.957 for the easiest one on the benchmark dataset of Zitnik et al. Finally, we illustrate how our predictions can be used to improve post-market surveillance systems or detect drug–drug interaction effects earlier during drug development.
Collapse
|
11
|
Edrees H, Amato MG, Wong A, Seger DL, Bates DW. High-priority drug-drug interaction clinical decision support overrides in a newly implemented commercial computerized provider order-entry system: Override appropriateness and adverse drug events. J Am Med Inform Assoc 2021; 27:893-900. [PMID: 32337561 DOI: 10.1093/jamia/ocaa034] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2020] [Revised: 02/21/2020] [Accepted: 03/12/2020] [Indexed: 01/01/2023] Open
Abstract
OBJECTIVE The study sought to determine frequency and appropriateness of overrides of high-priority drug-drug interaction (DDI) alerts and whether adverse drug events (ADEs) were associated with overrides in a newly implemented electronic health record. MATERIALS AND METHODS We conducted a retrospective study of overridden high-priority DDI alerts occurring from April 1, 2016, to March 31, 2017, from inpatient and outpatient settings at an academic health center. We studied highest-severity DDIs that were previously designated as "hard stops" and additional high-priority DDIs identified from clinical experience and literature review. All highest-severity alert overrides (n = 193) plus a stratified random sample of additional overrides (n = 371) were evaluated for override appropriateness, using predetermined criteria. Charts were reviewed to identify ADEs for overrides that resulted in medication administration. A chi-square test was used to compare ADE rate by override appropriateness. RESULTS Of 16 011 alerts presented to providers, 15 318 (95.7%) were overridden, including 193 (87.3%) of the highest-severity DDIs and 15 125 (95.8%) of additional DDIs. Override appropriateness was 45.4% overall, 0.5% for highest-severity DDIs and 68.7% for additional DDIs. For alerts that resulted in medication administration (n = 423, 75.0%), 29 ADEs were identified (6.9%, 5.1 per 100 overrides). The rate of ADEs was higher with inappropriate vs appropriate overrides (9.4% vs 4.3%; P = .038). CONCLUSIONS The override rate was nearly 90% for even the highest-severity DDI alerts, indicating that stronger suggestions should be made for these alerts, while other alerts should be evaluated for potential suppression.
Collapse
Affiliation(s)
- Heba Edrees
- Department of Pharmacy Practice, Massachusetts College of Pharmacy and Health Sciences, Boston, Massachusetts, USA
| | - Mary G Amato
- Department of Pharmacy Practice, Massachusetts College of Pharmacy and Health Sciences, Boston, Massachusetts, USA.,Center for Patient Safety Research and Practice, Division of General Internal Medicine and Primary Care; Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Adrian Wong
- Department of Pharmacy Practice, Massachusetts College of Pharmacy and Health Sciences, Boston, Massachusetts, USA.,Center for Patient Safety Research and Practice, Division of General Internal Medicine and Primary Care; Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Diane L Seger
- Center for Patient Safety Research and Practice, Division of General Internal Medicine and Primary Care; Brigham and Women's Hospital, Boston, Massachusetts, USA.,Clinical and Quality Analysis, Information Systems, Partners HealthCare, Somerville, Massachusetts, USA
| | - David W Bates
- Center for Patient Safety Research and Practice, Division of General Internal Medicine and Primary Care; Brigham and Women's Hospital, Boston, Massachusetts, USA.,Clinical and Quality Analysis, Information Systems, Partners HealthCare, Somerville, Massachusetts, USA.,Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
12
|
Shah SN, Seger DL, Fiskio JM, Horn JR, Bates DW. Comparison of Medication Alerts from Two Commercial Applications in the USA. Drug Saf 2021; 44:661-668. [PMID: 33616888 PMCID: PMC8184526 DOI: 10.1007/s40264-021-01048-0] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/01/2021] [Indexed: 11/17/2022]
Abstract
Introduction Medication organizations across the USA have adopted electronic health records, and one of the most anticipated benefits of these was improved medication safety, but alert fatigue has been a major issue. Objective We compared the appropriateness of medication-related clinical decision support alerts triggered by two commercial applications: EPIC and Seegnal’s platform. Methods This was a retrospective comparison of two commercial applications. We provided Seegnal with deidentified inpatient, outpatient, and inpatient genetic electronic medical record (EMR)-extracted datasets for 657, 2731, and 413 patients, respectively. Seegnal then provided the alerts that would have triggered, which we compared with those triggered by EPIC in clinical care. A random sample of the alerts triggered were reviewed for appropriateness, and the positive predictive value (PPV) and negative predictive value (NPV) were calculated. We also reviewed all the inpatient and outpatient charts for patients within our cohort who were receiving ten or more concomitant medications with alerts we found to be appropriate to assess whether any adverse events had occurred and whether Seegnal’s platform could have prevented them. Results Results from EPIC and the Seegnal platform were compared based on alert load, PPV, NPV, and potential adverse events. Overall, compared with EPIC, the Seegnal platform triggered fewer alerts in the inpatient (1697 vs. 27,540), outpatient (2341 vs. 35,134), and inpatient genetic (1493 vs. 20,975) cohorts. The Seegnal platform had higher specificity in the inpatient (99 vs. 0.3%; p < 0.0001), outpatient (99 vs. 0.3%; p < 0.0001), and inpatient genetic (97.9 vs. 1.2%; p < 0.0001) groups and higher sensitivity in the inpatient (100 vs. 68.8%; p < 0.0001) and outpatient (88.6 vs.78.3%; p < 0.0001) groups but not in the inpatient genetic cohort (81 vs. 78.5%; p = 0.11). We identified 16 adverse events that occurred in the inpatient setting, 11 (69%) of which potentially could have been prevented with the Seegnal platform. Conclusions Overall, the Seegnal platform triggered 94% fewer alerts than EPIC in the inpatient setting and 93% fewer in the outpatient setting, with much higher sensitivity and specificity. This application could substantially reduce alert fatigue and improve medication safety at the same time. Supplementary Information The online version of this article (10.1007/s40264-021-01048-0) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Sonam N Shah
- Department of General Internal Medicine, Brigham and Women's Hospital, 41 Avenue Louis Pasteur, Office 103, Boston, MA, 02115, USA. .,Department of Pharmacy Practice, MCPHS University, Boston, MA, USA.
| | - Diane L Seger
- Clinical and Quality Analysis, Mass General Brigham, Somerville, MA, USA
| | - Julie M Fiskio
- Department of General Internal Medicine, Brigham and Women's Hospital, 41 Avenue Louis Pasteur, Office 103, Boston, MA, 02115, USA
| | - John R Horn
- University of Washington Medicine Pharmacy Services, Seattle, WA, USA
| | - David W Bates
- Department of General Internal Medicine, Brigham and Women's Hospital, 41 Avenue Louis Pasteur, Office 103, Boston, MA, 02115, USA.,Clinical and Quality Analysis, Mass General Brigham, Somerville, MA, USA.,Harvard Medical School, Boston, MA, USA
| |
Collapse
|
13
|
Abstract
OBJECTIVES This survey aimed to review aspects of clinical decision support (CDS) that contribute to burnout and identify key themes for improving the acceptability of CDS to clinicians, with the goal of decreasing said burnout. METHODS We performed a survey of relevant articles from 2018-2019 addressing CDS and aspects of clinician burnout from PubMed and Web of Science™. Themes were manually extracted from publications that met inclusion criteria. RESULTS Eighty-nine articles met inclusion criteria, including 12 review articles. Review articles were either prescriptive, describing how CDS should work, or analytic, describing how current CDS tools are deployed. The non-review articles largely demonstrated poor relevance and acceptability of current tools, and few studies showed benefits in terms of efficiency or patient outcomes from implemented CDS. Encouragingly, multiple studies highlighted steps that succeeded in improving both acceptability and relevance of CDS. CONCLUSIONS CDS can contribute to clinician frustration and burnout. Using the techniques of improving relevance, soliciting feedback, customization, measurement of outcomes and metrics, and iteration, the effects of CDS on burnout can be ameliorated.
Collapse
Affiliation(s)
- Ivana Jankovic
- Division of Endocrinology, Stanford University School of Medicine, Stanford, CA, USA
| | - Jonathan H. Chen
- Center for Biomedical Informatics Research and Division of Hospital Medicine, Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|