1
|
Mahowald GK, Lewandrowski KB, Dighe AS. Clinical decision support to improve CBC and differential ordering. Am J Clin Pathol 2024; 162:151-159. [PMID: 38507618 DOI: 10.1093/ajcp/aqae024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2023] [Accepted: 01/31/2024] [Indexed: 03/22/2024] Open
Abstract
OBJECTIVES Complete blood count and differential (CBC diff) is a common laboratory test that may be overused or misordered, particularly in an inpatient setting. We assessed the ability of a clinical decision support (CDS) alert to decrease unnecessary orders for CBC diff and analyzed its impact in the laboratory. METHODS We designed 3 CDS alerts to provide guidance to providers ordering CBC diff on inpatients at frequencies of daily, greater than once daily, or as needed. RESULTS The 3 alerts were highly effective in reducing orders for CBC diff at the frequencies targeted by the alert. Overall, test volume for CBC diff decreased by 32% (mean of 5257 tests per month) after implementation of the alerts, with a corresponding decrease of 22% in manual differentials performed (mean of 898 per month). Turnaround time for manual differentials decreased by a mean of 41.5 minutes, with a mean decrease of up to 90 minutes during peak morning hours. CONCLUSIONS The 3 CDS alerts successfully decreased inpatient orders for CBC diff and improved the quality of patient care by decreasing turnaround time for manual differentials.
Collapse
Affiliation(s)
- Grace K Mahowald
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, US
| | - Kent B Lewandrowski
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, US
| | - Anand S Dighe
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, US
| |
Collapse
|
2
|
Henry KE, Giannini HM. Early Warning Systems for Critical Illness Outside the Intensive Care Unit. Crit Care Clin 2024; 40:561-581. [PMID: 38796228 DOI: 10.1016/j.ccc.2024.03.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/28/2024]
Abstract
Early warning systems (EWSs) are designed and deployed to create a rapid assessment and response for patients with clinical deterioration outside the intensive care unit (ICU). These models incorporate patient-level data such as vital signs and laboratory values to detect or prevent adverse clinical events, such as vital signs and laboratories to allow detection and prevention of adverse clinical events such as cardiac arrest, intensive care transfer, or sepsis. The applicability, development, clinical utility, and general perception of EWS in clinical practice vary widely. Here, we review the field as it has grown from early vital sign-based scoring systems to contemporary multidimensional algorithms and predictive technologies for clinical decompensation outside the ICU.
Collapse
Affiliation(s)
- Katharine E Henry
- Department of Computer Science, Johns Hopkins University, Malone Hall, 3400 N Charles Street, Baltimore, MD 21218, USA
| | - Heather M Giannini
- Division of Pulmonary, Allergy and Critical Care, Hospital of the University of Pennsylvania, 5 West Gates Building, 5048, 3400 Spruce Street, Philadelphia, PA 19104, USA.
| |
Collapse
|
3
|
Sundermann M, Clendon O, McNeill R, Doogue M, Chin PKL. Optimising interruptive clinical decision support alerts for antithrombotic duplicate prescribing in hospital. Int J Med Inform 2024; 186:105418. [PMID: 38518676 DOI: 10.1016/j.ijmedinf.2024.105418] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Revised: 03/05/2024] [Accepted: 03/17/2024] [Indexed: 03/24/2024]
Abstract
INTRODUCTION Duplicate prescribing clinical decision support alerts can prevent important prescribing errors but are frequently the cause of much alert fatigue. Stat dose prescriptions are a known reason for overriding these alerts. This study aimed to evaluate the effect of excluding stat dose prescriptions from duplicate prescribing alerts for antithrombotic medicines on alert burden, prescriber adherence, and prescribing. MATERIALS AND METHODS A before (January 1st, 2017 to August 31st, 2022) and after (October 5th, 2022 to September 30th, 2023) study was undertaken of antithrombotic duplicate prescribing alerts and prescribing following a change in alert settings. Alert and prescribing data for antithrombotic medicines were joined, processed, and analysed to compare alert rates, adherence, and prescribing. Alert burden was assessed as alerts per 100 prescriptions. Adherence was measured at the point of the alert as whether the prescriber accepted the alert and following the alert as whether a relevant prescription was ceased within an hour. Co-prescribing of antithrombotic stat dose prescriptions was assessed pre- and post-alert reconfiguration. RESULTS Reconfiguration of the alerts reduced the alert rate by 29 % (p < 0.001). The proportion of alerts associated with cessation of antithrombotic duplication significantly increased (32.8 % to 44.5 %, p < 0.001). Adherence at the point of the alert increased 1.2 % (4.8 % to 6.0 %, p = 0.012) and 11.5 % (29.4 % to 40.9 %, p < 0.001) within one hour of the alert. When ceased after the alert over 80 % of duplicate prescriptions were ceased within 2 min of overriding. Antithrombotic stat dose co-prescribing was unchanged for 4 out of 5 antithrombotic duplication alert rules. CONCLUSION By reconfiguring our antithrombotic duplicate prescribing alerts, we reduced alert burden and increased alert adherence. Many prescribers ceased duplicate prescribing within 2 min of alert override highlighting the importance of incorporating post-alert measures in accurately determining prescriber alert adherence.
Collapse
Affiliation(s)
- Milan Sundermann
- Department of Medicine, University of Otago, Christchurch, New Zealand
| | - Olivia Clendon
- Department of Clinical Pharmacology, Te Whatu Ora Health New Zealand - Waitaha Canterbury, New Zealand
| | - Richard McNeill
- Department of Medicine, University of Otago, Christchurch, New Zealand
| | - Matthew Doogue
- Department of Medicine, University of Otago, Christchurch, New Zealand; Department of Clinical Pharmacology, Te Whatu Ora Health New Zealand - Waitaha Canterbury, New Zealand
| | - Paul K L Chin
- Department of Medicine, University of Otago, Christchurch, New Zealand; Department of Clinical Pharmacology, Te Whatu Ora Health New Zealand - Waitaha Canterbury, New Zealand.
| |
Collapse
|
4
|
Barton HJ, Maru A, Leaf MA, Hekman DJ, Wiegmann DA, Shah MN, Patterson BW. Academic Detailing as a Health Information Technology Implementation Method: Supporting the Design and Implementation of an Emergency Department-Based Clinical Decision Support Tool to Prevent Future Falls. JMIR Hum Factors 2024; 11:e52592. [PMID: 38635318 PMCID: PMC11066751 DOI: 10.2196/52592] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 02/08/2024] [Accepted: 03/02/2024] [Indexed: 04/19/2024] Open
Abstract
BACKGROUND Clinical decision support (CDS) tools that incorporate machine learning-derived content have the potential to transform clinical care by augmenting clinicians' expertise. To realize this potential, such tools must be designed to fit the dynamic work systems of the clinicians who use them. We propose the use of academic detailing-personal visits to clinicians by an expert in a specific health IT tool-as a method for both ensuring the correct understanding of that tool and its evidence base and identifying factors influencing the tool's implementation. OBJECTIVE This study aimed to assess academic detailing as a method for simultaneously ensuring the correct understanding of an emergency department-based CDS tool to prevent future falls and identifying factors impacting clinicians' use of the tool through an analysis of the resultant qualitative data. METHODS Previously, our team designed a CDS tool to identify patients aged 65 years and older who are at the highest risk of future falls and prompt an interruptive alert to clinicians, suggesting the patient be referred to a mobility and falls clinic for an evidence-based preventative intervention. We conducted 10-minute academic detailing interviews (n=16) with resident emergency medicine physicians and advanced practice providers who had encountered our CDS tool in practice. We conducted an inductive, team-based content analysis to identify factors that influenced clinicians' use of the CDS tool. RESULTS The following categories of factors that impacted clinicians' use of the CDS were identified: (1) aspects of the CDS tool's design (2) clinicians' understanding (or misunderstanding) of the CDS or referral process, (3) the busy nature of the emergency department environment, (4) clinicians' perceptions of the patient and their associated fall risk, and (5) the opacity of the referral process. Additionally, clinician education was done to address any misconceptions about the CDS tool or referral process, for example, demonstrating how simple it is to place a referral via the CDS and clarifying which clinic the referral goes to. CONCLUSIONS Our study demonstrates the use of academic detailing for supporting the implementation of health information technologies, allowing us to identify factors that impacted clinicians' use of the CDS while concurrently educating clinicians to ensure the correct understanding of the CDS tool and intervention. Thus, academic detailing can inform both real-time adjustments of a tool's implementation, for example, refinement of the language used to introduce the tool, and larger scale redesign of the CDS tool to better fit the dynamic work environment of clinicians.
Collapse
Affiliation(s)
- Hanna J Barton
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin-Madison, Madison, WI, United States
| | - Apoorva Maru
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin-Madison, Madison, WI, United States
| | - Margaret A Leaf
- Department of Information Services, UW Health, Madison, WI, United States
| | - Daniel J Hekman
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin-Madison, Madison, WI, United States
| | - Douglas A Wiegmann
- Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI, United States
| | - Manish N Shah
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin-Madison, Madison, WI, United States
| | - Brian W Patterson
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin-Madison, Madison, WI, United States
- Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI, United States
| |
Collapse
|
5
|
Devis L, Catry E, Honore PM, Mansour A, Lippi G, Mullier F, Closset M. Interventions to improve appropriateness of laboratory testing in the intensive care unit: a narrative review. Ann Intensive Care 2024; 14:9. [PMID: 38224401 PMCID: PMC10789714 DOI: 10.1186/s13613-024-01244-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Accepted: 01/03/2024] [Indexed: 01/16/2024] Open
Abstract
Healthcare expenses are increasing, as is the utilization of laboratory resources. Despite this, between 20% and 40% of requested tests are deemed inappropriate. Improper use of laboratory resources leads to unwanted consequences such as hospital-acquired anemia, infections, increased costs, staff workload and patient stress and discomfort. The most unfavorable consequences result from unnecessary follow-up tests and treatments (overuse) and missed or delayed diagnoses (underuse). In this context, several interventions have been carried out to improve the appropriateness of laboratory testing. To date, there have been few published assessments of interventions specific to the intensive care unit. We reviewed the literature for interventions implemented in the ICU to improve the appropriateness of laboratory testing. We searched literature from 2008 to 2023 in PubMed, Embase, Scopus, and Google Scholar databases between April and June 2023. Five intervention categories were identified: education and guidance (E&G), audit and feedback, gatekeeping, computerized physician order entry (including reshaping of ordering panels), and multifaceted interventions (MFI). We included a sixth category exploring the potential role of artificial intelligence and machine learning (AI/ML)-based assisting tools in such interventions. E&G-based interventions and MFI are the most frequently used approaches. MFI is the most effective type of intervention, and shows the strongest persistence of effect over time. AI/ML-based tools may offer valuable assistance to the improvement of appropriate laboratory testing in the near future. Patient safety outcomes are not impaired by interventions to reduce inappropriate testing. The literature focuses mainly on reducing overuse of laboratory tests, with only one intervention mentioning underuse. We highlight an overall poor quality of methodological design and reporting and argue for standardization of intervention methods. Collaboration between clinicians and laboratory staff is key to improve appropriate laboratory utilization. This article offers practical guidance for optimizing the effectiveness of an intervention protocol designed to limit inappropriate use of laboratory resources.
Collapse
Affiliation(s)
- Luigi Devis
- Department of Laboratory Medicine, Biochemistry, CHU UCL Namur, Université catholique de Louvain, Yvoir, Belgium
| | - Emilie Catry
- Department of Laboratory Medicine, Biochemistry, CHU UCL Namur, Université catholique de Louvain, Yvoir, Belgium
- Institute for Experimental and Clinical Research (IREC), Pôle Mont Godinne (MONT), UCLouvain, Yvoir, Belgium
| | - Patrick M Honore
- Department of Intensive Care, CHU UCL Namur, Université catholique de Louvain, Yvoir, Belgium
| | - Alexandre Mansour
- Department of Anesthesia and Critical Care, Pontchaillou University Hospital of Rennes, Rennes, France
- IRSET-INSERM-1085, Univ Rennes, Rennes, France
| | - Giuseppe Lippi
- Section of Clinical Biochemistry and School of Medicine, University Hospital of Verona, Verona, Italy
| | - François Mullier
- Department of Laboratory Medicine, Hematology, CHU UCL Namur, Université catholique de Louvain, Yvoir, Belgium
- Namur Thrombosis and Hemostasis Center (NTHC), Namur Research Institute for Life Sciences (NARILIS), Namur, Belgium
- Institute for Experimental and Clinical Research (IREC), Pôle Mont Godinne (MONT), UCLouvain, Yvoir, Belgium
| | - Mélanie Closset
- Department of Laboratory Medicine, Biochemistry, CHU UCL Namur, Université catholique de Louvain, Yvoir, Belgium.
- Institute for Experimental and Clinical Research (IREC), Pôle Mont Godinne (MONT), UCLouvain, Yvoir, Belgium.
| |
Collapse
|
6
|
Colicchio TK, Osborne JD, Do Rosario CV, Anand A, Timkovich NA, Wyatt MC, Cimino JJ. Semantically oriented EHR navigation with a patient specific knowledge base and a clinical context ontology. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2024; 2023:309-318. [PMID: 38222434 PMCID: PMC10785934] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 01/16/2024]
Abstract
Widespread adoption of electronic health records (EHR) in the U.S. has been followed by unintended consequences, overexposing clinicians to widely reported EHR limitations. As an attempt to fixing the EHR, we propose the use of a clinical context ontology (CCO), applied to turn implicit contextual statements into formally represented data in the form of concept-relationship-concept tuples. These tuples form what we call a patient specific knowledge base (PSKB), a collection of formally defined tuples containing facts about the patient's care context. We report the process to create a CCO, which guides annotation of structured and narrative patient data to produce a PSKB. We also present an application of our PSKB using real patient data displayed on a semantically oriented patient summary to improve EHR navigation. Our approach can potentially save precious time spent by clinicians using today's EHRs, by showing a chronological view of the patient's record along with contextual statements needed for care decisions with minimum effort. We propose several other applications of a PSKB to improve multiple EHR functions to guide future research.
Collapse
Affiliation(s)
| | - John D Osborne
- Informatics Institute, University of Alabama at Birmingham
| | | | - Ankit Anand
- Informatics Institute, University of Alabama at Birmingham
| | | | | | - James J Cimino
- Informatics Institute, University of Alabama at Birmingham
| |
Collapse
|
7
|
Colicchio TK, Cimino JJ. Beyond the override: Using evidence of previous drug tolerance to suppress drug allergy alerts; a retrospective study of opioid alerts. J Biomed Inform 2023; 147:104508. [PMID: 37748541 DOI: 10.1016/j.jbi.2023.104508] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Revised: 08/29/2023] [Accepted: 09/22/2023] [Indexed: 09/27/2023]
Abstract
OBJECTIVE Despite the extensive literature exploring alert fatigue, most studies have focused on describing the phenomenon, but not on fixing it. The authors aimed to identify data useful to avert clinically irrelevant alerts to inform future research on clinical decision support (CDS) design. METHODS We conducted a retrospective observational study of opioid drug allergy alert (DAA) overrides for the calendar year of 2019 at a large academic medical center, to identify data elements useful to find irrelevant alerts to be averted. RESULTS Overall, 227,815 DAAs were fired in 2019, with an override rate of 91 % (n = 208196). Opioids represented nearly two-thirds of these overrides (n = 129063; 62 %) and were the drug class with the highest override rate (96 %). On average, 29 opioid DAAs were overridden per patient. While most opioid alerts (97.1 %) are fired for a possible match (the drug class of the allergen matches the drug class of the prescribed drug), they are overridden significantly less frequently for definite match (exact match between allergen and prescribed drug) (88 % vs. 95.9 %, p < 0.001). When comparing the triggering drug with previously administered drugs, override rates were equally high for both definite match (95.9 %), no match (95.5 %), and possible match (95.1 %). Likewise, when comparing to home medications, overrides were excessively high for possible match (96.3 %), no match (96 %), and definite match (94.4 %). CONCLUSION We estimate that 74.5% of opioid DAAs (46.4% of all DAAs) at our institution could be relatively safely averted, since they either have a definite match for previous inpatient administrations suggesting drug tolerance or are fired as possible match with low risk of cross-sensitivity. Future research should focus on identifying other relevant data elements ideally with automated methods and use of emerging standards to empower CDS systems to suppress false-positive alerts while avoiding safety hazards.
Collapse
Affiliation(s)
- Tiago K Colicchio
- Informatics Institute, University of Alabama at Birmingham, AL, USA.
| | - James J Cimino
- Informatics Institute, University of Alabama at Birmingham, AL, USA
| |
Collapse
|
8
|
Ng HJH, Kansal A, Abdul Naseer JF, Hing WC, Goh CJM, Poh H, D’souza JLA, Lim EL, Tan G. Optimizing Best Practice Advisory alerts in electronic medical records with a multi-pronged strategy at a tertiary care hospital in Singapore. JAMIA Open 2023; 6:ooad056. [PMID: 37538232 PMCID: PMC10393867 DOI: 10.1093/jamiaopen/ooad056] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Revised: 05/23/2023] [Accepted: 07/26/2023] [Indexed: 08/05/2023] Open
Abstract
Objective Clinical decision support (CDS) alerts can aid in improving patient care. One CDS functionality is the Best Practice Advisory (BPA) alert notification system, wherein BPA alerts are automated alerts embedded in the hospital's electronic medical records (EMR). However, excessive alerts can change clinician behavior; redundant and repetitive alerts can contribute to alert fatigue. Alerts can be optimized through a multipronged strategy. Our study aims to describe these strategies adopted and evaluate the resultant BPA alert optimization outcomes. Materials and Methods This retrospective single-center study was done at Jurong Health Campus. Aggregated, anonymized data on patient demographics and alert statistics were collected from January 1, 2018 to December 31, 2021. "Preintervention" period was January 1-December 31, 2018, and "postintervention" period was January 1-December 31, 2021. The intervention period was the intervening period. Categorical variables were reported as frequencies and proportions and compared using the chi-square test. Continuous data were reported as median (interquartile range, IQR) and compared using the Wilcoxon rank-sum test. Statistical significance was defined at P < .05. Results There was a significant reduction of 59.6% in the total number of interruptive BPA alerts, despite an increase in the number of unique BPAs from 54 to 360 from pre- to postintervention. There was a 74% reduction in the number of alerts from the 7 BPAs that were optimized from the pre- to postintervention period. There was a significant increase in percentage of overall interruptive BPA alerts with action taken (8 [IQR 7.7-8.4] to 54.7 [IQR 52.5-58.9], P-value < .05) and optimized BPAs with action taken (32.6 [IQR 32.3-32.9] to 72.6 [IQR 64.3-73.4], P-value < .05). We estimate that the reduction in alerts saved 3600 h of providers' time per year. Conclusions A significant reduction in interruptive alert volume, and a significant increase in action taken rates despite manifold increase in the number of unique BPAs could be achieved through concentrated efforts focusing on governance, data review, and visualization using a system-embedded tool, combined with the CDS Five Rights framework, to optimize alerts. Improved alert compliance was likely multifactorial-due to decreased repeated alert firing for the same patient; better awareness due to stakeholders' involvement; and less fatigue since unnecessary alerts were removed. Future studies should prospectively focus on patients' clinical chart reviews to assess downstream effects of various actions taken, identify any possibility of harm, and collect end-user feedback regarding the utility of alerts.
Collapse
Affiliation(s)
- Hannah Jia Hui Ng
- Corresponding Author: Hannah Jia Hui Ng, MBBS, MRCS, Department of Medical Informatics, Ng Teng Fong General Hospital, 1 Jurong East Street 21, Singapore 609606, Singapore;
| | - Amit Kansal
- Department of Medical Informatics, Ng Teng Fong General Hospital, Singapore, Singapore
| | | | - Wee Chuan Hing
- Department of Medical Informatics, Ng Teng Fong General Hospital, Singapore, Singapore
| | - Carmen Jia Man Goh
- Department of Medical Informatics, Ng Teng Fong General Hospital, Singapore, Singapore
| | - Hermione Poh
- Department of Medical Informatics, Ng Teng Fong General Hospital, Singapore, Singapore
| | | | - Er Luen Lim
- Department of Medical Informatics, Ng Teng Fong General Hospital, Singapore, Singapore
| | - Gamaliel Tan
- Department of Medical Informatics, Ng Teng Fong General Hospital, Singapore, Singapore
| |
Collapse
|
9
|
Vittery ER, Bayliss E, Heed A, Fagan C, Thomas M, Tse Y. Reducing prescribing errors: making electronic prescribing work for cystic fibrosis inpatients. Arch Dis Child Educ Pract Ed 2023; 108:112-114. [PMID: 35264442 DOI: 10.1136/archdischild-2021-322446] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/12/2021] [Accepted: 02/11/2022] [Indexed: 11/04/2022]
Abstract
Children admitted to our hospital with cystic fibrosis had frequent medication errors due to polypharmacy and addition of specialist and high-risk medications despite an electronic prescribing and medicines administration system in place. We describe a multidisciplinary quality improvement project that combined a computerised order entry system (CPOE) with human factor process changes. Over 12 months, our run chart showed a 43% reduction in prescription errors. For medications prescribable via the CPOE, errors reaching the patient reduced from 50% to 29%. Electronic prescribing can be seen by clinicians as a fixed unalterable system contributing to rather than ameliorating errors. Improving safety requires whole team engagement and working closely with programmers to adapt function and influence human factors.
Collapse
Affiliation(s)
| | - Emily Bayliss
- Department of Paediatric Pharmacy, Newcastle Upon Tyne Hospitals NHS Trust, Newcastle Upon Tyne, UK
| | - Andrew Heed
- Department of Clinical Informatics Pharmacy, Newcastle Upon Tyne Hospitals NHS Trust, Newcastle Upon Tyne, UK
| | - Claire Fagan
- Department of Respiratory Paediatrics, Great North Children's Hospital, Newcastle Upon Tyne, UK
| | - Matthew Thomas
- Department of Respiratory Paediatrics, Great North Children's Hospital, Newcastle Upon Tyne, UK
- Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK
| | - Yincent Tse
- Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK
- Great North Children's Hospital, Newcastle Upon Tyne, UK
| |
Collapse
|
10
|
Ho VT, Aikens RC, Tso G, Heidenreich PA, Sharp C, Asch SM, Chen JH, Shah NK. Interruptive Electronic Alerts for Choosing Wisely Recommendations: A Cluster Randomized Controlled Trial. J Am Med Inform Assoc 2022; 29:1941-1948. [PMID: 36018731 PMCID: PMC10161518 DOI: 10.1093/jamia/ocac139] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2022] [Revised: 07/13/2022] [Accepted: 08/17/2022] [Indexed: 11/13/2022] Open
Abstract
OBJECTIVE To assess the efficacy of interruptive electronic alerts in improving adherence to the American Board of Internal Medicine's Choosing Wisely recommendations to reduce unnecessary laboratory testing. MATERIALS AND METHODS We administered 5 cluster randomized controlled trials simultaneously, using electronic medical record alerts regarding prostate-specific antigen (PSA) testing, acute sinusitis treatment, vitamin D testing, carotid artery ultrasound screening, and human papillomavirus testing. For each alert, we assigned 5 outpatient clinics to an interruptive alert and 5 were observed as a control. Primary and secondary outcomes were the number of postalert orders per 100 patients at each clinic and number of triggered alerts divided by orders, respectively. Post hoc analysis evaluated whether physicians experiencing interruptive alerts reduced their alert-triggering behaviors. RESULTS Median postalert orders per 100 patients did not differ significantly between treatment and control groups; absolute median differences ranging from 0.04 to 0.40 for PSA testing. Median alerts per 100 orders did not differ significantly between treatment and control groups; absolute median differences ranged from 0.004 to 0.03. In post hoc analysis, providers receiving alerts regarding PSA testing in men were significantly less likely to trigger additional PSA alerts than those in the control sites (Incidence Rate Ratio 0.12, 95% CI [0.03-0.52]). DISCUSSION Interruptive point-of-care alerts did not yield detectable changes in the overall rate of undesired orders or the order-to-alert ratio between active and silent sites. Complementary behavioral or educational interventions are likely needed to improve efforts to curb medical overuse. CONCLUSION Implementation of interruptive alerts at the time of ordering was not associated with improved adherence to 5 Choosing Wisely guidelines. TRIAL REGISTRATION NCT02709772.
Collapse
Affiliation(s)
- Vy T Ho
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| | - Rachael C Aikens
- Department of Biomedical Informatics, Stanford University School of Medicine, Stanford, California, USA
| | - Geoffrey Tso
- Division of Primary Care and Population Health, Stanford University School of Medicine, Palo Alto, California, USA
| | - Paul A Heidenreich
- Division of Cardiovascular Medicine, Department of Medicine, Stanford University School of Medicine, Stanford, California, USA
- Center for Innovation to Implementation, VA Palo Alto Healthcare System, Palo Alto, California, USA
| | - Christopher Sharp
- Department of Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Steven M Asch
- Division of Primary Care and Population Health, Stanford University School of Medicine, Palo Alto, California, USA
- Center for Innovation to Implementation, VA Palo Alto Healthcare System, Palo Alto, California, USA
| | - Jonathan H Chen
- Stanford Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California, USA
| | - Neil K Shah
- Department of Pathology, Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
11
|
Park H, Chae MK, Jeong W, Yu J, Jung W, Chang H, Cha WC. Appropriateness of Alerts and Physicians’ Responses with a Medication-related Clinical Decision Support System (Preprint). JMIR Med Inform 2022; 10:e40511. [PMID: 36194461 PMCID: PMC9579928 DOI: 10.2196/40511] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2022] [Revised: 09/13/2022] [Accepted: 09/18/2022] [Indexed: 12/02/2022] Open
Abstract
Background Alert fatigue is unavoidable when many irrelevant alerts are generated in response to a small number of useful alerts. It is necessary to increase the effectiveness of the clinical decision support system (CDSS) by understanding physicians’ responses. Objective This study aimed to understand the CDSS and physicians’ behavior by evaluating the clinical appropriateness of alerts and the corresponding physicians’ responses in a medication-related passive alert system. Methods Data on medication-related orders, alerts, and patients’ electronic medical records were analyzed. The analyzed data were generated between August 2019 and June 2020 while the patient was in the emergency department. We evaluated the appropriateness of alerts and physicians’ responses for a subset of 382 alert cases and classified them. Results Of the 382 alert cases, only 7.3% (n=28) of the alerts were clinically appropriate. Regarding the appropriateness of the physicians’ responses about the alerts, 92.4% (n=353) were deemed appropriate. In the classification of alerts, only 3.4% (n=13) of alerts were successfully triggered, and 2.1% (n=8) were inappropriate in both alert clinical relevance and physician’s response. In this study, the override rate was 92.9% (n=355). Conclusions We evaluated the appropriateness of alerts and physicians’ responses through a detailed medical record review of the medication-related passive alert system. An excessive number of unnecessary alerts are generated, because the algorithm operates as a rule base without reflecting the individual condition of the patient. It is important to maximize the value of the CDSS by comprehending physicians’ responses.
Collapse
Affiliation(s)
- Hyunjung Park
- Department of Digital Health, Samsung Advanced Institute of Health Sciences & Technology, Sungkyunkwan University, Seoul, Republic of Korea
| | - Minjung Kathy Chae
- Department of Digital Health, Samsung Advanced Institute of Health Sciences & Technology, Sungkyunkwan University, Seoul, Republic of Korea
| | - Woohyeon Jeong
- Department of Digital Health, Samsung Advanced Institute of Health Sciences & Technology, Sungkyunkwan University, Seoul, Republic of Korea
| | - Jaeyong Yu
- Department of Digital Health, Samsung Advanced Institute of Health Sciences & Technology, Sungkyunkwan University, Seoul, Republic of Korea
| | - Weon Jung
- Department of Digital Health, Samsung Advanced Institute of Health Sciences & Technology, Sungkyunkwan University, Seoul, Republic of Korea
| | - Hansol Chang
- Department of Digital Health, Samsung Advanced Institute of Health Sciences & Technology, Sungkyunkwan University, Seoul, Republic of Korea
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Won Chul Cha
- Department of Digital Health, Samsung Advanced Institute of Health Sciences & Technology, Sungkyunkwan University, Seoul, Republic of Korea
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
- Digital Innovation Center, Samsung Medical Center, Seoul, Republic of Korea
| |
Collapse
|
12
|
Chaparro JD, Beus JM, Dziorny AC, Hagedorn PA, Hernandez S, Kandaswamy S, Kirkendall ES, McCoy AB, Muthu N, Orenstein EW. Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts. Appl Clin Inform 2022; 13:560-568. [PMID: 35613913 PMCID: PMC9132737 DOI: 10.1055/s-0042-1748856] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022] Open
Abstract
Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.
Collapse
Affiliation(s)
- Juan D Chaparro
- Division of Clinical Informatics, Nationwide Children's Hospital, Columbus, Ohio, United States.,Departments of Pediatrics and Biomedical Informatics, The Ohio State University College of Medicine, Columbus, Ohio, United States
| | - Jonathan M Beus
- Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States.,Children's Healthcare of Atlanta, Atlanta, Georgia, United States
| | - Adam C Dziorny
- Department of Pediatrics, University of Rochester School of Medicine, Rochester, New York, United States
| | - Philip A Hagedorn
- Department of Pediatrics, University of Cincinnati, College of Medicine, Cincinnati, Ohio, United States.,Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
| | - Sean Hernandez
- Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States.,Department of General Internal Medicine, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
| | - Swaminathan Kandaswamy
- Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
| | - Eric S Kirkendall
- Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States.,Department of Pediatrics, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States.,Center for Biomedical Informatics, Wake Forest School of Medicine, Winston-Salem NC, United States
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Naveen Muthu
- Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States.,Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
| | - Evan W Orenstein
- Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States.,Children's Healthcare of Atlanta, Atlanta, Georgia, United States
| |
Collapse
|
13
|
Baysari MT, Dort BAV, Zheng WY, Li L, Hilmer S, Westbrook J, Day R. Prescribers’ reported acceptance and use of drug-drug interaction alerts: An Australian survey. Health Informatics J 2022; 28:14604582221100678. [DOI: 10.1177/14604582221100678] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Drug-drug interaction (DDI) alerts are frequently included in electronic medical record (eMR) systems to provide users with relevant information and guidance at the point of care. In this study, we aimed to examine views of DDI alerts among prescribers, including junior doctors, registrars and senior doctors, across Australia. A validated survey for assessing prescribers’ reported acceptance and use of DDI alerts was distributed among researcher networks and in newsletters. Fifty useable responses were received, more than half ( n = 28) from senior doctors. Prescribers at all levels expected DDI alerts to improve performance but junior doctors reported that this was at a high cost, with respect to time and effort. Senior doctors and registrars reported rarely reading alerts and rarely changing prescribing decisions based on alerts. Respondents identified a number of problems with current alerts including limited relevance, repetition, and poor design, highlighting some clear areas for alert improvement.
Collapse
Affiliation(s)
- Melissa T Baysari
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, Charles Perkins Centre, The University of Sydney, NSW, Australia
| | - Bethany A Van Dort
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, Charles Perkins Centre, The University of Sydney, NSW, Australia
| | - Wu Yi Zheng
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, Charles Perkins Centre, The University of Sydney, NSW, Australia
- Black Dog Institute, NSW Australia
| | - Ling Li
- Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia
| | - Sarah Hilmer
- Kolling Institute of Medical Research, Faculty of Medicine and Health, The University of Sydney, NSW, Australia
- Departments of Clinical Pharmacology and Aged Care, Royal North Shore Hospital, Sydney, NSW, Australia
| | - Johanna Westbrook
- Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia
| | - Richard Day
- Department of Clinical Pharmacology and Toxicology, St Vincent’s Hospital, Sydney, NSW, Australia
- St Vincent’s Clinical School, Faculty of Medicine, UNSW, Sydney, NSW, Australia
| |
Collapse
|
14
|
Alsaidan JA, Portlock J, Ata SI, Aljadhey HS, Franklin BD. Retrospective descriptive assessment of clinical decision support medication-related alerts in two Saudi Arabian hospitals. BMC Med Inform Decis Mak 2022; 22:101. [PMID: 35428282 PMCID: PMC9012024 DOI: 10.1186/s12911-022-01838-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2021] [Accepted: 03/31/2022] [Indexed: 11/10/2022] Open
Abstract
Objectives To determine the frequency of clinical decision support system (CDSS) medication-related alerts generated, accepted, or overridden, to assess appropriateness of alert display and overrides, and to characterise the documentation of clinician justification for these overrides in an academic medical centre in Saudi Arabia. Materials and methods System-generated CDSS reports for the period June 2015 to December 2017 were retrospectively reviewed and analysed. Alerts were classified into different types, and rates of alert overrides calculated as percentages of all generated alerts. A subset of 307 overridden alerts was assessed for appropriateness of display and override by two clinical pharmacists. Physician documentation of reasons for overriding alerts were categorised. Results A total of 4,446,730 medication-related alerts were generated from both inpatient and outpatient settings, and 4,231,743 (95.2%) were overridden. The most common alert type was ‘duplicate drug’, accounting for 3,549,736 (79.8%) of alerts. Of 307 alerts assessed for appropriateness, 246 (80%) were judged to be appropriately displayed and 244 (79%) were overridden appropriately. New drug allergy and drug allergy alerts had the highest percentage of being judged as inappropriately overridden. For 1,594,313 alerts (37.7%), ‘no overridden reason selected’ was chosen from the drop-down menu. Conclusions The alert generation and override rate were higher than reported previously in the literature. The small sample size of 307 alerts assessed for appropriateness of alert display and override is a potential limitation. Revision of the CDSS rules for alerts (focusing on specificity and relevance for the local context) is now recommended. Future research should prospectively assess providers’ perspectives, and determine patient harm associated with overridden alerts. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-022-01838-1.
Collapse
|
15
|
Smith K, Durant KM, Zimmerman C. Impact of an electronic health record alert on inappropriate prescribing of high-risk medications to patients with concurrent "do not give" orders. Am J Health Syst Pharm 2022; 79:1198-1204. [PMID: 35333916 DOI: 10.1093/ajhp/zxac092] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
DISCLAIMER In an effort to expedite the publication of articles related to the COVID-19 pandemic, AJHP is posting these manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time. PURPOSE To evaluate the effectiveness of clinical decision support (CDS) alerts tied to high-risk medications at a Michigan health system by determining the true prescriber action rate in response to select "do not give" (DNG) alerts. METHODS A retrospective review of prescriber actions in response to CDS alerts was conducted to evaluate the effectiveness of alerts designed to prevent prescribing of high-risk medications to patients with concurrent DNG orders. The primary endpoint was the overall action rate, determined by totaling orders cancelled within the alert display and orders modified shortly after an alert. The overall action rate was hypothesized to significantly exceed the action rate estimated on the basis of alert overrides alone. Following the initial review, changes were made to the alert format and preset documentation choices ("acknowledgement comments"), and it was hypothesized that these changes would increase the overall action rate. A repeat analysis was conducted to evaluate the impact of these changes. RESULTS Across a total of 506 CDS alerts over 14 months, 78% resulted in prescribers modifying orders to comply with alert recommendations. Prescribers cancelled orders in response to only 26% of alerts, often overriding alerts prior to modifying orders. Documentation of rationale or approval for overrides was inconsistent, and while requiring acknowledgement comments facilitated documentation of prescriber rationale, it did not consistently improve overall action rates. CONCLUSION These findings demonstrate that override rates alone are not good markers for the true effectiveness of CDS alerts and support the need for frequent evaluation of alerts at the institutional level. CDS alerts remain a valuable tool to prevent inappropriate prescribing of high-risk medications and for promoting patient safety.
Collapse
Affiliation(s)
- Kirsten Smith
- Department of Pharmacy Services and Clinical Pharmacy, Michigan Medicine, Ann Arbor, MI, and University of Michigan College of Pharmacy, Ann Arbor, MI, USA
| | - Karin M Durant
- Department of Pharmacy Services and Clinical Pharmacy, Michigan Medicine, Ann Arbor, MI, and University of Michigan College of Pharmacy, Ann Arbor, MI, USA
| | - Chris Zimmerman
- University of Michigan College of Pharmacy, Ann Arbor, MI, and Department of Health Information Technology & Services, Michigan Medicine, Ann Arbor, MI, USA
| |
Collapse
|
16
|
Post AR, Burningham Z, Halwani AS. Electronic Health Record Data in Cancer Learning Health Systems: Challenges and Opportunities. JCO Clin Cancer Inform 2022; 6:e2100158. [PMID: 35353547 PMCID: PMC9005105 DOI: 10.1200/cci.21.00158] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2021] [Revised: 01/04/2022] [Accepted: 02/18/2022] [Indexed: 12/21/2022] Open
Affiliation(s)
- Andrew R. Post
- Research Informatics Shared Resource, Huntsman Cancer Institute, University of Utah, Salt Lake City, UT
- Department of Biomedical Informatics, University of Utah School of Medicine, Salt Lake City, UT
| | - Zachary Burningham
- Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, UT
| | - Ahmad S. Halwani
- Division of Hematology and Hematologic Malignancies, Department of Internal Medicine, University of Utah, Salt Lake City, UT
| |
Collapse
|
17
|
Wang L, Goh KH, Yeow A, Poh H, Li K, Yeow JJL, Tan G, Soh C. Habit and Automaticity in Medical Alert Override: Cohort Study. J Med Internet Res 2022; 24:e23355. [PMID: 35171102 PMCID: PMC8892274 DOI: 10.2196/23355] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2020] [Revised: 04/10/2021] [Accepted: 12/10/2021] [Indexed: 11/13/2022] Open
Abstract
Background Prior literature suggests that alert dismissal could be linked to physicians’ habits and automaticity. The evidence for this perspective has been mainly observational data. This study uses log data from an electronic medical records system to empirically validate this perspective. Objective We seek to quantify the association between habit and alert dismissal in physicians. Methods We conducted a retrospective analysis using the log data comprising 66,049 alerts generated from hospitalized patients in a hospital from March 2017 to December 2018. We analyzed 1152 physicians exposed to a specific clinical support alert triggered in a hospital’s electronic medical record system to estimate the extent to which the physicians’ habit strength, which had been developed from habitual learning, impacted their propensity toward alert dismissal. We further examined the association between a physician’s habit strength and their subsequent incidences of alert dismissal. Additionally, we recorded the time taken by the physician to respond to the alert and collected data on other clinical and environmental factors related to the alerts as covariates for the analysis. Results We found that a physician’s prior dismissal of alerts leads to their increased habit strength to dismiss alerts. Furthermore, a physician’s habit strength to dismiss alerts was found to be positively associated with incidences of subsequent alert dismissals after their initial alert dismissal. Alert dismissal due to habitual learning was also found to be pervasive across all physician ranks, from junior interns to senior attending specialists. Further, the dismissal of alerts had been observed to typically occur after a very short processing time. Our study found that 72.5% of alerts were dismissed in under 3 seconds after the alert appeared, and 13.2% of all alerts were dismissed in under 1 second after the alert appeared. We found empirical support that habitual dismissal is one of the key factors associated with alert dismissal. We also found that habitual dismissal of alerts is self-reinforcing, which suggests significant challenges in disrupting or changing alert dismissal habits once they are formed. Conclusions Habitual tendencies are associated with the dismissal of alerts. This relationship is pervasive across all levels of physician rank and experience, and the effect is self-reinforcing.
Collapse
Affiliation(s)
- Le Wang
- City University of Hong Kong, Hong Kong, China (Hong Kong)
| | - Kim Huat Goh
- Nanyang Technological University, Singapore, Singapore
| | - Adrian Yeow
- Singapore University of Social Sciences, Singapore, Singapore
| | - Hermione Poh
- Medical Informatics, National University Health System, Singapore, Singapore
| | - Ke Li
- Medical Informatics, National University Health System, Singapore, Singapore
| | | | - Gamaliel Tan
- Medical Informatics, National University Health System, Singapore, Singapore.,Ng Teng Fong General Hospital, Singapore, Singapore
| | - Christina Soh
- Nanyang Technological University, Singapore, Singapore
| |
Collapse
|
18
|
Kaas-Hansen BS, Leal Rodríguez C, Placido D, Thorsen-Meyer HC, Nielsen AP, Dérian N, Brunak S, Andersen SE. Using Machine Learning to Identify Patients at High Risk of Inappropriate Drug Dosing in Periods with Renal Dysfunction. Clin Epidemiol 2022; 14:213-223. [PMID: 35228820 PMCID: PMC8881932 DOI: 10.2147/clep.s344435] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2021] [Accepted: 01/24/2022] [Indexed: 12/23/2022] Open
Abstract
Purpose Dosing of renally cleared drugs in patients with kidney failure often deviates from clinical guidelines, so we sought to elicit predictors of receiving inappropriate doses of renal risk drugs. Patients and methods We combined data from the Danish National Patient Register and in-hospital data on drug administrations and estimated glomerular filtration rates for admissions between 1 October 2009 and 1 June 2016, from a pool of about 2.6 million persons. We trained artificial neural network and linear logistic ridge regression models to predict the risk of five outcomes (>0, ≥1, ≥2, ≥3 and ≥5 inappropriate doses daily) with index set 24 hours after admission. We used time-series validation for evaluating discrimination, calibration, clinical utility and explanations. Results Of 52,451 admissions included, 42,250 (81%) were used for model development. The median age was 77 years; 50% of admissions were of women. ≥5 drugs were used between admission start and index in 23,124 admissions (44%); the most common drug classes were analgesics, systemic antibacterials, diuretics, antithrombotics, and antacids. The neural network models had better discriminative power (all AUROCs between 0.77 and 0.81) and were better calibrated than their linear counterparts. The main prediction drivers were use of anti-inflammatory, antidiabetic and anti-Parkinson's drugs as well as having a diagnosis of chronic kidney failure. Sex and age affected predictions but slightly. Conclusion Our models can flag patients at high risk of receiving at least one inappropriate dose daily in a controlled in-silico setting. A prospective clinical study may confirm that this holds in real-life settings and translates into benefits in hard endpoints.
Collapse
Affiliation(s)
- Benjamin Skov Kaas-Hansen
- Clinical Pharmacology Unit, Zealand University Hospital, Roskilde, Denmark
- NNF Center for Protein Research, University of Copenhagen, Copenhagen, Denmark
- Section for Biostatistics, Department of Public Health, University of Copenhagen, Copenhagen, Denmark
- Correspondence: Benjamin Skov Kaas-Hansen, Clinical Pharmacology Unit, Zealand University Hospital, Munkesoevej 18, Roskilde, 4000, Denmark, Tel +45 60 19 68 02, Email
| | | | - Davide Placido
- NNF Center for Protein Research, University of Copenhagen, Copenhagen, Denmark
| | - Hans-Christian Thorsen-Meyer
- NNF Center for Protein Research, University of Copenhagen, Copenhagen, Denmark
- Department of Intensive Care Medicine, Copenhagen University Hospital (Rigshospitalet), Copenhagen, Denmark
| | - Anna Pors Nielsen
- NNF Center for Protein Research, University of Copenhagen, Copenhagen, Denmark
| | - Nicolas Dérian
- Data and Development Support, Region Zealand, Sorø, Denmark
| | - Søren Brunak
- NNF Center for Protein Research, University of Copenhagen, Copenhagen, Denmark
| | | |
Collapse
|
19
|
Orenstein EW, Kandaswamy S, Muthu N, Chaparro JD, Hagedorn PA, Dziorny AC, Moses A, Hernandez S, Khan A, Huth HB, Beus JM, Kirkendall ES. Alert burden in pediatric hospitals: a cross-sectional analysis of six academic pediatric health systems using novel metrics. J Am Med Inform Assoc 2021; 28:2654-2660. [PMID: 34664664 DOI: 10.1093/jamia/ocab179] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Revised: 07/02/2021] [Accepted: 09/10/2021] [Indexed: 11/14/2022] Open
Abstract
BACKGROUND Excessive electronic health record (EHR) alerts reduce the salience of actionable alerts. Little is known about the frequency of interruptive alerts across health systems and how the choice of metric affects which users appear to have the highest alert burden. OBJECTIVE (1) Analyze alert burden by alert type, care setting, provider type, and individual provider across 6 pediatric health systems. (2) Compare alert burden using different metrics. MATERIALS AND METHODS We analyzed interruptive alert firings logged in EHR databases at 6 pediatric health systems from 2016-2019 using 4 metrics: (1) alerts per patient encounter, (2) alerts per inpatient-day, (3) alerts per 100 orders, and (4) alerts per unique clinician days (calendar days with at least 1 EHR log in the system). We assessed intra- and interinstitutional variation and how alert burden rankings differed based on the chosen metric. RESULTS Alert burden varied widely across institutions, ranging from 0.06 to 0.76 firings per encounter, 0.22 to 1.06 firings per inpatient-day, 0.98 to 17.42 per 100 orders, and 0.08 to 3.34 firings per clinician day logged in the EHR. Custom alerts accounted for the greatest burden at all 6 sites. The rank order of institutions by alert burden was similar regardless of which alert burden metric was chosen. Within institutions, the alert burden metric choice substantially affected which provider types and care settings appeared to experience the highest alert burden. CONCLUSION Estimates of the clinical areas with highest alert burden varied substantially by institution and based on the metric used.
Collapse
Affiliation(s)
- Evan W Orenstein
- Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, USA.,Division of Hospital Medicine, Children's Healthcare of Atlanta, Atlanta, Georgia, USA
| | | | - Naveen Muthu
- Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA.,Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
| | - Juan D Chaparro
- Division of Clinical Informatics, Nationwide Children's Hospital, Columbus, Ohio, USA.,Department of Pediatrics, The Ohio State University, Columbus, Ohio, USA
| | - Philip A Hagedorn
- Department of Pediatrics, University of Cincinnati, Cincinnati, Ohio, USA.,Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
| | - Adam C Dziorny
- Department of Pediatrics, University of Rochester School of Medicine, Rochester, New York, USA.,Division of Critical Care Medicine, Golisano Children's Hospital at Strong, Rochester, New York, USA
| | - Adam Moses
- Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA
| | - Sean Hernandez
- Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA.,Department of General Internal Medicine, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA
| | - Amina Khan
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
| | - Hannah B Huth
- Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA
| | - Jonathan M Beus
- Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA.,Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
| | - Eric S Kirkendall
- Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA.,Department of Pediatrics, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA
| |
Collapse
|
20
|
Kinlay M, Ho LMR, Zheng WY, Burke R, Juraskova I, Moles R, Baysari M. Electronic Medication Management Systems: Analysis of Enhancements to Reduce Errors and Improve Workflow. Appl Clin Inform 2021; 12:1049-1060. [PMID: 34758493 DOI: 10.1055/s-0041-1739196] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/29/2023] Open
Abstract
BACKGROUND Electronic medication management (eMM) has been shown to reduce medication errors; however, new safety risks have also been introduced that are associated with system use. No research has specifically examined the changes made to eMM systems to mitigate these risks. OBJECTIVES To (1) identify system-related medication errors or workflow blocks that were the target of eMM system updates, including the types of medications involved, and (2) describe and classify the system enhancements made to target these risks. METHODS In this retrospective qualitative study, documents detailing updates made from November 2014 to December 2019 to an eMM system were reviewed. Medication-related updates were classified according to "rationale for changes" and "changes made to the system." RESULTS One hundred and seventeen updates, totaling 147 individual changes, were made to the eMM system over the 4-year period. The most frequent reasons for changes being made to the eMM were to prevent medication errors (24% of reasons), optimize workflow (22%), and support "work as done" on paper (16%). The most frequent changes made to the eMM were options added to lists (14% of all changes), extra information made available on the screen (8%), and the wording or phrasing of text modified (8%). Approximately a third of the updates (37%) related to high-risk medications. The reasons for system changes appeared to vary over time, as eMM functionality and use expanded. CONCLUSION To our knowledge, this is the first study to systematically review and categorize system updates made to overcome new safety risks associated with eMM use. Optimization of eMM is an ongoing process, which changes over time as users become more familiar with the system and use is expanded to more sites. Continuous monitoring of the system is necessary to detect areas for improvement and capitalize on the benefits an electronic system can provide.
Collapse
Affiliation(s)
- Madaline Kinlay
- Biomedical Informatics and Digital Health, School of Medical Sciences, Charles Perkins Centre, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia
| | | | | | - Rosemary Burke
- Pharmacy Services, Sydney Local Health District, Sydney, Australia
| | - Ilona Juraskova
- School of Psychology, Faculty of Science, The University of Sydney, Sydney, Australia
| | - Rebekah Moles
- School of Pharmacy, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia
| | - Melissa Baysari
- Biomedical Informatics and Digital Health, School of Medical Sciences, Charles Perkins Centre, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia
| |
Collapse
|
21
|
Hong P, Herigon JC, Uptegraft C, Samuel B, Brown DL, Bickel J, Hron JD. Use of clinical data to augment healthcare worker contact tracing during the COVID-19 pandemic. J Am Med Inform Assoc 2021; 29:142-148. [PMID: 34623426 DOI: 10.1093/jamia/ocab231] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2021] [Revised: 09/28/2021] [Accepted: 10/06/2021] [Indexed: 11/13/2022] Open
Abstract
OBJECTIVE This work examined the secondary use of clinical data from the electronic health record (EHR) for screening our healthcare worker (HCW) population for potential exposures to patients with coronavirus disease 2019. MATERIALS AND METHODS We conducted a cross-sectional study at a free-standing, quaternary care pediatric hospital comparing first-degree, patient-HCW pairs identified by the hospital's COVID-19 contact tracing team (CTT) to those identified using EHR clinical event data (EHR Report). The primary outcome was the number of patient-HCW pairs detected by each process. RESULTS Among 233 patients with COVID-19, our EHR Report identified 4,116 patient-HCW pairs, including 2,365 (30.0%) of the 7,890 pairs detected by the CTT. The EHR Report also revealed 1,751 pairs not identified by the CTT. The highest number of patient-HCW pairs per patient was detected in the inpatient care venue. Nurses comprised the most frequently identified HCW role overall. CONCLUSION Automated methods to screen HCWs for potential exposure to patients with COVID-19 using clinical event data from the EHR are likely to improve epidemiologic surveillance by contact tracing programs and represent a viable and readily available strategy which should be considered by other institutions.
Collapse
Affiliation(s)
- Peter Hong
- Division of General Pediatrics, Department of Pediatrics, Boston Children's Hospital, Boston, Massachusetts, USA.,Department of Pediatrics, Harvard Medical School, Boston, Massachusetts, USA
| | - Joshua C Herigon
- Division of Pediatric Infectious Diseases, Department of Pediatrics, Children's Mercy Kansas City, Kansas City, Missouri, USA.,Department of Pediatrics, University of Missouri-Kansas City School of Medicine, USA, Kansas City, Missouri
| | - Colby Uptegraft
- Health Informatics Branch, Defense Health Agency, Falls Church, Virginia, USA
| | - Bassem Samuel
- Information Services Department, Boston Children's Hospital, Boston, Massachusetts, USA
| | - D Levin Brown
- Information Services Department, Boston Children's Hospital, Boston, Massachusetts, USA
| | - Jonathan Bickel
- Division of General Pediatrics, Department of Pediatrics, Boston Children's Hospital, Boston, Massachusetts, USA.,Department of Pediatrics, Harvard Medical School, Boston, Massachusetts, USA.,Information Services Department, Boston Children's Hospital, Boston, Massachusetts, USA.,Computational Health Informatics Program, Boston Children's Hospital, Boston, Massachusetts, USA
| | - Jonathan D Hron
- Division of General Pediatrics, Department of Pediatrics, Boston Children's Hospital, Boston, Massachusetts, USA.,Department of Pediatrics, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
22
|
Xiao SQ, Liu JE, Chang H. Physician-Nurse Communication Surrounding Computerized Physician Order Entry Systems From Social and Technical Perspective: An Ethnographic Study. Comput Inform Nurs 2021; 40:258-268. [PMID: 35394959 DOI: 10.1097/cin.0000000000000809] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
Although computerized physician order entry systems improve order transmission and patient safety, overdependence on these systems can impede users' communication. This ethnographic study explored physician-nurse communication surrounding computerized physician order entry systems using a sociotechnical framework. Fieldwork conducted in a tertiary teaching hospital comprised 89 hours of participant observation, and individual semistructured interviews were held with seven nurses and five physicians. In addition, documents and artifacts were collected. Three core themes emerged. First, computerized physician order entry quality-related issues undermined the work efficiency of physicians and nurses. Specifically, usability was error prone because of cognitive overload, and the system was unable to perform relevant traces and raise alerts, demonstrating poor interoperability. Second, social factors, including insufficient training, unclear responsibilities, and a lack of awareness concerning interdisciplinary communication, compounded communication problems. Last, environmental factors, including noncoterminous spaces and times and insufficient technical support, impeded the resolution of communication problems. Technical and social contextual factors relating to computerized physician order entry systems jointly affected physician-nurse communication. Cognitive issues and insufficient alerts impacted work efficiency the most and were compounded by contextual individual- and team-related factors and environmental factors. Therefore, improved functions of computerized physician order entry systems and interprofessional communication training are required to optimize technical and social aspects of physician-nurse communication.
Collapse
Affiliation(s)
- Shu-Qin Xiao
- Author Affiliations: School of Nursing (Ms Xiao and Dr Liu) and Department of Neurology, Xuanwu Hospital (Ms Chang), Capital Medical University, Beijing, China
| | | | | |
Collapse
|
23
|
Grant SR, Smith BD, Pandey P, Maldonado JA, Kim M, Moon BS, Colbert LE. Does a Custom Electronic Health Record Alert System Improve Physician Compliance With National Quality Measures for Palliative Bone Metastasis Radiotherapy? JCO Clin Cancer Inform 2021; 5:36-44. [PMID: 33411621 DOI: 10.1200/cci.20.00115] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
PURPOSE In an effort to promote cost-conscious, high-quality, and patient-centered care in the palliative radiation of painful bone metastases, the National Quality Forum (NQF) formed measure 1822 in 2012, which recommends the use of one of the four dose-fractionation schemes (30 Gy in 10 fractions, 24 Gy in 6 fractions, 20 Gy in 5 fractions, or 8 Gy in a single fraction). We investigated whether a custom electronic health record (EHR) alert system improved quality measure compliance among 88 physicians at a large academic center and institutional network. METHODS In March 2018, a multiphase alert system was embedded in a custom web-based EHR. Prior to a course of palliative bone radiation, the alert system notified the user of NQF 1822 recommendations and, once prescription was completed, either affirmed compliance or advised a change in treatment schedule. Rates of compliance were evaluated before and after implementation of alert system. RESULTS Of 2,399 treatment courses, 86.5% were compliant with NQF 1822 recommendations. There was no difference in rates of NQF 1822 compliance before or after implementation of the custom EHR alert (86.0% before March 2018 v 86.9% during and after March 2018, P = .551). CONCLUSION There was no change in rates of compliance following implementation of a custom EHR alert system designed to make treatment recommendations based on national quality measure guidelines. To be of most benefit, future palliative bone metastasis decision aids should leverage peer review, target a clear practice deficiency, center upon high-quality practice guidelines, and allow flexibility to reflect the diversity of clinical scenarios.
Collapse
Affiliation(s)
- Stephen R Grant
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX
| | - Benjamin D Smith
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX
| | - Prasamsa Pandey
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX
| | - J Alberto Maldonado
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX.,Department of Radiation Oncology, The University of Texas Medical Branch, Galveston, TX
| | - Minsoo Kim
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX
| | - Bryan S Moon
- Department of Orthopedic Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX
| | - Lauren E Colbert
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX
| |
Collapse
|
24
|
Marcilly R, Zheng WY, Beuscart R, Baysari MT. Comparison of the validity, perceived usefulness and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability: a study protocol. BMJ Open 2021; 11:e050448. [PMID: 34353806 PMCID: PMC8344302 DOI: 10.1136/bmjopen-2021-050448] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
Abstract
INTRODUCTION Research has shown that improvements to the usability of medication alert systems are needed. For designers and decisions-makers to assess usability of their alert systems, two paper-based tools are currently available: the instrument for evaluating human-factors principles in medication-related decision support alerts (I-MeDeSA) and the tool for evaluating medication alerting systems (TEMAS). This study aims to compare the validity, usability and usefulness of both tools to identify their strengths and limitations and assist designers and decision-makers in making an informed decision about which tool is most suitable for assessing their current or prospective system. METHODS AND ANALYSIS First, TEMAS and I-MeDeSA will be translated into French. This translation will be validated by three experts in human factors. Then, in 12 French hospitals with a medication alert system in place, staff with expertise in the system will evaluate their alert system using the two tools successively. After the use of each tool, participants will be asked to fill in the System Usability Scale (SUS) and complete a survey on the understandability and perceived usefulness of each tool. Following the completion of both assessments, participants will be asked to nominate their preferred tool and relay their opinions on the tools. The design philosophy of TEMAS and I-MeDeSA differs on the calculation of a score, impacting the way the comparison between the tools can be performed. Convergent validity will be evaluated by matching the items of the two tools with respect to the usability dimensions they assess. SUS scores and answers to the survey will be statistically compared for I-MeDeSA and TEMAS to identify differences. Free-text responses in surveys will be analysed using an inductive approach. ETHICS AND DISSEMINATION Ethical approval is not required in France for a study of this nature. The results will be published in a peer-reviewed journal.
Collapse
Affiliation(s)
- Romaric Marcilly
- Univ. Lille, CHU Lille, ULR 2694 - METRICS: Évaluation des technologies de santé et des pratiques médicales, F-59000 Lille, France
- Inserm, CIC-IT 1403, F-59000 Lille, France
| | - Wu Yi Zheng
- Black Dog Institute, Randwick, New South Wales, Australia
| | - Regis Beuscart
- Univ. Lille, CHU Lille, ULR 2694 - METRICS: Évaluation des technologies de santé et des pratiques médicales, F-59000 Lille, France
| | - Melissa T Baysari
- The University of Sydney, Faculty of Medicine and Health, School of Medical Sciences, Biomedical Informatics and Digital Health, Sydney, New South Wales, Australia
| |
Collapse
|
25
|
Hirsch JS, Brar R, Forrer C, Sung C, Roycroft R, Seelamneni P, Dabir H, Naseer A, Gautam-Goyal P, Bock KR, Oppenheim MI. Design, development, and deployment of an indication- and kidney function-based decision support tool to optimize treatment and reduce medication dosing errors. JAMIA Open 2021; 4:ooab039. [PMID: 34222830 PMCID: PMC8242134 DOI: 10.1093/jamiaopen/ooab039] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2021] [Revised: 04/13/2021] [Accepted: 04/26/2021] [Indexed: 11/19/2022] Open
Abstract
Delivering clinical decision support (CDS) at the point of care has long been considered a major advantage of computerized physician order entry (CPOE). Despite the widespread implementation of CPOE, medication ordering errors and associated adverse events still occur at an unacceptable level. Previous attempts at indication- and kidney function-based dosing have mostly employed intrusive CDS, including interruptive alerts with poor usability. This descriptive work describes the design, development, and deployment of the Adult Dosing Methodology (ADM) module, a novel CDS tool that provides indication- and kidney-based dosing at the time of order entry. Inclusion of several antimicrobials in the initial set of medications allowed for the additional goal of optimizing therapy duration for appropriate antimicrobial stewardship. The CDS aims to decrease order entry errors and burden on providers by offering automatic dose and frequency recommendations, integration within the native electronic health record, and reasonable knowledge maintenance requirements. Following implementation, early utilization demonstrated high acceptance of automated recommendations, with up to 96% of provided automated recommendations accepted by users.
Collapse
Affiliation(s)
- Jamie S Hirsch
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA.,Division of Kidney Diseases and Hypertension, Department of Medicine, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Great Neck, New York, USA.,Center for Health Innovations and Outcomes Research, Institute of Health System Science, Feinstein Institutes for Medical Research, Manhasset, New York, USA
| | - Rajdeep Brar
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA
| | - Christopher Forrer
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA
| | - Christine Sung
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA
| | - Richard Roycroft
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA
| | - Pradeep Seelamneni
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA
| | - Hemala Dabir
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA
| | - Ambareen Naseer
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA
| | - Pranisha Gautam-Goyal
- Division of Infectious Diseases, Department of Medicine, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Manhasset, New York, USA
| | - Kevin R Bock
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA.,Department of Pediatrics, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York, USA
| | - Michael I Oppenheim
- Department of Information Services, Northwell Health, New Hyde Park, New York, USA.,Division of Infectious Diseases, Department of Medicine, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Manhasset, New York, USA
| |
Collapse
|
26
|
Colicchio TK, Dissanayake PI, Cimino JJ. Formal representation of patients' care context data: the path to improving the electronic health record. J Am Med Inform Assoc 2021; 27:1648-1657. [PMID: 32935127 PMCID: PMC7671623 DOI: 10.1093/jamia/ocaa134] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2020] [Revised: 05/15/2020] [Accepted: 06/10/2020] [Indexed: 11/24/2022] Open
Abstract
Objective To develop a collection of concept-relationship-concept tuples to formally represent patients’ care context data to inform electronic health record (EHR) development. Materials and Methods We reviewed semantic relationships reported in the literature and developed a manual annotation schema. We used the initial schema to annotate sentences extracted from narrative note sections of cardiology, urology, and ear, nose, and throat (ENT) notes. We audio recorded ENT visits and annotated their parsed transcripts. We combined the results of each annotation into a consolidated set of concept-relationship-concept tuples. We then compared the tuples used within and across the multiple data sources. Results We annotated a total of 626 sentences. Starting with 8 relationships from the literature, we annotated 182 sentences from 8 inpatient consult notes (initial set of tuples = 43). Next, we annotated 232 sentences from 10 outpatient visit notes (enhanced set of tuples = 75). Then, we annotated 212 sentences from transcripts of 5 outpatient visits (final set of tuples = 82). The tuples from the visit transcripts covered 103 (74%) concepts documented in the notes of their respective visits. There were 20 (24%) tuples used across all data sources, 10 (12%) used only in inpatient notes, 15 (18%) used only in visit notes, and 7 (9%) used only in the visit transcripts. Conclusions We produced a robust set of 82 tuples useful to represent patients’ care context data. We propose several applications of our tuples to improve EHR navigation, data entry, learning health systems, and decision support.
Collapse
Affiliation(s)
| | | | - James J Cimino
- Informatics Institute, University of Alabama at Birmingham, USA
| |
Collapse
|
27
|
Chin YPH, Song W, Lien CE, Yoon CH, Wang WC, Liu J, Nguyen PA, Feng YT, Zhou L, Li YCJ, Bates DW. Assessing the International Transferability of a Machine Learning Model for Detecting Medication Error in the General Internal Medicine Clinic: Multicenter Preliminary Validation Study. JMIR Med Inform 2021; 9:e23454. [PMID: 33502331 PMCID: PMC7875695 DOI: 10.2196/23454] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2020] [Revised: 11/27/2020] [Accepted: 12/12/2020] [Indexed: 01/17/2023] Open
Abstract
BACKGROUND Although most current medication error prevention systems are rule-based, these systems may result in alert fatigue because of poor accuracy. Previously, we had developed a machine learning (ML) model based on Taiwan's local databases (TLD) to address this issue. However, the international transferability of this model is unclear. OBJECTIVE This study examines the international transferability of a machine learning model for detecting medication errors and whether the federated learning approach could further improve the accuracy of the model. METHODS The study cohort included 667,572 outpatient prescriptions from 2 large US academic medical centers. Our ML model was applied to build the original model (O model), the local model (L model), and the hybrid model (H model). The O model was built using the data of 1.34 billion outpatient prescriptions from TLD. A validation set with 8.98% (60,000/667,572) of the prescriptions was first randomly sampled, and the remaining 91.02% (607,572/667,572) of the prescriptions served as the local training set for the L model. With a federated learning approach, the H model used the association values with a higher frequency of co-occurrence among the O and L models. A testing set with 600 prescriptions was classified as substantiated and unsubstantiated by 2 independent physician reviewers and was then used to assess model performance. RESULTS The interrater agreement was significant in terms of classifying prescriptions as substantiated and unsubstantiated (κ=0.91; 95% CI 0.88 to 0.95). With thresholds ranging from 0.5 to 1.5, the alert accuracy ranged from 75%-78% for the O model, 76%-78% for the L model, and 79%-85% for the H model. CONCLUSIONS Our ML model has good international transferability among US hospital data. Using the federated learning approach with local hospital data could further improve the accuracy of the model.
Collapse
Affiliation(s)
- Yen Po Harvey Chin
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, United States
- College of Medical Science and Technology, Graduate Institute of Biomedical Informatics, Taipei Medical University, Taipei City, Taiwan
| | - Wenyu Song
- Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States
| | - Chia En Lien
- Doctor of Public Health Program, Harvard TH Chan School of Public Health, Boston, MA, United States
| | - Chang Ho Yoon
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, United States
| | - Wei-Chen Wang
- Department of Epidemiology, Harvard TH Chan School of Public Health, Boston, MA, United States
| | - Jennifer Liu
- Department of Emergency Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States
| | - Phung Anh Nguyen
- College of Medical Science and Technology, Graduate Institute of Biomedical Informatics, Taipei Medical University, Taipei City, Taiwan
- International Center for Health Information Technology, Taipei Medical University, Taipei City, Taiwan
| | - Yi Ting Feng
- College of Medical Science and Technology, Graduate Institute of Biomedical Informatics, Taipei Medical University, Taipei City, Taiwan
| | - Li Zhou
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States
| | - Yu Chuan Jack Li
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei City, Taiwan
- Department of Dermatology, Taipei Municipal Wan Fang Hospital, Taipei City, Taiwan
| | - David Westfall Bates
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States
- Clinical and Quality Analysis, Information Systems, Partners HealthCare, Somerville, MA, United States
| |
Collapse
|
28
|
Poly TN, Islam MM, Muhtar MS, Yang HC, Nguyen PAA, Li YCJ. Machine Learning Approach to Reduce Alert Fatigue Using a Disease Medication-Related Clinical Decision Support System: Model Development and Validation. JMIR Med Inform 2020; 8:e19489. [PMID: 33211018 PMCID: PMC7714650 DOI: 10.2196/19489] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2020] [Revised: 09/12/2020] [Accepted: 09/19/2020] [Indexed: 12/28/2022] Open
Abstract
Background Computerized physician order entry (CPOE) systems are incorporated into clinical decision support systems (CDSSs) to reduce medication errors and improve patient safety. Automatic alerts generated from CDSSs can directly assist physicians in making useful clinical decisions and can help shape prescribing behavior. Multiple studies reported that approximately 90%-96% of alerts are overridden by physicians, which raises questions about the effectiveness of CDSSs. There is intense interest in developing sophisticated methods to combat alert fatigue, but there is no consensus on the optimal approaches so far. Objective Our objective was to develop machine learning prediction models to predict physicians’ responses in order to reduce alert fatigue from disease medication–related CDSSs. Methods We collected data from a disease medication–related CDSS from a university teaching hospital in Taiwan. We considered prescriptions that triggered alerts in the CDSS between August 2018 and May 2019. Machine learning models, such as artificial neural network (ANN), random forest (RF), naïve Bayes (NB), gradient boosting (GB), and support vector machine (SVM), were used to develop prediction models. The data were randomly split into training (80%) and testing (20%) datasets. Results A total of 6453 prescriptions were used in our model. The ANN machine learning prediction model demonstrated excellent discrimination (area under the receiver operating characteristic curve [AUROC] 0.94; accuracy 0.85), whereas the RF, NB, GB, and SVM models had AUROCs of 0.93, 0.91, 0.91, and 0.80, respectively. The sensitivity and specificity of the ANN model were 0.87 and 0.83, respectively. Conclusions In this study, ANN showed substantially better performance in predicting individual physician responses to an alert from a disease medication–related CDSS, as compared to the other models. To our knowledge, this is the first study to use machine learning models to predict physician responses to alerts; furthermore, it can help to develop sophisticated CDSSs in real-world clinical settings.
Collapse
Affiliation(s)
- Tahmina Nasrin Poly
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.,International Center for Health Information Technology, Taipei Medical University, Taipei, Taiwan.,Research Center of Big Data and Meta-analysis, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan
| | - Md Mohaimenul Islam
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.,International Center for Health Information Technology, Taipei Medical University, Taipei, Taiwan.,Research Center of Big Data and Meta-analysis, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan
| | | | - Hsuan-Chia Yang
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.,International Center for Health Information Technology, Taipei Medical University, Taipei, Taiwan.,Research Center of Big Data and Meta-analysis, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan
| | - Phung Anh Alex Nguyen
- International Center for Health Information Technology, Taipei Medical University, Taipei, Taiwan.,Department of Healthcare Information & Management, Ming Chuan University, Taoyuan City, Taiwan
| | - Yu-Chuan Jack Li
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.,International Center for Health Information Technology, Taipei Medical University, Taipei, Taiwan.,Research Center of Big Data and Meta-analysis, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan.,Department of Dermatology, Wan Fang Hospital, Taipei, Taiwan.,TMU Research Center of Cancer Translational Medicine, Taipei Medical University, Taipei, Taiwan
| |
Collapse
|
29
|
Monteith S, Glenn T, Gitlin M, Bauer M. Potential Drug interactions with Drugs used for Bipolar Disorder: A Comparison of 6 Drug Interaction Database Programs. PHARMACOPSYCHIATRY 2020; 53:220-227. [DOI: 10.1055/a-1156-4193] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
Abstract
AbstractBackground Patients with bipolar disorder frequently experience polypharmacy, putting them at risk for clinically significant drug-drug interactions (DDI). Online drug interaction database programs are used to alert physicians, but there are no internationally recognized standards to define DDI. This study compared the category of potential DDI returned by 6 commercial drug interaction database programs for drug interaction pairs involving drugs commonly prescribed for bipolar disorder.Methods The category of potential DDI provided by 6 drug interaction database programs (3 subscription, 3 open access) was obtained for 125 drug interaction pairs. The pairs involved 103 drugs (38 psychiatric, 65 nonpsychiatric); 88 pairs included a psychiatric and nonpsychiatric drug; 37 pairs included 2 psychiatric drugs. Every pair contained at least 1 mood stabilizer or antidepressant. The category provided by 6 drug interaction database programs was compared using percent agreement and Fleiss kappa statistic of interrater reliability.Results For the 125 drug pairs, the overall percent agreement among the 6 drug interaction database programs was 60%; the Fleiss kappa agreement was slight. For drug interaction pairs with any category rating of severe (contraindicated), the kappa agreement was moderate. For drug interaction pairs with any category rating of major, the kappa agreement was slight.Conclusion There is poor agreement among drug interaction database programs for the category of potential DDI involving psychiatric drugs. Drug interaction database programs provide valuable information, but the lack of consistency should be recognized as a limitation. When assistance is needed, physicians should check more than 1 drug interaction database program.
Collapse
Affiliation(s)
- Scott Monteith
- Michigan State University College of Human Medicine, Traverse City Campus, Traverse City, MI, USA
| | - Tasha Glenn
- ChronoRecord Association, Fullerton, CA, USA
| | - Michael Gitlin
- Department of Psychiatry and Biobehavioral Sciences, Semel Institute for Neuroscience and Human Behavior, University of California Los Angeles (UCLA), Los Angeles, CA, USA
| | - Michael Bauer
- Department of Psychiatry and Psychotherapy, University Hospital Carl Gustav Carus, Medical Faculty, Technische Universität Dresden, Dresden, Germany
| |
Collapse
|
30
|
Baysari MT, Zheng WY, Van Dort B, Reid-Anderson H, Gronski M, Kenny E. A Late Attempt to Involve End Users in the Design of Medication-Related Alerts: Survey Study. J Med Internet Res 2020; 22:e14855. [PMID: 32167479 PMCID: PMC7101499 DOI: 10.2196/14855] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2019] [Revised: 12/04/2019] [Accepted: 12/16/2019] [Indexed: 11/21/2022] Open
Abstract
Background When users of electronic medical records (EMRs) are presented with large numbers of irrelevant computerized alerts, they experience alert fatigue, begin to ignore alert information, and override alerts without processing or heeding alert recommendations. Anecdotally, doctors at our study site were dissatisfied with the medication-related alerts being generated, both in terms of volume being experienced and clinical relevance. Objective This study aimed to involve end users in the redesign of medication-related alerts in a hospital EMR, 4 years post implementation. Methods This work was undertaken at a private not-for-profit teaching hospital in Sydney, Australia. Since EMR implementation in 2015, the organization elected to implement all medication-related alert types available in the system for prescribers: allergy and intolerance alerts, therapeutic duplication alerts, pregnancy alerts, and drug-drug interaction alerts. The EMR included no medication administration alerts for nurses. To obtain feedback on current alerts and suggestions for redesign, a Web-based survey was distributed to all doctors and nurses at the site via hospital mailing lists. Results Despite a general dissatisfaction with alerts, very few end users completed the survey. In total, only 3.37% (36/1066) of doctors and 14.5% (60/411) of nurses took part. Approximately 90% (30/33) of doctors who responded held the view that too many alerts were triggered in the EMR. Doctors suggested that most alerts be removed and that alerts be more specific and less sensitive. In contrast, 97% (58/60) of the nurse respondents indicated that they would like to receive medication administration alerts in the EMR. Most nurses indicated that they would like to receive all the alert types available at all severity levels. Conclusions Attempting to engage with end users several years post implementation was challenging. Involving users so late in the implementation process may lead to clinicians viewing the provision of feedback to be futile. Seeking user feedback on usefulness, volume, and design of alerts is extremely valuable; however, we suggest this is undertaken early, preferably before system implementation.
Collapse
Affiliation(s)
- Melissa Therese Baysari
- Faculty of Health Sciences, The University of Sydney, Sydney, Australia.,Centre for Health Systems & Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Wu Yi Zheng
- Faculty of Health Sciences, The University of Sydney, Sydney, Australia.,Centre for Health Systems & Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Bethany Van Dort
- Faculty of Health Sciences, The University of Sydney, Sydney, Australia.,Centre for Health Systems & Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | | | | | - Eliza Kenny
- Macquarie University Hospital, Sydney, Australia
| |
Collapse
|
31
|
Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing Interruptive Alert Burden Using Quality Improvement Methodology. Appl Clin Inform 2020; 11:46-58. [PMID: 31940671 DOI: 10.1055/s-0039-3402757] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022] Open
Abstract
BACKGROUND Increased adoption of electronic health records (EHR) with integrated clinical decision support (CDS) systems has reduced some sources of error but has led to unintended consequences including alert fatigue. The "pop-up" or interruptive alert is often employed as it requires providers to acknowledge receipt of an alert by taking an action despite the potential negative effects of workflow interruption. We noted a persistent upward trend of interruptive alerts at our institution and increasing requests for new interruptive alerts. OBJECTIVES Using Institute for Healthcare Improvement (IHI) quality improvement (QI) methodology, the primary objective was to reduce the total volume of interruptive alerts received by providers. METHODS We created an interactive dashboard for baseline alert data and to monitor frequency and outcomes of alerts as well as to prioritize interventions. A key driver diagram was developed with a specific aim to decrease the number of interruptive alerts from a baseline of 7,250 to 4,700 per week (35%) over 6 months. Interventions focused on the following key drivers: appropriate alert display within workflow, clear alert content, alert governance and standardization, user feedback regarding overrides, and respect for user knowledge. RESULTS A total of 25 unique alerts accounted for 90% of the total interruptive alert volume. By focusing on these 25 alerts, we reduced interruptive alerts from 7,250 to 4,400 per week. CONCLUSION Systematic and structured improvements to interruptive alerts can lead to overall reduced interruptive alert burden. Using QI methods to prioritize our interventions allowed us to maximize our impact. Further evaluation should be done on the effects of reduced interruptive alerts on patient care outcomes, usability heuristics on cognitive burden, and direct feedback mechanisms on alert utility.
Collapse
Affiliation(s)
- Juan D Chaparro
- Division of Clinical Informatics, Nationwide Children's Hospital, Columbus, Ohio, United States.,Department of Pediatrics, The Ohio State University College of Medicine, Columbus, Ohio, United States
| | - Cory Hussain
- Department of Family Medicine, The Ohio State University College of Medicine, Columbus, Ohio, United States
| | - Jennifer A Lee
- Department of Family Medicine, The Ohio State University College of Medicine, Columbus, Ohio, United States
| | - Jessica Hehmeyer
- Department of Information Services, Nationwide Children's Hospital, Columbus, Ohio, United States
| | - Manjusri Nguyen
- Department of Information Services, Nationwide Children's Hospital, Columbus, Ohio, United States
| | - Jeffrey Hoffman
- Division of Clinical Informatics, Nationwide Children's Hospital, Columbus, Ohio, United States.,Department of Pediatrics, The Ohio State University College of Medicine, Columbus, Ohio, United States
| |
Collapse
|
32
|
McGreevey JD, Mallozzi CP, Perkins RM, Shelov E, Schreiber R. Reducing Alert Burden in Electronic Health Records: State of the Art Recommendations from Four Health Systems. Appl Clin Inform 2020; 11:1-12. [PMID: 31893559 DOI: 10.1055/s-0039-3402715] [Citation(s) in RCA: 66] [Impact Index Per Article: 16.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022] Open
Abstract
BACKGROUND Electronic health record (EHR) alert fatigue, while widely recognized as a concern nationally, lacks a corresponding comprehensive mitigation plan. OBJECTIVES The goal of this manuscript is to provide practical guidance to clinical informaticists and other health care leaders who are considering creating a program to manage EHR alerts. METHODS This manuscript synthesizes several approaches and recommendations for better alert management derived from four U.S. health care institutions that presented their experiences and recommendations at the American Medical Informatics Association 2019 Clinical Informatics Conference in Atlanta, Georgia, United States. The assembled health care institution leaders represent academic, pediatric, community, and specialized care domains. We describe governance and management, structural concepts and components, and human-computer interactions with alerts, and make recommendations regarding these domains based on our experience supplemented with literature review. This paper focuses on alerts that impact bedside clinicians. RESULTS The manuscript addresses the range of considerations relevant to alert management including a summary of the background literature about alerts, alert governance, alert metrics, starting an alert management program, approaches to evaluating alerts prior to deployment, and optimization of existing alerts. The manuscript includes examples of alert optimization successes at two of the represented institutions. In addition, we review limitations on the ability to evaluate alerts in the current state and identify opportunities for further scholarship. CONCLUSION Ultimately, alert management programs must strive to meet common goals of improving patient care, while at the same time decreasing the alert burden on clinicians. In so doing, organizations have an opportunity to promote the wellness of patients, clinicians, and EHRs themselves.
Collapse
Affiliation(s)
- John D McGreevey
- Office of the CMIO, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States.,Section of Hospital Medicine, Division of General Internal Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, United States
| | - Colleen P Mallozzi
- Office of the CMIO, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
| | - Randa M Perkins
- H. Lee Moffitt Cancer Center and Research Institute, Tampa, Florida, United States
| | - Eric Shelov
- Division of General Pediatrics, Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
| | - Richard Schreiber
- Physician Informatics and Department of Medicine, Geisinger Health System, Geisinger Holy Spirit, Camp Hill, Pennsylvania, United States
| |
Collapse
|
33
|
Moja L, Polo Friz H, Capobussi M, Kwag K, Banzi R, Ruggiero F, González-Lorenzo M, Liberati EG, Mangia M, Nyberg P, Kunnamo I, Cimminiello C, Vighi G, Grimshaw JM, Delgrossi G, Bonovas S. Effectiveness of a Hospital-Based Computerized Decision Support System on Clinician Recommendations and Patient Outcomes: A Randomized Clinical Trial. JAMA Netw Open 2019; 2:e1917094. [PMID: 31825499 PMCID: PMC6991299 DOI: 10.1001/jamanetworkopen.2019.17094] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
IMPORTANCE Sophisticated evidence-based information resources can filter medical evidence from the literature, integrate it into electronic health records, and generate recommendations tailored to individual patients. OBJECTIVE To assess the effectiveness of a computerized clinical decision support system (CDSS) that preappraises evidence and provides health professionals with actionable, patient-specific recommendations at the point of care. DESIGN, SETTING, AND PARTICIPANTS Open-label, parallel-group, randomized clinical trial among internal medicine wards of a large Italian general hospital. All analyses in this randomized clinical trial followed the intent-to-treat principle. Between November 1, 2015, and December 31, 2016, patients were randomly assigned to the intervention group, in which CDSS-generated reminders were displayed to physicians, or to the control group, in which reminders were generated but not shown. Data were analyzed between February 1 and July 31, 2018. INTERVENTIONS Evidence-Based Medicine Electronic Decision Support (EBMEDS), a commercial CDSS covering a wide array of health conditions across specialties, was integrated into the hospital electronic health records to generate patient-specific recommendations. MAIN OUTCOMES AND MEASURES The primary outcome was the resolution rate, the rate at which medical problems identified and alerted by the CDSS were addressed by a change in practice. Secondary outcomes included the length of hospital stay and in-hospital all-cause mortality. RESULTS In this randomized clinical trial, 20 563 patients were admitted to the hospital. Of these, 6480 (31.5%) were admitted to the internal medicine wards (study population) and randomized (3242 to CDSS and 3238 to control). The mean (SD) age of patients was 70.5 (17.3) years, and 54.5% were men. In total, 28 394 reminders were generated throughout the course of the trial (median, 3 reminders per patient per hospital stay; interquartile range [IQR], 1-6). These messages led to a change in practice in approximately 4 of 100 patients. The resolution rate was 38.0% (95% CI, 37.2%-38.8%) in the intervention group and 33.7% (95% CI, 32.9%-34.4%) in the control group, corresponding to an odds ratio of 1.21 (95% CI, 1.11-1.32; P < .001). The length of hospital stay did not differ between the groups, with a median time of 8 days (IQR, 5-13 days) for the intervention group and a median time of 8 days (IQR, 5-14 days) for the control group (P = .36). In-hospital all-cause mortality also did not differ between groups (odds ratio, 0.95; 95% CI, 0.77-1.17; P = .59). Alert fatigue did not differ between early and late study periods. CONCLUSIONS AND RELEVANCE An international commercial CDSS intervention marginally influenced routine practice in a general hospital, although the change did not statistically significantly affect patient outcomes. TRIAL REGISTRATION ClinicalTrials.gov identifier: NCT02577198.
Collapse
Affiliation(s)
- Lorenzo Moja
- Department of Biomedical Sciences for Health, University of Milan, Milan, Italy
- Clinical Epidemiology Unit, Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS) Orthopedic Institute Galeazzi, Milan, Italy
| | - Hernan Polo Friz
- Internal Medicine Division, Medical Department, Vimercate Hospital, Vimercate, Italy
| | - Matteo Capobussi
- Clinical Epidemiology Unit, Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS) Orthopedic Institute Galeazzi, Milan, Italy
| | - Koren Kwag
- Medical School of International Health, Ben Gurion University of the Negev, Beer Sheva, Israel
| | - Rita Banzi
- IRCCS Mario Negri Institute for Pharmacological Research, Milan, Italy
| | - Francesca Ruggiero
- Department of Biomedical Sciences for Health, University of Milan, Milan, Italy
- Clinical Epidemiology Unit, Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS) Orthopedic Institute Galeazzi, Milan, Italy
| | - Marien González-Lorenzo
- Humanitas Clinical and Research Center, Milan, Italy
- Department of Biomedical Sciences, Humanitas University, Milan, Italy
| | - Elisa G. Liberati
- The Healthcare Improvement Studies Institute, University of Cambridge, Cambridge, United Kingdom
| | | | - Peter Nyberg
- Duodecim Medical Publications Ltd, Helsinki, Finland
| | - Ilkka Kunnamo
- Duodecim Medical Publications Ltd, Helsinki, Finland
| | - Claudio Cimminiello
- Internal Medicine Division, Medical Department, Vimercate Hospital, Vimercate, Italy
| | - Giuseppe Vighi
- Internal Medicine Division, Medical Department, Vimercate Hospital, Vimercate, Italy
| | - Jeremy M. Grimshaw
- Clinical Epidemiology Program, Ottawa Hospital Research Institute and the Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Giovanni Delgrossi
- Internal Medicine Division, Medical Department, Vimercate Hospital, Vimercate, Italy
| | - Stefanos Bonovas
- Humanitas Clinical and Research Center, Milan, Italy
- Department of Biomedical Sciences, Humanitas University, Milan, Italy
| |
Collapse
|
34
|
Elias P, Peterson E, Wachter B, Ward C, Poon E, Navar AM. Evaluating the Impact of Interruptive Alerts within a Health System: Use, Response Time, and Cumulative Time Burden. Appl Clin Inform 2019; 10:909-917. [PMID: 31777057 DOI: 10.1055/s-0039-1700869] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022] Open
Abstract
BACKGROUND Health systems often employ interruptive alerts through the electronic health record to improve patient care. However, concerns of "alert fatigue" have been raised, highlighting the importance of understanding the time burden and impact of these alerts on providers. OBJECTIVES Our main objective was to determine the total time providers spent on interruptive alerts in both inpatient and outpatient settings. Our secondary objectives were to analyze dwell time for individual alerts and examine both provider and alert-related factors associated with dwell time variance. METHODS We retrospectively evaluated use and response to the 75 most common interruptive ("popup") alerts between June 1st, 2015 and November 1st, 2016 in a large academic health care system. Alert "dwell time" was calculated as the time between the alert appearing on a provider's screen until it was closed. The total number of alerts and dwell times per provider per month was calculated for inpatient and outpatient alerts and compared across alert type. RESULTS The median number of alerts seen by a provider was 12 per month (IQR 4-34). Overall, 67% of inpatient and 39% of outpatient alerts were closed in under 3 seconds. Alerts related to patient safety and those requiring more than a single click to proceed had significantly longer median dwell times of 5.2 and 6.7 seconds, respectively. The median total monthly time spent by providers viewing alerts was 49 seconds on inpatient alerts and 28 seconds on outpatient alerts. CONCLUSION Most alerts were closed in under 3 seconds and a provider's total time spent on alerts was less than 1 min/mo. Alert fatigue may lie in their interruptive and noncritical nature rather than time burden. Monitoring alert interaction time can function as a valuable metric to assess the impact of alerts on workflow and potentially identify routinely ignored alerts.
Collapse
Affiliation(s)
- Pierre Elias
- Department of Biomedical Informatics, Columbia University, New York, New York, United States
| | - Eric Peterson
- Duke Clinical Research Institute, Duke University, Durham, North Carolina, United States
| | - Bob Wachter
- Department of Medicine, University of California, San Francisco, California, United States
| | - Cary Ward
- Duke Clinical Research Institute, Duke University, Durham, North Carolina, United States
| | - Eric Poon
- Duke Health Technology Solutions, Duke University School of Medicine, Duke University, Durham, North Carolina, United States
| | - Ann Marie Navar
- Duke Clinical Research Institute, Duke University, Durham, North Carolina, United States
| |
Collapse
|
35
|
Kimmel HJ, Brice YN, Trikalinos TA, Sarkar IN, Ranney ML. Real-Time Emergency Department Electronic Notifications Regarding High-Risk Patients: A Systematic Review. Telemed J E Health 2019; 25:604-618. [PMID: 30129886 PMCID: PMC6664825 DOI: 10.1089/tmj.2018.0117] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2018] [Accepted: 06/10/2018] [Indexed: 11/13/2022] Open
Abstract
Background: To systematically review evidence on the feasibility and efficacy of real-time electronic notifications about patients at high risk of emergency department (ED) recidivism. Methods: Eight electronic databases were searched for empirical studies of real-time ED-based electronic tools, identifying adult patients at high risk of frequent utilization. Study selection and data extraction were performed independently by two reviewers. Qualitative data synthesis and assessment of strength of evidence were conducted through consensus discussion. Results: Of 2,256 records found through the search, 210 were duplicates, 2,004 were excluded based on abstract review, and 31 were excluded after full text review. The final sample consisted of 10 studies described in 11 articles describing the effect of real-time ED-based electronic notifications for high-risk patients. Three were randomized controlled trials (RCTs). All notifications were based on prespecified markers of risk. Seven studies integrated complex care plans into the electronic health record. Effect on ED use and length of stay (LOS) was mixed: nine studies reported decreased ED use, although results were statistically significant in only three studies; for LOS, one study reported a statistically significant reduction. Impact on cost and financial metrics was promising, with three (of three studies reporting this metric) showing improved organizational financial metrics. Three RCTs reported a reduction in opioid prescriptions. Conclusions: Real-time electronic notifications of ED providers regarding patients at high risk of ED recidivism are feasible. They may help reduce resource utilization and costs. Large knowledge gaps remain regarding patient- and provider-centered outcomes.
Collapse
Affiliation(s)
- Hannah J. Kimmel
- Center for Evidence Synthesis in Health, Brown University School of Public Health, Providence, Rhode Island
| | - Yanick N. Brice
- Center for Evidence Synthesis in Health, Brown University School of Public Health, Providence, Rhode Island
- Department of Health Services, Policy and Practice, Brown University School of Public Health, Providence, Rhode Island
| | - Thomas A. Trikalinos
- Center for Evidence Synthesis in Health, Brown University School of Public Health, Providence, Rhode Island
- Department of Health Services, Policy and Practice, Brown University School of Public Health, Providence, Rhode Island
| | - Indra Neil Sarkar
- Department of Health Services, Policy and Practice, Brown University School of Public Health, Providence, Rhode Island
- Center for Biomedical Informatics, Brown University, Providence, Rhode Island
| | - Megan L. Ranney
- Department of Health Services, Policy and Practice, Brown University School of Public Health, Providence, Rhode Island
- Emergency Digital Health Innovation Program, Department of Emergency Medicine, Warren Alpert Medical School, Brown University, Providence, Rhode Island
| |
Collapse
|
36
|
Colicchio TK, Cimino JJ, Del Fiol G. Unintended Consequences of Nationwide Electronic Health Record Adoption: Challenges and Opportunities in the Post-Meaningful Use Era. J Med Internet Res 2019; 21:e13313. [PMID: 31162125 PMCID: PMC6682280 DOI: 10.2196/13313] [Citation(s) in RCA: 71] [Impact Index Per Article: 14.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2019] [Revised: 04/09/2019] [Accepted: 04/26/2019] [Indexed: 12/19/2022] Open
Abstract
The US health system has recently achieved widespread adoption of electronic health record (EHR) systems, primarily driven by financial incentives provided by the Meaningful Use (MU) program. Although successful in promoting EHR adoption and use, the program, and other contributing factors, also produced important unintended consequences (UCs) with far-reaching implications for the US health system. Based on our own experiences from large health information technology (HIT) adoption projects and a collection of key studies in HIT evaluation, we discuss the most prominent UCs of MU: failed expectations, EHR market saturation, innovation vacuum, physician burnout, and data obfuscation. We identify challenges resulting from these UCs and provide recommendations for future research to empower the broader medical and informatics communities to realize the full potential of a now digitized health system. We believe that fixing these unanticipated effects will demand efforts from diverse players such as health care providers, administrators, HIT vendors, policy makers, informatics researchers, funding agencies, and outside developers; promotion of new business models; collaboration between academic medical centers and informatics research departments; and improved methods for evaluations of HIT.
Collapse
Affiliation(s)
- Tiago K Colicchio
- Informatics Institute, University of Alabama at Birmingham, Birmingham, AL, United States
| | - James J Cimino
- Informatics Institute, University of Alabama at Birmingham, Birmingham, AL, United States
| | - Guilherme Del Fiol
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, United States
| |
Collapse
|
37
|
Abstract
Laboratory tests are an integral part of the electronic health record (EHR). Providing clinical decision support (CDS) for the ordering, collection, reporting, viewing, and interpretation of laboratory testing is a fundamental function of the EHR. The implementation of a sustainable, effective laboratory CDS program requires a commitment to standardization and harmonization of the laboratory dictionaries that are the foundation of laboratory-based CDS. In this review, the authors provide an overview of the tools available within the EHR to improve decision making throughout the entire laboratory testing process, from test order to clinical action.
Collapse
Affiliation(s)
- Joseph W Rudolf
- Department of Laboratory Medicine and Pathology, University of Minnesota Medical School, 420 Delaware Street Southeast, MMC 609 Mayo, Minneapolis, MN 55455, USA
| | - Anand S Dighe
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, 55 Fruit Street, Boston, MA 02114-2696, USA.
| |
Collapse
|
38
|
Cimino JJ, Li Z, Weng C. An Exploration of the Terminology of Clinical Cognition and Reasoning. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2018; 2018:321-329. [PMID: 30815071 PMCID: PMC6371388] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
We hypothesize that the functionality of electronic health records could be improved with the addition of formal representations of clinicians' cognitive processes, including such things as the interpretation and synthesis of patient findings and the rational for diagnostic and therapeutic decisions. We carried out a four-phase analysis of clinical case studies to characterize how such processes are represented through relationships between clinical terms. The result is an terminology of 26 relationships that were validated against published clinical cases with 85.4% interrater reliability. We believe that capturing patient-specific information with these relationships can lead to improvements in clinical decision support systems, information retrieval and learning health systems.
Collapse
Affiliation(s)
- James J Cimino
- Informatics Institute, University of Alabama at Birmingham, Birmingham, Alabama
- Department of Biomedical Informatics, Columbia University, New York, New York
| | - Ziran Li
- Department of Biomedical Informatics, Columbia University, New York, New York
| | - Chunhua Weng
- Department of Biomedical Informatics, Columbia University, New York, New York
| |
Collapse
|
39
|
Reese TJ, Kawamoto K, Fiol GD, Drews F, Taft T, Kramer H, Weir C. When an Alert is Not an Alert: A Pilot Study to Characterize Behavior and Cognition Associated with Medication Alerts. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2018; 2018:1488-1497. [PMID: 30815194 PMCID: PMC6371356] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Introduction. Preventable adverse drug events are a significant patient-safety concern, yet most medication alerts are disregarded. Pharmacists encounter the highest number of medication alerts and likely have developed behaviors to cope with alerting inefficiencies. The study objective was to better understand alert override behavior relating to a motivational construct framework. Methods. Mixed-methods study of 10 pharmacists (567 verifications) with eye-tracking observations and retrospective think aloud interviews. Results. Pharmacists spent on average 14 seconds longer verifying orders with alerts than orders without alerts (p<0.001). Verification occurred before alerts were triggered, and no order changes occurred after alerts. Pharmacists reported 62% of alerts as unhelpful and 21% as frustrating. Alert interactions took on average 3.9 seconds. Discussion. Pharmacists anticipate alerts by making appropriate checks and changes before alert prompts. Medication alerts seem to be useful. However, the observed pharmacists' behavior suggests changes in the alert context are needed to match cognition.
Collapse
Affiliation(s)
- Thomas J Reese
- University of Utah, Department of Biomedical Informatics, Salt Lake City, Utah
| | - Kensaku Kawamoto
- University of Utah, Department of Biomedical Informatics, Salt Lake City, Utah
| | - Guilherme Del Fiol
- University of Utah, Department of Biomedical Informatics, Salt Lake City, Utah
| | - Frank Drews
- University of Utah, Department of Psychology, Salt Lake City, Utah
| | - Teresa Taft
- University of Utah, Department of Biomedical Informatics, Salt Lake City, Utah
| | - Heidi Kramer
- University of Utah, Department of Biomedical Informatics, Salt Lake City, Utah
| | - Charlene Weir
- University of Utah, Department of Biomedical Informatics, Salt Lake City, Utah
| |
Collapse
|
40
|
Reynolds C, Esrailian E, Hommes D. Quality Improvement in Gastroenterology: A Systematic Review of Practical Interventions for Clinicians. Dig Dis Sci 2018; 63:2507-2518. [PMID: 30014225 DOI: 10.1007/s10620-018-5198-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/26/2018] [Accepted: 07/05/2018] [Indexed: 01/06/2023]
Abstract
BACKGROUND Quality improvement (QI) identifies practical methods to improve patient care; however, it is not always widely known which QI methods are successful. We sought to create a primer of QI in gastroenterology for the practicing clinician. METHODS We performed a systematic review of QI literature in gastroenterology. We included search terms for inflammatory bowel disease, irritable bowel syndrome, celiac disease, gastroesophageal reflux disease, pancreatitis, liver disease, colorectal cancer screening, endoscopy, and gastrointestinal bleeding. We used general search terms for QI as well as specific terms to capture established quality metrics for each GI disease area. RESULTS We found 33 studies that met our definitions for QI. There were 17 studies of endoscopy including screening colonoscopy, six on liver disease, four on IBD, two on GERD, three on GI bleeding, and one on celiac disease. Education was the most common intervention, although most successful studies combined education with another intervention. Other effective interventions included retraining sessions to reach ADR goals in colonoscopy, nursing protocols to increase HCC screening, and EMR decision support tools to prompt reassessment of PPI therapy. Many studies showed improved compliance to metrics, but few were able to show differences in length of stay, readmissions, or mortality. CONCLUSIONS Our review of quality improvement literature in gastroenterology revealed common themes of successful programs: Education was frequently used but often insufficient, the EMR may be underutilized in guiding decision making, and patient-reported outcomes were infrequently assessed. Further research may be needed to compare QI strategies directly.
Collapse
Affiliation(s)
- Courtney Reynolds
- Division of Digestive Diseases, Department of Medicine, University of California Los Angeles, 10945 Le Conte Ave, Suite 2338, Los Angeles, CA, 90095, USA.
| | - Eric Esrailian
- Division of Digestive Diseases, Department of Medicine, University of California Los Angeles, 10945 Le Conte Ave, Suite 2338, Los Angeles, CA, 90095, USA.,Division of Digestive Diseases, Department of Medicine, University of California Los Angeles, 200 Medical Plaza Driveway, Los Angeles, CA, 90024, USA
| | - Daniel Hommes
- Division of Digestive Diseases, Department of Medicine, University of California Los Angeles, 10945 Le Conte Ave, Suite 2338, Los Angeles, CA, 90095, USA
| |
Collapse
|
41
|
Wide variation and patterns of physicians’ responses to drug–drug interaction alerts. Int J Qual Health Care 2018; 31:89-95. [DOI: 10.1093/intqhc/mzy102] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2017] [Revised: 03/19/2018] [Accepted: 04/19/2018] [Indexed: 01/04/2023] Open
|
42
|
Schreiber R, Sittig DF, Ash J, Wright A. Orders on file but no labs drawn: investigation of machine and human errors caused by an interface idiosyncrasy. J Am Med Inform Assoc 2018; 24:958-963. [PMID: 28339629 DOI: 10.1093/jamia/ocw188] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2016] [Accepted: 12/31/2016] [Indexed: 02/05/2023] Open
Abstract
In this report, we describe 2 instances in which expert use of an electronic health record (EHR) system interfaced to an external clinical laboratory information system led to unintended consequences wherein 2 patients failed to have laboratory tests drawn in a timely manner. In both events, user actions combined with the lack of an acknowledgment message describing the order cancellation from the external clinical system were the root causes. In 1 case, rapid, near-simultaneous order entry was the culprit; in the second, astute order management by a clinician, unaware of the lack of proper 2-way interface messaging from the external clinical system, led to the confusion. Although testing had shown that the laboratory system would cancel duplicate laboratory orders, it was thought that duplicate alerting in the new order entry system would prevent such events.
Collapse
Affiliation(s)
| | - Dean F Sittig
- University of Texas Health Science Center at Houston's School of Biomedical Informatics and the UT-Memorial Hermann Center for Healthcare Quality & Safety, Houston, TX, USA
| | - Joan Ash
- Department of Medical Informatics and Clinical Epidemiology, School of Medicine, Oregon Health & Science University, Portland, OR, USA
| | - Adam Wright
- Brigham & Women's Hospital, Boston, MA, USA; Harvard Medical School, Boston, MA, USA; Partners HealthCare, Boston, MA, USA
| |
Collapse
|