The use of artificial intelligence to detect voided medication orders in oral and maxillofacial surgery inpatients

OBJECTIVE

The aim of this study is to determine if supervised machine learning algorithms can accurately predict voided computerized physician order entry in oral and maxillofacial surgery inpatients.

METHODS

Data from Electronic Medical Record included patient demographics, comorbidities, procedures, vital signs, laboratory values, and medication orders were retrospectively collected. Predictor variables included patient demographics, comorbidities, procedures, vital signs, and laboratory values. Outcome of interest is if a medication order was voided or not. Data was cleaned and processed using Microsoft Excel and Python v3.12. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Naïve Bayes were trained, validated, and tested for accuracy of the prediction of voided medication orders.

RESULTS

37,493 medication orders from 1,204 patient admissions over 5 years were used for this study. 3,892 (10.4%) medication orders were voided. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Naïve Bayes had an Area Under the Receiver Operating Curve of 0.802 with 95% CI [0.787, 0.825], 0.746 with 95% CI [0.722, 0.765], 0.685 with 95% CI [0.667, 0.699], and 0.505 with 95% CI [0.489, 0.539], respectively. Area Under the Precision Recall Curve was 0.684 with 95% CI [0.679, 0.702], 0.647 with 95% CI [0.638, 0.664], 0.429 with 95% CI [0.417, 0.434], and 0.551 with 95% CI [0.551, 0.552], respectively.

CONCLUSION

Gradient Boosted Decision Trees was the best performing model of the supervised machine learning algorithms with satisfactory outcomes in the test cohort for predicting voided Computerized Physician Order Entry in Oral and Maxillofacial Surgery inpatients.

Development of an Austrian HL7 Member FHIR IG Supporting Medication Ordering, Prescription and Dispense Processes

The Austrian research project Linked Care explored digitalization in mobile care, focusing on streamlining the medication process to save nursing staff time. A FHIR R5-based workflow was developed to support medication ordering by nurses, prescriptions by practitioners, and dispensing by pharmacies. Key FHIR resources were profiled and published in an HL7 Austria Member Implementation Guide (IG). The IG includes specifications and technical details for implementation and was the first member-contributed IG approved by the HL7 Austria FHIR community in early 2024. These specifications are now being implemented and will be tested in late 2024.

Linked Care: A Harmonized Workflow for Medication Ordering, Prescription and Dispensing in Mobile Care Settings

The growing demand for care amid changing demographics poses significant challenges exacerbated by decreasing healthcare professional availability. In Austria, the Linked Care project aims to address these challenges by developing an intersectoral, harmonized IT-supported workflow for medication ordering, prescription, and dispense in mobile care settings. A human centered design approach, with user-focused interviews and workshops was used to identify requirements and analyze the workflows. Activity diagrams were used represent workflows. The resulting harmonized workflow, developed through iterative collaboration with care organizations, integrates the LC platform into existing care software. To test and demonstrate the harmonized workflow, mockups were created and evaluated for usability, resulting in positive feedback and suggestions for enhancements. Current workflows revealed media breaches and inefficiencies, which the proposed harmonized workflow seeks to address. The paper concludes with implications for future developments, including the subsequent adoption of a HL7 FHIR Implementation Guide for Austria, based on the defined harmonized workflow, to streamline intersectoral communication and improve efficiency in mobile care settings.

A Real-Time Dashboard Approach to Decision Support: An Exercise in Co-Design

Laboratory tests play an integral role in the delivery of quality health care. However, evidence indicates variations in diagnostic testing, which can lead to patient safety risks. Electronic decision support systems are often identified as key to reducing diagnostic error. However, such tools are often introduced into a clinical setting with little understanding of clinician workflow and how tools are likely to impact this. This study reports a qualitative co-design methodology and results from the first phase in the design and development of an analytics-driven, dashboard approach to supporting clinician test ordering in the Emergency Department.

A Decision Support Alert for Promoting Prophylactic Laxative Use with Clozapine

We investigated the effectiveness of a decision support alert to improve prophylactic laxative use with clozapine in hospital. Prescribing data for first clozapine and laxative prescriptions were extracted and linked. Proportions of first clozapine prescriptions for which a laxative was co-prescribed within 24 hours was compared before and after alert implementation. The alert was associated with increased and earlier laxative co-prescribing.

Applying Machine Learning for Prescriptive Support: A Use Case with Unfractionated Heparin in Intensive Care Units

Continuous unfractionated heparin is widely used in intensive care, yet its complex pharmacokinetic properties complicate the determination of appropriate doses. To address this challenge, we developed machine learning models to predict over- and under-dosing, based on anti-Xa results, using a monocentric retrospective dataset. The random forest model achieved a mean AUROC of 0.80 [0.77-0.83], while the XGB model reached a mean AUROC of 0.80 [0.76-0.83]. Feature importance was employed to enhance the interpretability of the model, a critical factor for clinician acceptance. After prospective validation, machine learning models such as those developed in this study could be implemented within a computerized physician order entry (CPOE) as a clinical decision support system (CDSS).

In adult inpatients with pneumonia, CPOE-based vs. routine stewardship reduced extended-spectrum antibiotic use

SOURCE CITATION

Gohil SK, Septimus E, Kleinman K, et al. Stewardship prompts to improve antibiotic selection for pneumonia: the INSPIRE randomized clinical trial. JAMA. 2024;331:2007-2017. 38639729.

Electronic Alert Systems for Patients With Acute Kidney Injury: A Systematic Review and Meta-Analysis

Importance

The acute kidney injury (AKI) electronic alert (e-alert) system was hypothesized to improve the outcomes of AKI. However, its association with different patient outcomes and clinical practice patterns remains systematically unexplored.

Objective

To assess the association of AKI e-alerts with patient outcomes (mortality, AKI progression, dialysis, and kidney recovery) and clinical practice patterns.

Data Sources

A search of Embase and PubMed on March 18, 2024, and a search of the Cochrane Library on March 20, 2024, to identify all relevant studies. There were no limitations on language or article types.

Study Selection

Studies evaluating the specified outcomes in adult patients with AKI comparing AKI e-alerts with standard care or no e-alerts were included. Studies were excluded if they were duplicate cohorts, had insufficient outcome data, or had no control group.

Data Extraction and Synthesis

Two investigators independently extracted data and assessed bias. The systematic review and meta-analysis followed the PRISMA guidelines. Random-effects model meta-analysis, with predefined subgroup analysis and trial sequential analyses, were conducted.

Main Outcomes and Measures

Primary outcomes included mortality, AKI progression, dialysis, and kidney recovery. Secondary outcomes were nephrologist consultations, post-AKI exposure to nonsteroidal anti-inflammatory drugs (NSAID), post-AKI angiotensin-converting enzyme inhibitor and/or angiotensin receptor blocker (ACEI/ARB) prescription, hospital length of stay, costs, and AKI documentation.

Results

Thirteen unique studies with 41 837 unique patients were included (mean age range, 60.5-79.0 years]; 29.3%-48.5% female). The risk ratios (RRs) for the AKI e-alerts group compared with standard care were 0.96 for mortality (95% CI, 0.89-1.03), 0.91 for AKI stage progression (95% CI, 0.84-0.99), 1.16 for dialysis (95% CI, 1.05-1.28), and 1.13 for kidney recovery (95% CI, 0.86-1.49). The AKI e-alerts group had RRs of 1.45 (95% CI, 1.04-2.02) for nephrologist consultation, 0.75 (95% CI, 0.59-0.95) for post-AKI NSAID exposure. The pooled RR for post-AKI ACEI/ARB exposure in the AKI e-alerts group compared with the control group was 0.91 (95% CI, 0.78-1.06) and 1.28 (95% CI, 1.04-1.58) for AKI documentation. Use of AKI e-alerts was not associated with lower hospital length of stay (mean difference, -0.09 [95% CI, -0.47 to 0.30] days) or lower cost (mean difference, US $655.26 [95% CI, -$656.98 to $1967.5]) but was associated with greater AKI documentation (RR, 1.28 [95% CI, 1.04-1.58]). Trial sequential analysis confirmed true-positive results of AKI e-alerts on increased nephrologist consultations and reduced post-AKI NSAID exposure and its lack of association with mortality.

Conclusions and Relevance

In this systematic review and meta-analysis, AKI e-alerts were not associated with a lower risk for mortality but were associated with changes in clinical practices. They were associated with lower risk for AKI progression. Further research is needed to confirm these results and integrate early AKI markers or prediction models to improve outcomes.

Poor performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction

PURPOSE

Clinical decision support systems (CDSS) are used to identify drugs with potential need for dose modification in patients with renal impairment. ChatGPT holds the potential to be integrated in the electronic health record (EHR) system to give such dosing advices. In this study, we aim to evaluate the performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal impairment.

METHODS

This cross-sectional study was performed at Tergooi Medical Center, the Netherlands. CDSS alerts regarding renal dysfunction were collected from the electronic health record (EHR) during a 2-week period and were presented to ChatGPT and an expert panel. Alerts were presented with and without patient variables. To evaluate the performance, suggested medication interventions were compared.

RESULTS

In total, 172 CDDS alerts were generated for 80 patients. Indecisive responses by ChatGPT to alerts were excluded. For alerts presented without patient variables, ChatGPT provided "correct and identical" responses to 19.9%, "correct and different" responses to 26.7%, and "incorrect responses to 53.4% of the alerts. For alerts including patient variables, ChatGPT provided "correct and identical" responses to 16.7%, "correct and different" responses to 16.0%, and "incorrect responses to 67.3% of the alerts. Accuracy was better for newer drugs such as direct oral anticoagulants.

CONCLUSION

The performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction was poor. Based on these results, we conclude that ChatGPT, in its current state, is not appropriate for automatic integration into our EHR to handle CDSS alerts related to renal dysfunction.

Clinical decision support to improve CBC and differential ordering

OBJECTIVES

Complete blood count and differential (CBC diff) is a common laboratory test that may be overused or misordered, particularly in an inpatient setting. We assessed the ability of a clinical decision support (CDS) alert to decrease unnecessary orders for CBC diff and analyzed its impact in the laboratory.

METHODS

We designed 3 CDS alerts to provide guidance to providers ordering CBC diff on inpatients at frequencies of daily, greater than once daily, or as needed.

RESULTS

The 3 alerts were highly effective in reducing orders for CBC diff at the frequencies targeted by the alert. Overall, test volume for CBC diff decreased by 32% (mean of 5257 tests per month) after implementation of the alerts, with a corresponding decrease of 22% in manual differentials performed (mean of 898 per month). Turnaround time for manual differentials decreased by a mean of 41.5 minutes, with a mean decrease of up to 90 minutes during peak morning hours.

CONCLUSIONS

The 3 CDS alerts successfully decreased inpatient orders for CBC diff and improved the quality of patient care by decreasing turnaround time for manual differentials.

In adult inpatients with a UTI, CPOE-based vs. routine stewardship reduced extended-spectrum antibiotic use

SOURCE CITATION

Gohil SK, Septimus E, Kleinman K, et al. Stewardship prompts to improve antibiotic selection for urinary tract infection: the INSPIRE randomized clinical trial. JAMA. 2024;331:2018-2028. 38639723.

Quantifying and Reducing Errors in Vascular Imaging Examination Orders Through a Multistage Quality Improvement Intervention

PURPOSE

The aims of this study were to quantify order error rates for vascular imaging examinations and to assess the effects of a multistage quality improvement intervention on those rates.

METHODS

In this prospective, institutional review board-exempt project at a large academic quaternary care hospital, the authors aimed to quantify and reduce the order error rate by 50%. The authors analyzed 844 orders for all vascular imaging examinations placed before the intervention (July 19 to August 1, 2021, and September 13 to September 26, 2021), after an intervention in the cardiac surgery department consisting of a new customized order option in the electronic health record for routine preoperative patients (postintervention 1, February 28 to March 27, 2022); and after an educational and feedback campaign (postintervention 2, May 23 to June 5, 2022). Incorrect orders were identified by a radiology trainee during protocoling if the reasons for ordered examination and imaging examination were discordant and subsequently confirmed with the ordering provider. The primary outcome, order error rate, was compared across the project periods using the χ2 test and by ordering department using the χ2 and Fisher exact tests.

RESULTS

The preintervention order error rate of 16% (50 of 306) decreased by 83% to 3% (10 of 353) at postintervention 1 (P < .001) and was durable at 3% (6 of 185) by project end. Chest CT with or without contrast constituted the majority of incorrect orders (44%, 22 of 50); "Pre-Op" was the most common examination reason (32% [16 of 50]). Cardiac surgery orderers were responsible for the most incorrect orders (32% [16 of 50]). All four most common ordering departments, including cardiac surgery, reduced their order error rates after the intervention (P < .001).

CONCLUSIONS

Incorrect orders for imaging examinations can be reduced through targeted quality improvement interventions combining tailored electronic health record order options with education and feedback on practice habits.

Navigating Alert Fatigue: A Case Study in Electronic Health Record Alert Design Optimization

Clinical decision support (CDS) systems play a crucial role in enhancing patient outcomes, but inadequate design contributes to alert fatigue, inundating clinicians with disruptive alerts that lack clinical relevance. This case study delves into a quality improvement (QI) project addressing nursing electronic health record (EHR) alert fatigue by strategically redesigning four high-firing/low action alerts. Employing a mixed-methods approach, including quantitative analysis, empathy mapping sessions, and user feedback, the project sought to understand and alleviate the challenges posed by these alerts. Virtual empathy mapping sessions with clinical nurses provided valuable insights into user experiences. Qualitative findings, CDS design principles, and organizational practice expectations informed the redesign process, resulting in the removal of all four identified disruptive alerts and redesign of passive alerts. This initiative released 877 unactionable disruptive nursing hours, emphasizing the significance of proper alert design and the necessity for organizational structures ensuring sustained governance in healthcare system optimization.

Observations from Optimizing an Electronic Order Set for Withdrawal of Life-Sustaining Treatment

Background: Withdrawal of life-sustaining treatment (WLST) is a process with unique pressure for all involved. The use of an electronic order set can facilitate best care. Objective: To assess utilization of a WLST order set and time to inpatient death before and after optimization. Design: A retrospective chart review for 12-month periods before and after enhancements to a WLST order set. Setting/Subjects: Multicenter study within an American, not-for-profit health care system of inpatient decedents July 2017-June 2018 and April 2021-March 2022 with orders placed via WLST order set. Measurements: Co-primary outcomes included order set utilization and time from activation of orders to patient death. Descriptive post hoc analyses featured demographics, palliative consultation, ordering clinician type/specialty, and COVID-19. Results: A total of 1949 patients had orders placed via the WLST order set and died in-hospital. Compared with the 2017-2018 period, use increased 35.8% in 2021-2022. Time to death after release of orders was significantly longer for the 2021-2022 group (4.4 vs. 3.7 hours). Demographic details included nurse practitioners (39%) as most frequent WLST order set utilizer and palliative consultation in 46% of terminal hospitalizations. Among decedents with consultation, palliative clinicians were the WLST order set utilizer for 47% of cases (i.e., 21% of all WLST order set utilizations). The median time to death was significantly longer when orders were placed by a palliative clinician (4.5 hours) compared with nonpalliative specialists (3.9 hours). COVID-19 was a hospital diagnosis for 29% of decedents in the 2021-2022 group. Conclusions: In the emotionally and cognitively intense process that is WLST, an order set provides a modifiable panel of defaults. Our experience highlights the power in guiding primary palliative care for WLST in the hospital setting and suggests that advanced practice providers and nonpalliative clinicians, as primary utilizers, be integral in the design of a WLST order set.

Development and validation of a clinical decision support system to prevent anticoagulant duplications

BACKGROUND AND OBJECTIVE

Unintended duplicate prescriptions of anticoagulants increase the risk of serious adverse events. Clinical Decision Support Systems (CDSSs) can help prevent such medication errors; however, sophisticated algorithms are needed to avoid alert fatigue. This article describes the steps taken in our hospital to develop a CDSS to prevent anticoagulant duplication (AD).

METHODS

The project was composed of three phases. In phase I, the status quo was established. In phase II, a clinical pharmacist developed an algorithm to detect ADs using daily data exports. In phase III, the algorithm was integrated into the hospital's electronic health record system. Alerts were reviewed by clinical pharmacists before being sent to the prescribing physician. We conducted a retrospective analysis of all three phases to assess the impact of the interventions on the occurrence and duration of ADs. Phase III was analyzed in more detail regarding the acceptance rate, sensitivity, and specificity of the alerts.

RESULTS

We identified 91 ADs in 1581 patients receiving two or more anticoagulants during phase I, 70 ADs in 1692 patients in phase II, and 57 ADs in 1575 patients in phase III. Mean durations of ADs were 1.8, 1.4, and 1.1 calendar days during phases I, II, and III, respectively. In comparison to the baseline in phase I, the relative risk reduction of AD in patients treated with at least two different anticoagulants during phase III was 42% (RR: 0.58, CI: 0.42-0.81). A total of 429 alerts were generated during phase III, many of which were self-limiting, and 186 alerts were sent to the respective prescribing physician. The acceptance rate was high at 97%. We calculated a sensitivity of 87.4% and a specificity of 87.9%.

CONCLUSION

The stepwise development of a CDSS for the detection of AD markedly reduced the frequency and duration of medication errors in our hospital, thereby improving patient safety.

Stewardship Prompts to Improve Antibiotic Selection for Pneumonia: The INSPIRE Randomized Clinical Trial

Importance

Pneumonia is the most common infection requiring hospitalization and is a major reason for overuse of extended-spectrum antibiotics. Despite low risk of multidrug-resistant organism (MDRO) infection, clinical uncertainty often drives initial antibiotic selection. Strategies to limit empiric antibiotic overuse for patients with pneumonia are needed.

Objective

To evaluate whether computerized provider order entry (CPOE) prompts providing patient- and pathogen-specific MDRO infection risk estimates could reduce empiric extended-spectrum antibiotics for non-critically ill patients admitted with pneumonia.

Design, Setting, and Participants

Cluster-randomized trial in 59 US community hospitals comparing the effect of a CPOE stewardship bundle (education, feedback, and real-time MDRO risk-based CPOE prompts; n = 29 hospitals) vs routine stewardship (n = 30 hospitals) on antibiotic selection during the first 3 hospital days (empiric period) in non-critically ill adults (≥18 years) hospitalized with pneumonia. There was an 18-month baseline period from April 1, 2017, to September 30, 2018, and a 15-month intervention period from April 1, 2019, to June 30, 2020.

Intervention

CPOE prompts recommending standard-spectrum antibiotics in patients ordered to receive extended-spectrum antibiotics during the empiric period who have low estimated absolute risk (<10%) of MDRO pneumonia, coupled with feedback and education.

Main Outcomes and Measures

The primary outcome was empiric (first 3 days of hospitalization) extended-spectrum antibiotic days of therapy. Secondary outcomes included empiric vancomycin and antipseudomonal days of therapy and safety outcomes included days to intensive care unit (ICU) transfer and hospital length of stay. Outcomes compared differences between baseline and intervention periods across strategies.

Results

Among 59 hospitals with 96 451 (51 671 in the baseline period and 44 780 in the intervention period) adult patients admitted with pneumonia, the mean (SD) age of patients was 68.1 (17.0) years, 48.1% were men, and the median (IQR) Elixhauser comorbidity count was 4 (2-6). Compared with routine stewardship, the group using CPOE prompts had a 28.4% reduction in empiric extended-spectrum days of therapy (rate ratio, 0.72 [95% CI, 0.66-0.78]; P < .001). Safety outcomes of mean days to ICU transfer (6.5 vs 7.1 days) and hospital length of stay (6.8 vs 7.1 days) did not differ significantly between the routine and CPOE intervention groups.

Conclusions and Relevance

Empiric extended-spectrum antibiotic use was significantly lower among adults admitted with pneumonia to non-ICU settings in hospitals using education, feedback, and CPOE prompts recommending standard-spectrum antibiotics for patients at low risk of MDRO infection, compared with routine stewardship practices. Hospital length of stay and days to ICU transfer were unchanged.

Trial Registration

ClinicalTrials.gov Identifier: NCT03697070.

Stewardship Prompts to Improve Antibiotic Selection for Urinary Tract Infection: The INSPIRE Randomized Clinical Trial

Importance

Urinary tract infection (UTI) is the second most common infection leading to hospitalization and is often associated with gram-negative multidrug-resistant organisms (MDROs). Clinicians overuse extended-spectrum antibiotics although most patients are at low risk for MDRO infection. Safe strategies to limit overuse of empiric antibiotics are needed.

Objective

To evaluate whether computerized provider order entry (CPOE) prompts providing patient- and pathogen-specific MDRO risk estimates could reduce use of empiric extended-spectrum antibiotics for treatment of UTI.

Design, Setting, and Participants

Cluster-randomized trial in 59 US community hospitals comparing the effect of a CPOE stewardship bundle (education, feedback, and real-time and risk-based CPOE prompts; 29 hospitals) vs routine stewardship (n = 30 hospitals) on antibiotic selection during the first 3 hospital days (empiric period) in noncritically ill adults (≥18 years) hospitalized with UTI with an 18-month baseline (April 1, 2017-September 30, 2018) and 15-month intervention period (April 1, 2019-June 30, 2020).

Interventions

CPOE prompts recommending empiric standard-spectrum antibiotics in patients ordered to receive extended-spectrum antibiotics who have low estimated absolute risk (<10%) of MDRO UTI, coupled with feedback and education.

Main Outcomes and Measures

The primary outcome was empiric (first 3 days of hospitalization) extended-spectrum antibiotic days of therapy. Secondary outcomes included empiric vancomycin and antipseudomonal days of therapy. Safety outcomes included days to intensive care unit (ICU) transfer and hospital length of stay. Outcomes were assessed using generalized linear mixed-effect models to assess differences between the baseline and intervention periods.

Results

Among 127 403 adult patients (71 991 baseline and 55 412 intervention period) admitted with UTI in 59 hospitals, the mean (SD) age was 69.4 (17.9) years, 30.5% were male, and the median Elixhauser Comorbidity Index count was 4 (IQR, 2-5). Compared with routine stewardship, the group using CPOE prompts had a 17.4% (95% CI, 11.2%-23.2%) reduction in empiric extended-spectrum days of therapy (rate ratio, 0.83 [95% CI, 0.77-0.89]; P < .001). The safety outcomes of mean days to ICU transfer (6.6 vs 7.0 days) and hospital length of stay (6.3 vs 6.5 days) did not differ significantly between the routine and intervention groups, respectively.

Conclusions and Relevance

Compared with routine stewardship, CPOE prompts providing real-time recommendations for standard-spectrum antibiotics for patients with low MDRO risk coupled with feedback and education significantly reduced empiric extended-spectrum antibiotic use among noncritically ill adults admitted with UTI without changing hospital length of stay or days to ICU transfers.

Trial Registration

ClinicalTrials.gov Identifier: NCT03697096.

Evaluation of appropriateness of alerts overrides and physicians' responses of the medication-related clinical decision support system in China, a hospital-based study

This study was designed to investigate the state quo of the appropriateness of alerts overrides of the medication-related clinical decision support system (MRCDSS) in China. The medication-related alerts in one hospital from Jan 2022 to Dec 2022 were acquired and sampled. Rates of alert overrides, appropriateness of alert generation and physicians' responses were observed. Total 14,612 medication-related alerts (≤ level 3) were recorded, of those, 12,659 (86.6%) alerts were overridden. The top 3 alert types were: drug and diagnosis contraindications (23.8%), drug and test value contraindications (23.3%), and compatibility issues (17.7%). Of all sampled 1,501 alerts, 80.2% of them were appropriately overridden by the physicians. The appropriate rate of alert generation was 57.9% and the inappropriate rate was 42.1%. The inappropriate rate of physicians' responses was 17.8%, and 2.0% physicians' responses were undetermined. A few medications accounted for over 10% of overrides, 88.3% of "overridden reasons" inputted by the physicians were meaningless characters or values, indicating an obvious "alert fatigue" in these physicians. Our results indicated that the overridden rate of MRCDSS in China was still high, and appropriateness of generation of alert was quite low. These data indicated that the MRCDSS currently using in China still needs constantly optimization and timely maintenance. Proper sensitivity to reduce triggering of useless alerts and generation of alert fatigue might play a vital role. We believed that these findings are helpful for better understanding the state quo of MRCDSS in China and providing useful insights for future developing and improving MRCDSS.

Optimising interruptive clinical decision support alerts for antithrombotic duplicate prescribing in hospital

INTRODUCTION

Duplicate prescribing clinical decision support alerts can prevent important prescribing errors but are frequently the cause of much alert fatigue. Stat dose prescriptions are a known reason for overriding these alerts. This study aimed to evaluate the effect of excluding stat dose prescriptions from duplicate prescribing alerts for antithrombotic medicines on alert burden, prescriber adherence, and prescribing.

MATERIALS AND METHODS

A before (January 1st, 2017 to August 31st, 2022) and after (October 5th, 2022 to September 30th, 2023) study was undertaken of antithrombotic duplicate prescribing alerts and prescribing following a change in alert settings. Alert and prescribing data for antithrombotic medicines were joined, processed, and analysed to compare alert rates, adherence, and prescribing. Alert burden was assessed as alerts per 100 prescriptions. Adherence was measured at the point of the alert as whether the prescriber accepted the alert and following the alert as whether a relevant prescription was ceased within an hour. Co-prescribing of antithrombotic stat dose prescriptions was assessed pre- and post-alert reconfiguration.

RESULTS

Reconfiguration of the alerts reduced the alert rate by 29 % (p < 0.001). The proportion of alerts associated with cessation of antithrombotic duplication significantly increased (32.8 % to 44.5 %, p < 0.001). Adherence at the point of the alert increased 1.2 % (4.8 % to 6.0 %, p = 0.012) and 11.5 % (29.4 % to 40.9 %, p < 0.001) within one hour of the alert. When ceased after the alert over 80 % of duplicate prescriptions were ceased within 2 min of overriding. Antithrombotic stat dose co-prescribing was unchanged for 4 out of 5 antithrombotic duplication alert rules.

CONCLUSION

By reconfiguring our antithrombotic duplicate prescribing alerts, we reduced alert burden and increased alert adherence. Many prescribers ceased duplicate prescribing within 2 min of alert override highlighting the importance of incorporating post-alert measures in accurately determining prescriber alert adherence.

Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support

OBJECTIVES

To evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts.

MATERIALS AND METHODS

We extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness.

RESULTS

Five CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4 ± 1.2, human: 2.7 ± 1.2, P = .001).

CONCLUSION

End-user comments provide clinicians' immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.

The use of artificial intelligence to optimize medication alerts generated by clinical decision support systems: a scoping review

OBJECTIVE

Current Clinical Decision Support Systems (CDSSs) generate medication alerts that are of limited clinical value, causing alert fatigue. Artificial Intelligence (AI)-based methods may help in optimizing medication alerts. Therefore, we conducted a scoping review on the current state of the use of AI to optimize medication alerts in a hospital setting. Specifically, we aimed to identify the applied AI methods used together with their performance measures and main outcome measures.

MATERIALS AND METHODS

We searched Medline, Embase, and Cochrane Library database on May 25, 2023 for studies of any quantitative design, in which the use of AI-based methods was investigated to optimize medication alerts generated by CDSSs in a hospital setting. The screening process was supported by ASReview software.

RESULTS

Out of 5625 citations screened for eligibility, 10 studies were included. Three studies (30%) reported on both statistical performance and clinical outcomes. The most often reported performance measure was positive predictive value ranging from 9% to 100%. Regarding main outcome measures, alerts optimized using AI-based methods resulted in a decreased alert burden, increased identification of inappropriate or atypical prescriptions, and enabled prediction of user responses. In only 2 studies the AI-based alerts were implemented in hospital practice, and none of the studies conducted external validation.

DISCUSSION AND CONCLUSION

AI-based methods can be used to optimize medication alerts in a hospital setting. However, reporting on models' development and validation should be improved, and external validation and implementation in hospital practice should be encouraged.

Adoption by clinicians of electronic order communications in NHS secondary care: a descriptive account

BACKGROUND

Due to the rapid advancement in information technology, changes to communication modalities are increasingly implemented in healthcare. One such modality is Computerised Provider Order Entry (CPOE) systems which replace paper, verbal or telephone orders with electronic booking of requests. We aimed to understand the uptake, and user acceptability, of CPOE in a large National Health Service hospital system.

METHODS

This retrospective single-centre study investigates the longitudinal uptake of communications through the Prescribing, Information and Communication System (PICS). The development and configuration of PICS are led by the doctors, nurses and allied health professionals that use it and requests for CPOE driven by clinical need have been described.Records of every request (imaging, specialty review, procedure, laboratory) made through PICS were collected between October 2008 and July 2019 and resulting counts were presented. An estimate of the proportion of completed requests made through the system has been provided for three example requests. User surveys were completed.

RESULTS

In the first 6 months of implementation, a total of 832 new request types (imaging types and specialty referrals) were added to the system. Subsequently, an average of 6.6 new request types were added monthly. In total, 8 035 132 orders were requested through PICS. In three example request types (imaging, endoscopy and full blood count), increases in the proportion of requests being made via PICS were seen. User feedback at 6 months reported improved communications using the electronic system.

CONCLUSION

CPOE was popular, rapidly adopted and diversified across specialties encompassing wide-ranging requests.

Interprofessional Evaluation of a Medication Clinical Decision Support System Prior to Implementation

BACKGROUND

 Computerized physician order entry (CPOE) and clinical decision support systems (CDSS) are widespread due to increasing digitalization of hospitals. They can be associated with reduced medication errors and improved patient safety, but also with well-known risks (e.g., overalerting, nonadoption).

OBJECTIVES

 Therefore, we aimed to evaluate a commonly used CDSS containing Medication-Safety-Validators (e.g., drug-drug interactions), which can be locally activated or deactivated, to identify limitations and thereby potentially optimize the use of the CDSS in clinical routine.

METHODS

 Within the implementation process of Meona (commercial CPOE/CDSS) at a German University hospital, we conducted an interprofessional evaluation of the CDSS and its included Medication-Safety-Validators following a defined algorithm: (1) general evaluation, (2) systematic technical and content-related validation, (3) decision of activation or deactivation, and possibly (4) choosing the activation mode (interruptive or passive). We completed the in-depth evaluation for exemplarily chosen Medication-Safety-Validators. Moreover, we performed a survey among 12 German University hospitals using Meona to compare their configurations.

RESULTS

 Based on the evaluation, we deactivated 3 of 10 Medication-Safety-Validators due to technical or content-related limitations. For the seven activated Medication-Safety-Validators, we chose the interruptive option ["PUSH-(&PULL)-modus"] four times (4/7), and a new, on-demand option ["only-PULL-modus"] three times (3/7). The site-specific configuration (activation or deactivation) differed across all participating hospitals in the survey and led to varying medication safety alerts for identical patient cases.

CONCLUSION

 An interprofessional evaluation of CPOE and CDSS prior to implementation in clinical routine is crucial to detect limitations. This can contribute to a sustainable utilization and thereby possibly increase medication safety.

[Inability of hospital computerised physician order entry systems to secure the use of concentrated potassium intravenous solutions]

OBJECTIVES

To determine whether hospital computerised physician order entry (CPOE) systems contribute to securing intravenous potassium chloride (KCl) prescriptions with reference to the recommendations issued by French healthcare agencies.

METHODS

We sent a questionnaire to the members of the Association pour le Digital et l'Information en Pharmacie.

RESULTS

More than three quarters of the 84 responses received involving 23 CPOE systems indicate that it is possible to: prescribe an ampoule of concentrated potassium chloride 10% 10mL intravenously without any diluents (80%); prescribe 4g of KCl in a bag of 500mL of NaCl 0,9% (98%); prescribe a solution that contains 6 grams of KCl per liter (94%); prescribe the administration of an injectable ampoule orally by means of a free text comment (83%). Nearly half of the responses indicate that it is possible to prescribe: concentrated KCl ampoules as administration solvent (50%); an injectable vial to be administered by oral route (52%).

CONCLUSION

At least 23 hospital CPOE systems are unable to secure the prescriptions of injectable KCl. This finding lifts the veil on an unthought, namely the role of CPOE systems in securing high-risk medications. In order to solve this problem, it should be mandatory that health information technology vendors pay particular attention to these drugs. With regard to injectable KCl, the utilisation of a dilution vehicle, maximum concentration and maximum infusion flow rate are the first four constraints to be satisfied.

Randomized Comparison of Electronic Health Record Alert Types in Eliciting Responses about Prognosis in Gynecologic Oncology Patients

OBJECTIVES

 To compare the ability of different electronic health record alert types to elicit responses from users caring for cancer patients benefiting from goals of care (GOC) conversations.

METHODS

 A validated question asking if the user would be surprised by the patient's 6-month mortality was built as an Epic BestPractice Advisory (BPA) alert in three versions-(1) Required on Open chart (pop-up BPA), (2) Required on Close chart (navigator BPA), and (3) Optional Persistent (Storyboard BPA)-randomized using patient medical record number. Meaningful responses were defined as "Yes" or "No," rather than deferral. Data were extracted over 6 months.

RESULTS

 Alerts appeared for 685 patients during 1,786 outpatient encounters. Measuring encounters where a meaningful response was elicited, rates were highest for Required on Open (94.8% of encounters), compared with Required on Close (90.1%) and Optional Persistent (19.7%) (p < 0.001). Measuring individual alerts to which responses were given, they were most likely meaningful with Optional Persistent (98.3% of responses) and least likely with Required on Open (68.0%) (p < 0.001). Responses of "No," suggesting poor prognosis and prompting GOC, were more likely with Optional Persistent (13.6%) and Required on Open (10.3%) than with Required on Close (7.0%) (p = 0.028).

CONCLUSION

 Required alerts had response rates almost five times higher than optional alerts. Timing of alerts affects rates of meaningful responses and possibly the response itself. The alert with the most meaningful responses was also associated with the most interruptions and deferral responses. Considering tradeoffs in these metrics is important in designing clinical decision support to maximize success.