Development and validation of a clinical decision support system to prevent anticoagulant duplications

BACKGROUND AND OBJECTIVE

Unintended duplicate prescriptions of anticoagulants increase the risk of serious adverse events. Clinical Decision Support Systems (CDSSs) can help prevent such medication errors; however, sophisticated algorithms are needed to avoid alert fatigue. This article describes the steps taken in our hospital to develop a CDSS to prevent anticoagulant duplication (AD).

METHODS

The project was composed of three phases. In phase I, the status quo was established. In phase II, a clinical pharmacist developed an algorithm to detect ADs using daily data exports. In phase III, the algorithm was integrated into the hospital's electronic health record system. Alerts were reviewed by clinical pharmacists before being sent to the prescribing physician. We conducted a retrospective analysis of all three phases to assess the impact of the interventions on the occurrence and duration of ADs. Phase III was analyzed in more detail regarding the acceptance rate, sensitivity, and specificity of the alerts.

RESULTS

We identified 91 ADs in 1581 patients receiving two or more anticoagulants during phase I, 70 ADs in 1692 patients in phase II, and 57 ADs in 1575 patients in phase III. Mean durations of ADs were 1.8, 1.4, and 1.1 calendar days during phases I, II, and III, respectively. In comparison to the baseline in phase I, the relative risk reduction of AD in patients treated with at least two different anticoagulants during phase III was 42% (RR: 0.58, CI: 0.42-0.81). A total of 429 alerts were generated during phase III, many of which were self-limiting, and 186 alerts were sent to the respective prescribing physician. The acceptance rate was high at 97%. We calculated a sensitivity of 87.4% and a specificity of 87.9%.

CONCLUSION

The stepwise development of a CDSS for the detection of AD markedly reduced the frequency and duration of medication errors in our hospital, thereby improving patient safety.

Stewardship Prompts to Improve Antibiotic Selection for Pneumonia: The INSPIRE Randomized Clinical Trial

Importance

Pneumonia is the most common infection requiring hospitalization and is a major reason for overuse of extended-spectrum antibiotics. Despite low risk of multidrug-resistant organism (MDRO) infection, clinical uncertainty often drives initial antibiotic selection. Strategies to limit empiric antibiotic overuse for patients with pneumonia are needed.

Objective

To evaluate whether computerized provider order entry (CPOE) prompts providing patient- and pathogen-specific MDRO infection risk estimates could reduce empiric extended-spectrum antibiotics for non-critically ill patients admitted with pneumonia.

Design, Setting, and Participants

Cluster-randomized trial in 59 US community hospitals comparing the effect of a CPOE stewardship bundle (education, feedback, and real-time MDRO risk-based CPOE prompts; n = 29 hospitals) vs routine stewardship (n = 30 hospitals) on antibiotic selection during the first 3 hospital days (empiric period) in non-critically ill adults (≥18 years) hospitalized with pneumonia. There was an 18-month baseline period from April 1, 2017, to September 30, 2018, and a 15-month intervention period from April 1, 2019, to June 30, 2020.

Intervention

CPOE prompts recommending standard-spectrum antibiotics in patients ordered to receive extended-spectrum antibiotics during the empiric period who have low estimated absolute risk (<10%) of MDRO pneumonia, coupled with feedback and education.

Main Outcomes and Measures

The primary outcome was empiric (first 3 days of hospitalization) extended-spectrum antibiotic days of therapy. Secondary outcomes included empiric vancomycin and antipseudomonal days of therapy and safety outcomes included days to intensive care unit (ICU) transfer and hospital length of stay. Outcomes compared differences between baseline and intervention periods across strategies.

Results

Among 59 hospitals with 96 451 (51 671 in the baseline period and 44 780 in the intervention period) adult patients admitted with pneumonia, the mean (SD) age of patients was 68.1 (17.0) years, 48.1% were men, and the median (IQR) Elixhauser comorbidity count was 4 (2-6). Compared with routine stewardship, the group using CPOE prompts had a 28.4% reduction in empiric extended-spectrum days of therapy (rate ratio, 0.72 [95% CI, 0.66-0.78]; P < .001). Safety outcomes of mean days to ICU transfer (6.5 vs 7.1 days) and hospital length of stay (6.8 vs 7.1 days) did not differ significantly between the routine and CPOE intervention groups.

Conclusions and Relevance

Empiric extended-spectrum antibiotic use was significantly lower among adults admitted with pneumonia to non-ICU settings in hospitals using education, feedback, and CPOE prompts recommending standard-spectrum antibiotics for patients at low risk of MDRO infection, compared with routine stewardship practices. Hospital length of stay and days to ICU transfer were unchanged.

Trial Registration

ClinicalTrials.gov Identifier: NCT03697070.

Stewardship Prompts to Improve Antibiotic Selection for Urinary Tract Infection: The INSPIRE Randomized Clinical Trial

Importance

Urinary tract infection (UTI) is the second most common infection leading to hospitalization and is often associated with gram-negative multidrug-resistant organisms (MDROs). Clinicians overuse extended-spectrum antibiotics although most patients are at low risk for MDRO infection. Safe strategies to limit overuse of empiric antibiotics are needed.

Objective

To evaluate whether computerized provider order entry (CPOE) prompts providing patient- and pathogen-specific MDRO risk estimates could reduce use of empiric extended-spectrum antibiotics for treatment of UTI.

Design, Setting, and Participants

Cluster-randomized trial in 59 US community hospitals comparing the effect of a CPOE stewardship bundle (education, feedback, and real-time and risk-based CPOE prompts; 29 hospitals) vs routine stewardship (n = 30 hospitals) on antibiotic selection during the first 3 hospital days (empiric period) in noncritically ill adults (≥18 years) hospitalized with UTI with an 18-month baseline (April 1, 2017-September 30, 2018) and 15-month intervention period (April 1, 2019-June 30, 2020).

Interventions

CPOE prompts recommending empiric standard-spectrum antibiotics in patients ordered to receive extended-spectrum antibiotics who have low estimated absolute risk (<10%) of MDRO UTI, coupled with feedback and education.

Main Outcomes and Measures

The primary outcome was empiric (first 3 days of hospitalization) extended-spectrum antibiotic days of therapy. Secondary outcomes included empiric vancomycin and antipseudomonal days of therapy. Safety outcomes included days to intensive care unit (ICU) transfer and hospital length of stay. Outcomes were assessed using generalized linear mixed-effect models to assess differences between the baseline and intervention periods.

Results

Among 127 403 adult patients (71 991 baseline and 55 412 intervention period) admitted with UTI in 59 hospitals, the mean (SD) age was 69.4 (17.9) years, 30.5% were male, and the median Elixhauser Comorbidity Index count was 4 (IQR, 2-5). Compared with routine stewardship, the group using CPOE prompts had a 17.4% (95% CI, 11.2%-23.2%) reduction in empiric extended-spectrum days of therapy (rate ratio, 0.83 [95% CI, 0.77-0.89]; P < .001). The safety outcomes of mean days to ICU transfer (6.6 vs 7.0 days) and hospital length of stay (6.3 vs 6.5 days) did not differ significantly between the routine and intervention groups, respectively.

Conclusions and Relevance

Compared with routine stewardship, CPOE prompts providing real-time recommendations for standard-spectrum antibiotics for patients with low MDRO risk coupled with feedback and education significantly reduced empiric extended-spectrum antibiotic use among noncritically ill adults admitted with UTI without changing hospital length of stay or days to ICU transfers.

Trial Registration

ClinicalTrials.gov Identifier: NCT03697096.

Evaluation of appropriateness of alerts overrides and physicians' responses of the medication-related clinical decision support system in China, a hospital-based study

This study was designed to investigate the state quo of the appropriateness of alerts overrides of the medication-related clinical decision support system (MRCDSS) in China. The medication-related alerts in one hospital from Jan 2022 to Dec 2022 were acquired and sampled. Rates of alert overrides, appropriateness of alert generation and physicians' responses were observed. Total 14,612 medication-related alerts (≤ level 3) were recorded, of those, 12,659 (86.6%) alerts were overridden. The top 3 alert types were: drug and diagnosis contraindications (23.8%), drug and test value contraindications (23.3%), and compatibility issues (17.7%). Of all sampled 1,501 alerts, 80.2% of them were appropriately overridden by the physicians. The appropriate rate of alert generation was 57.9% and the inappropriate rate was 42.1%. The inappropriate rate of physicians' responses was 17.8%, and 2.0% physicians' responses were undetermined. A few medications accounted for over 10% of overrides, 88.3% of "overridden reasons" inputted by the physicians were meaningless characters or values, indicating an obvious "alert fatigue" in these physicians. Our results indicated that the overridden rate of MRCDSS in China was still high, and appropriateness of generation of alert was quite low. These data indicated that the MRCDSS currently using in China still needs constantly optimization and timely maintenance. Proper sensitivity to reduce triggering of useless alerts and generation of alert fatigue might play a vital role. We believed that these findings are helpful for better understanding the state quo of MRCDSS in China and providing useful insights for future developing and improving MRCDSS.

Optimising interruptive clinical decision support alerts for antithrombotic duplicate prescribing in hospital

INTRODUCTION

Duplicate prescribing clinical decision support alerts can prevent important prescribing errors but are frequently the cause of much alert fatigue. Stat dose prescriptions are a known reason for overriding these alerts. This study aimed to evaluate the effect of excluding stat dose prescriptions from duplicate prescribing alerts for antithrombotic medicines on alert burden, prescriber adherence, and prescribing.

MATERIALS AND METHODS

A before (January 1st, 2017 to August 31st, 2022) and after (October 5th, 2022 to September 30th, 2023) study was undertaken of antithrombotic duplicate prescribing alerts and prescribing following a change in alert settings. Alert and prescribing data for antithrombotic medicines were joined, processed, and analysed to compare alert rates, adherence, and prescribing. Alert burden was assessed as alerts per 100 prescriptions. Adherence was measured at the point of the alert as whether the prescriber accepted the alert and following the alert as whether a relevant prescription was ceased within an hour. Co-prescribing of antithrombotic stat dose prescriptions was assessed pre- and post-alert reconfiguration.

RESULTS

Reconfiguration of the alerts reduced the alert rate by 29 % (p < 0.001). The proportion of alerts associated with cessation of antithrombotic duplication significantly increased (32.8 % to 44.5 %, p < 0.001). Adherence at the point of the alert increased 1.2 % (4.8 % to 6.0 %, p = 0.012) and 11.5 % (29.4 % to 40.9 %, p < 0.001) within one hour of the alert. When ceased after the alert over 80 % of duplicate prescriptions were ceased within 2 min of overriding. Antithrombotic stat dose co-prescribing was unchanged for 4 out of 5 antithrombotic duplication alert rules.

CONCLUSION

By reconfiguring our antithrombotic duplicate prescribing alerts, we reduced alert burden and increased alert adherence. Many prescribers ceased duplicate prescribing within 2 min of alert override highlighting the importance of incorporating post-alert measures in accurately determining prescriber alert adherence.

Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support

OBJECTIVES

To evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts.

MATERIALS AND METHODS

We extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness.

RESULTS

Five CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4 ± 1.2, human: 2.7 ± 1.2, P = .001).

CONCLUSION

End-user comments provide clinicians' immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.

The use of artificial intelligence to optimize medication alerts generated by clinical decision support systems: a scoping review

OBJECTIVE

Current Clinical Decision Support Systems (CDSSs) generate medication alerts that are of limited clinical value, causing alert fatigue. Artificial Intelligence (AI)-based methods may help in optimizing medication alerts. Therefore, we conducted a scoping review on the current state of the use of AI to optimize medication alerts in a hospital setting. Specifically, we aimed to identify the applied AI methods used together with their performance measures and main outcome measures.

MATERIALS AND METHODS

We searched Medline, Embase, and Cochrane Library database on May 25, 2023 for studies of any quantitative design, in which the use of AI-based methods was investigated to optimize medication alerts generated by CDSSs in a hospital setting. The screening process was supported by ASReview software.

RESULTS

Out of 5625 citations screened for eligibility, 10 studies were included. Three studies (30%) reported on both statistical performance and clinical outcomes. The most often reported performance measure was positive predictive value ranging from 9% to 100%. Regarding main outcome measures, alerts optimized using AI-based methods resulted in a decreased alert burden, increased identification of inappropriate or atypical prescriptions, and enabled prediction of user responses. In only 2 studies the AI-based alerts were implemented in hospital practice, and none of the studies conducted external validation.

DISCUSSION AND CONCLUSION

AI-based methods can be used to optimize medication alerts in a hospital setting. However, reporting on models' development and validation should be improved, and external validation and implementation in hospital practice should be encouraged.

Adoption by clinicians of electronic order communications in NHS secondary care: a descriptive account

BACKGROUND

Due to the rapid advancement in information technology, changes to communication modalities are increasingly implemented in healthcare. One such modality is Computerised Provider Order Entry (CPOE) systems which replace paper, verbal or telephone orders with electronic booking of requests. We aimed to understand the uptake, and user acceptability, of CPOE in a large National Health Service hospital system.

METHODS

This retrospective single-centre study investigates the longitudinal uptake of communications through the Prescribing, Information and Communication System (PICS). The development and configuration of PICS are led by the doctors, nurses and allied health professionals that use it and requests for CPOE driven by clinical need have been described.Records of every request (imaging, specialty review, procedure, laboratory) made through PICS were collected between October 2008 and July 2019 and resulting counts were presented. An estimate of the proportion of completed requests made through the system has been provided for three example requests. User surveys were completed.

RESULTS

In the first 6 months of implementation, a total of 832 new request types (imaging types and specialty referrals) were added to the system. Subsequently, an average of 6.6 new request types were added monthly. In total, 8 035 132 orders were requested through PICS. In three example request types (imaging, endoscopy and full blood count), increases in the proportion of requests being made via PICS were seen. User feedback at 6 months reported improved communications using the electronic system.

CONCLUSION

CPOE was popular, rapidly adopted and diversified across specialties encompassing wide-ranging requests.

[Inability of hospital computerised physician order entry systems to secure the use of concentrated potassium intravenous solutions]

OBJECTIVES

To determine whether hospital computerised physician order entry (CPOE) systems contribute to securing intravenous potassium chloride (KCl) prescriptions with reference to the recommendations issued by French healthcare agencies.

METHODS

We sent a questionnaire to the members of the Association pour le Digital et l'Information en Pharmacie.

RESULTS

More than three quarters of the 84 responses received involving 23 CPOE systems indicate that it is possible to: prescribe an ampoule of concentrated potassium chloride 10% 10mL intravenously without any diluents (80%); prescribe 4g of KCl in a bag of 500mL of NaCl 0,9% (98%); prescribe a solution that contains 6 grams of KCl per liter (94%); prescribe the administration of an injectable ampoule orally by means of a free text comment (83%). Nearly half of the responses indicate that it is possible to prescribe: concentrated KCl ampoules as administration solvent (50%); an injectable vial to be administered by oral route (52%).

CONCLUSION

At least 23 hospital CPOE systems are unable to secure the prescriptions of injectable KCl. This finding lifts the veil on an unthought, namely the role of CPOE systems in securing high-risk medications. In order to solve this problem, it should be mandatory that health information technology vendors pay particular attention to these drugs. With regard to injectable KCl, the utilisation of a dilution vehicle, maximum concentration and maximum infusion flow rate are the first four constraints to be satisfied.

Randomized Comparison of Electronic Health Record Alert Types in Eliciting Responses about Prognosis in Gynecologic Oncology Patients

OBJECTIVES

 To compare the ability of different electronic health record alert types to elicit responses from users caring for cancer patients benefiting from goals of care (GOC) conversations.

METHODS

 A validated question asking if the user would be surprised by the patient's 6-month mortality was built as an Epic BestPractice Advisory (BPA) alert in three versions-(1) Required on Open chart (pop-up BPA), (2) Required on Close chart (navigator BPA), and (3) Optional Persistent (Storyboard BPA)-randomized using patient medical record number. Meaningful responses were defined as "Yes" or "No," rather than deferral. Data were extracted over 6 months.

RESULTS

 Alerts appeared for 685 patients during 1,786 outpatient encounters. Measuring encounters where a meaningful response was elicited, rates were highest for Required on Open (94.8% of encounters), compared with Required on Close (90.1%) and Optional Persistent (19.7%) (p < 0.001). Measuring individual alerts to which responses were given, they were most likely meaningful with Optional Persistent (98.3% of responses) and least likely with Required on Open (68.0%) (p < 0.001). Responses of "No," suggesting poor prognosis and prompting GOC, were more likely with Optional Persistent (13.6%) and Required on Open (10.3%) than with Required on Close (7.0%) (p = 0.028).

CONCLUSION

 Required alerts had response rates almost five times higher than optional alerts. Timing of alerts affects rates of meaningful responses and possibly the response itself. The alert with the most meaningful responses was also associated with the most interruptions and deferral responses. Considering tradeoffs in these metrics is important in designing clinical decision support to maximize success.

Use of Computerized Physician Order Entry with Clinical Decision Support to Prevent Dose Errors in Pediatric Medication Orders: A Systematic Review

BACKGROUND

Prescribing is a high-risk task within the pediatric medication-use process and requires defenses to prevent errors. Such system-centric defenses include electronic health record systems with computerized physician order entry (CPOE) and clinical decision support (CDS) tools that assist safe prescribing. The objective of this study was to examine the effects of CPOE systems with CDS functions in preventing dose errors in pediatric medication orders.

MATERIAL AND METHODS

This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 criteria and Synthesis Without Meta-Analysis (SWiM) items. The study protocol was registered in PROSPERO (CRD42021277413). The final literature search on MEDLINE (Ovid), Scopus, Web of Science, and EMB Reviews was conducted on 10 September 2023. Only peer-reviewed studies considering both CPOE and CDS systems in pediatric inpatient or outpatient settings were included. Study selection, data extraction, and evidence quality assessment (JBI critical appraisal tool assessment and GRADE approach) were carried out by two individual reviewers. Vote counting method was used to evaluate the effects of CPOE-CDS systems on dose errors rates.

RESULTS

A total of 17 studies published in 2007-2021 met the inclusion criteria. The most used CDS tools were dose range check (n = 14), dose calculator (n = 8), and dosing frequency check (n = 8). Alerts were recorded in 15 studies. A statistically significant reduction in dose errors was found in eight studies, whereas an increase of dose errors was not reported.

CONCLUSIONS

The CPOE-CDS systems have the potential to reduce pediatric dose errors. Most beneficial interventions seem to be system customization, implementing CDS alerts, and the use of dose range check. While human factors are still present within the medication use process, further studies and development activities are needed to optimize the usability of CPOE-CDS systems.

Retrospective analysis of the impact of electronic medical record alerts on low value care in a pediatric hospital

OBJECTIVES

Hospital costs continue to rise unsustainably. Up to 20% of care is wasteful including low value care (LVC). This study aimed to understand whether electronic medical record (EMR) alerts are effective at reducing pediatric LVC and measure the impact on hospital costs.

MATERIALS AND METHODS

Using EMR data over a 76-month period, we evaluated changes in 4 LVC practices following the implementation of EMR alerts, using time series analysis to control for underlying time-based trends, in a large pediatric hospital in Australia. The main outcome measure was the change in rate of each LVC practice. Balancing measures included the rate of alert adherence as a proxy measure for risk of alert fatigue. Hospital costs were calculated by the volume of LVC avoided multiplied by the unit costs. Costs of the intervention were calculated from clinician and analyst time required.

RESULTS

All 4 LVC practices showed a statistically significant reduction following alert implementation. Two LVC practices (blood tests) showed an abrupt change, associated with high rates of alert adherence. The other 2 LVC practices (bronchodilator use in bronchiolitis and electrocardiogram ordering for sleeping bradycardia) showed an accelerated rate of improvement compared to baseline trends with lower rates of alert adherence. Hospital savings were $325 to $180 000 per alert.

DISCUSSION AND CONCLUSION

EMR alerts are effective in reducing pediatric LVC practices and offer a cost-saving opportunity to the hospital. Further efforts to leverage EMR alerts in pediatric settings to reduce LVC are likely to support future sustainable healthcare delivery.

A Practical Approach to Optimize Computerized Provider Order Entry Systems and Reduce Clinician Burden: Pre-Post Evaluation of Vendor-Derived "Order Friction" Data

Computerized provider order entry (CPOE) systems have been cited as a significant contributor to clinician burden. Vendor-derived measures and data sets have been developed to help with optimization of CPOE systems. We describe how we analyzed vendor-derived Order Friction (OF) EHR log data at our health system and propose a practical approach for optimizing CPOE systems by reducing OF. We also conducted a pre-post intervention study using OF data to evaluate the impact of defaulting the frequency of urine, stool and nasal swab tests and found that all modified orders had significantly fewer changes required per order (p<0.01). Our proposed approach is a six-step process: 1) understand the ordering process, 2) understand OF data elements contextually, 3) explore ordering user-level factors, 4) evaluate order volume and friction from different order sources, 5) optimize order-level design, 6) identify high volume alerts to evaluate for appropriateness.

Detecting atypical alert behavior through statistical process control: Clinical decision support alert frequency visualizations

Clinical decision support (CDS) alerts are designed to work according to a set of clearly defined criteria and have the potential to improve clinical care. To efficiently and proactively review abnormally functioning CDS alerts, we postulate that the introduction of a dashboard with statistical process control (SPC) charting will lead to effective detection of erratic alert behavior. We identified custom CDS alerts from an academic medical center that were recorded and monitored in a longitudinal fashion and the data warehouses where this information was stored. We created a dashboard of alert frequency using SPC charts, applied SPC rules for classification of variation, and validated dashboard data. From June-August 2022, the dashboard effectively pulled in data to visually depict alert behavior. SPC-defined parameters for standard deviation from the mean were applied to visualizations and allowed for rapid review of alerts with greatest variation. These alerts were subsequently investigated, and it was determined that they were functioning correctly. The most profound abnormalities detected during implementation reflected changes in practice and not system errors, though further investigation into thresholds for statistical significance will benefit this field. We conclude that SPC visualizations are a time-efficient and effective method of identifying CDS malfunctions.

Addressing Alert Fatigue by Replacing a Burdensome Interruptive Alert with Passive Clinical Decision Support

BACKGROUND

Recognizing that alert fatigue poses risks to patient safety and clinician wellness, there is a growing emphasis on evaluation and governance of electronic health record clinical decision support (CDS). This is particularly critical for interruptive alerts to ensure that they achieve desired clinical outcomes while minimizing the burden on clinicians. This study describes an improvement effort to address a problematic interruptive alert intended to notify clinicians about patients needing coronavirus disease 2019 (COVID) precautions and how we collaborated with operational leaders to develop an alternative passive CDS system in acute care areas.

OBJECTIVES

Our dual aim was to reduce the alert burden by redesigning the CDS to adhere to best practices for decision support while also improving the percent of admitted patients with symptoms of possible COVID who had appropriate and timely infection precautions orders.

METHODS

Iterative changes to CDS design included adjustment to alert triggers and acknowledgment reasons and development of a noninterruptive rule-based order panel for acute care areas. Data on alert burden and appropriate precautions orders on symptomatic admitted patients were followed over time on run and attribute (p) and individuals-moving range control charts.

RESULTS

At baseline, the COVID alert fired on average 8,206 times per week with an alert per encounter rate of 0.36. After our interventions, the alerts per week decreased to 1,449 and alerts per encounter to 0.07 equating to an 80% reduction for both metrics. Concurrently, the percentage of symptomatic admitted patients with COVID precautions ordered increased from 23 to 61% with a reduction in the mean time between COVID test and precautions orders from 19.7 to -1.3 minutes.

CONCLUSION

CDS governance, partnering with operational stakeholders, and iterative design led to successful replacement of a frequently firing interruptive alert with less burdensome passive CDS that improved timely ordering of COVID precautions.

Sharpening clinical decision support alert and reminder designs with MINDSPACE: A systematic review

BACKGROUND

Clinical decision support (CDS) alerts and reminders aim to influence clinical decisions, yet they are often designed without considering human decision-making behaviour. While this behaviour is comprehensively described by behavioural economics (BE), the sheer volume of BE literature poses a challenge to designers when identifying behavioural effects with utility to alert and reminder designs. This study tackles this challenge by focusing on the MINDSPACE framework for behaviour change, which collates nine behavioural effects that profoundly influence human decision-making behaviour: Messenger, Incentives, Norms, Defaults, Salience, Priming, Affect, Commitment, and Ego.

METHOD

A systematic review searching MEDLINE, Embase, PsycINFO, and CINAHL Plus to explore (i) the usage of MINDSPACE effects in alert and reminder designs and (ii) the efficacy of those alerts and reminders in influencing clinical decisions. The search queries comprised ten Boolean searches, with nine focusing on the MINDSPACE effects and one focusing on the term mindspace.

RESULTS

50 studies were selected from 1791 peer-reviewed journal articles in English from 1970 to 2022. Except for ego, eight of nine MINDSPACE effects were utilised to design alerts and reminders, with defaults and norms utilised the most in alerts and reminders, respectively. Overall, alerts and reminders informed by MINDSPACE effects showed an average 71% success rate in influencing clinical decisions (alerts 73%, reminders 69%). Most studies utilised a single effect in their design, with higher efficacy for alerts (64%) than reminders (41%). Others utilised multiple effects, showing higher efficacy for reminders (28%) than alerts (9%).

CONCLUSION

This review presents sufficient evidence demonstrating the MINDSPACE framework's merits for designing CDS alerts and reminders with human decision-making considerations. The framework can adequately address challenges in identifying behavioural effects pertinent to the effective design of CDS alerts and reminders. The review also identified opportunities for future research into other relevant effects (e.g., framing).

Ability of machine-learning based clinical decision support system to reduce alert fatigue, wrong-drug errors, and alert users about look alike, sound alike medication

BACKGROUND AND OBJECTIVE

The overall benefits of using clinical decision support systems (CDSSs) can be restrained if physicians inadvertently ignore clinically useful alerts due to "alert fatigue" caused by an excessive number of clinically irrelevant warnings. Moreover, inappropriate drug errors, look-alike/sound-alike (LASA) drug errors, and problem list documentation are common, costly, and potentially harmful. This study sought to evaluate the overall performance of a machine learning-based CDSS (MedGuard) for triggering clinically relevant alerts, acceptance rate, and to intercept inappropriate drug errors as well as LASA drug errors.

METHODS

We conducted a retrospective study that evaluated MedGuard alerts, the alert acceptance rate, and the rate of LASA alerts between July 1, 2019, and June 31, 2021, from outpatient settings at an academic hospital. An expert pharmacist checked the suitability of the alerts, rate of acceptance, wrong-drug errors, and confusing drug pairs.

RESULTS

Over the two-year study period, 1,206,895 prescriptions were ordered and a total of 28,536 alerts were triggered (alert rate: 2.36 %). Of the 28,536 alerts presented to physicians, 13,947 (48.88 %) were accepted. A total of 8,014 prescriptions were changed/modified (28.08 %, 8,014/28,534) with the most common reasons being adding and/or deleting diseases (52.04 %, 4,171/8,014), adding and/or deleting drugs (21.89 %, 1,755/8,014) and others (35.48 %, 2,844/ 8,014). However, the rate of drug error interception was 1.64 % (470 intercepted errors out of 28,536 alerts), which equates to 16.4 intercepted errors per 1000 alerted orders.

CONCLUSION

This study shows that machine learning based CDSS, MedGuard, has an ability to improve patients' safety by triggering clinically valid alerts. This system can also help improve problem list documentation and intercept inappropriate drug errors and LASA drug errors, which can improve medication safety. Moreover, high acceptance of alert rates can help reduce clinician burnout and adverse events.

Monitoring for adverse drug events of high-risk medications with a computerized clinical decision support system: a prospective cohort study

Monitoring is recommended to prevent severe adverse drug events, but such examinations are often missed. To increase the number of monitoring that should be ordered for high-risk medications, we introduced a clinical decision support system (CDSS) that alerts and orders the monitoring for high-risk medications in an outpatient setting. We conducted a 2-year prospective cohort study at a tertiary care teaching hospital before (phase 1) and after (phase 2) the activation of a CDSS. The CDSS automatically provided alerts for liver function tests for vildagliptin, thyroid function tests for immune checkpoint inhibitors (ICIs) and multikinase inhibitors (MKIs), and a slit-lamp examination of the eyes for oral amiodarone when outpatients were prescribed the medications but not examined for a fixed period. The order of laboratory tests automatically appeared if alert was accepted. The alerts were hidden and did not appear on the display before activation of the CDSS. The outcomes were the number of prescriptions with alerts and examinations. During the study period, 330 patients in phase 1 and 307 patients in phase 2 were prescribed vildagliptin, 20 patients in phase 1 and 19 patients in phase 2 were prescribed ICIs or MKIs, and 72 patients in phase 1 and 66 patients in phase 2 were prescribed oral amiodarone. The baseline characteristics were similar between the phases. In patients prescribed vildagliptin, the proportion of alerts decreased significantly (38% vs 27%, P < 0.0001), and the proportion of examinations increased significantly (0.9% vs 4.0%, P < 0.0001) after activation of the CDSS. In patients prescribed ICIs or MKIs, the proportion of alerts decreased significantly (43% vs 11%, P < 0.0001), and the proportion of examinations increased numerically, but not significantly (2.6% vs 7.0%, P = 0.13). In patients prescribed oral amiodarone, the proportion of alerts decreased (86% vs 81%, P = 0.055), and the proportion of examinations increased (2.2% and 3.0%, P = 0.47); neither was significant. The CDSS has potential to increase the monitoring for high-risk medications. Our study also highlighted the limited acceptance rate of monitoring by CDSS. Further studies are needed to explore the generalizability to other medications and the cause of the limited acceptance rates among physicians.

Clinician Response to Pharmacogenetic Clinical Decision Support Alerts

The objective of this study was to characterize clinician response following standardization of pharmacogenetic (PGx) clinical decision support alerts at University of Florida (UF) Health. A retrospective analysis of all PGx alerts that fired at a tertiary academic medical center from August 2020 through May 2022 was performed. Alert acceptance rate was calculated and compared across six gene-drug pairs, patient care setting, and clinician specialty. The disposition of the triggering medication was compared with the alert response and evaluated for congruence. There were a total of 818 alerts included for analysis of alert response, 557 alerts included in acceptance rate calculations, and 392 alerts with sufficient information to assess clinical response. The overall acceptance rate was 63%. The alert response and clinical response were congruent for 47% of alerts. Alert response was influenced by the triggering gene-drug pair, clinician specialty, patient care setting, and specialty of the provider who initially ordered the PGx test. Clinical response was mostly incongruent with alert response. Alert acceptance is influenced by the triggering gene-drug pair, clinician specialty, and care setting. Alert response is not a reliable surrogate marker for clinical action. Any future research into the impact of clinical decision support (CDS) alerts should focus on clinical response rather than alert response. Given the reliance on CDS alerts to enhance uptake of PGx, it is worthwhile to further investigate their impact on prescribing and patient outcomes.

Effect of knowledgebase transition of a clinical decision support system on medication order and alert patterns in an emergency department

A knowledgebase (KB) transition of a clinical decision support (CDS) system occurred at the study site. The transition was made from one commercial database to another, provided by a different vendor. The change was applied to all medications in the institute. The aim of this study was to analyze the effect of KB transition on medication-related orders and alert patterns in an emergency department (ED). Data of patients, medication-related orders and alerts, and physicians in the ED from January 2018 to December 2020 were analyzed in this study. A set of definitions was set to define orders, alerts, and alert overrides. Changes in order and alert patterns before and after the conversion, which took place in May 2019, were assessed. Overall, 101,450 patients visited the ED, and 1325 physicians made 829,474 prescription orders to patients during visit and at discharge. Alert rates (alert count divided by order count) for periods A and B were 12.6% and 14.1%, and override rates (alert override count divided by alert count) were 60.8% and 67.4%, respectively. Of the 296 drugs that were used more than 100 times during each period, 64.5% of the drugs had an increase in alert rate after the transition. Changes in alert rates were tested using chi-squared test and Fisher's exact test. We found that the CDS system knowledgebase transition was associated with a significant change in alert patterns at the medication level in the ED. Careful consideration is advised when such a transition is performed.

Impact of Predictive Text Clinical Decision Support on Imaging Order Entry in the Emergency Department

PURPOSE

Imaging clinical decision support (CDS) is designed to assist providers in selecting appropriate imaging studies and is now federally required. The aim of this study was to understand the effect of CDS on decisions and workflows in the emergency department (ED).

METHODS

The authors' institution's order entry platform serves up structured indications for imaging orders. Imaging orders are scored by CDS on the basis of appropriate use criteria (AUC). CDS triggers alerts for imaging orders with low AUC scores. Because free text alone cannot be scored by CDS, an artificial intelligence predictive text (AIPT) module was implemented to guide the selection of structured indications when free-text indications are entered. A total of 17,355 imaging orders in the ED over 6 months were retrospectively analyzed.

RESULTS

CDS alerts for low AUC scores were triggered for 3% of all imaging study orders (522 of 17,355). Providers spent an average of 24 seconds interacting with alerts. In 18 of 522 imaging orders with alerts, alternative studies were ordered. After AIPT implementation, the percentage of unscored studies significantly decreased from 81% to 45% (P < .001).

CONCLUSIONS

In a quaternary academic ED, CDS alerts triggered by low AUC scores caused minimal increase in time spent on imaging order entry but had a relatively marginal impact on imaging study selection. AIPT implementation increased the number of scored studies and could potentially enhance CDS effects. CDS implementation enables the collection of novel data regarding which imaging studies receive low AUC scores. Future work could include exploring alternative models of CDS implementation to maximize its impact.

Evaluation of clinical decision support systems in oncology: An updated systematic review

With increasing reliance on technology in oncology, the impact of digital clinical decision support (CDS) tools needs to be examined. A systematic review update was conducted and peer-reviewed literature from 2016 to 2022 were included if CDS tools were used for live decision making and comparatively assessed quantitative outcomes. 3369 studies were screened and 19 were included in this updated review. Combined with a previous review of 24 studies, a total of 43 studies were analyzed. Improvements in outcomes were observed in 42 studies, and 34 of these were of statistical significance. Computerized physician order entry and clinical practice guideline systems comprise the greatest number of evaluated CDS tools (13 and 10 respectively), followed by those that utilize patient-reported outcomes (8), clinical pathway systems (8) and prescriber alerts for best-practice advisories (4). Our review indicates that CDS can improve guideline adherence, patient-centered care, and care delivery processes in oncology.

Impact of Artificial Intelligence-Assisted Indication Selection on Appropriateness Order Scoring for Imaging Clinical Decision Support

PURPOSE

The aim of this study was to assess appropriateness scoring and structured order entry after the implementation of an artificial intelligence (AI) tool for analysis of free-text indications.

METHODS

Advanced outpatient imaging orders in a multicenter health care system were recorded 7 months before (March 1, 2020, to September 21, 2020) and after (October 20, 2020, to May 13, 2021) the implementation of an AI tool targeting free-text indications. Clinical decision support score (not appropriate, may be appropriate, appropriate, or unscored) and indication type (structured, free-text, both, or none) were assessed. The χ2 and multivariate logistic regression adjusting for covariables with bootstrapping were used.

RESULTS

In total, 115,079 orders before and 150,950 orders after AI tool deployment were analyzed. The mean patient age was 59.3 ± 15.5 years, and 146,035 (54.9%) were women; 49.9% of orders were for CT, 38.8% for MR, 5.9% for nuclear medicine, and 5.4% for PET. After deployment, scored orders increased to 52% from 30% (P < .001). Orders with structured indications increased to 67.3% from 34.6% (P < .001). On multivariate analysis, orders were more likely to be scored after tool deployment (odds ratio [OR], 2.7, 95% CI, 2.63-2.78; P < .001). Compared with physicians, orders placed by nonphysician providers were less likely to be scored (OR, 0.80; 95% CI, 0.78-0.83; P < .001). MR (OR, 0.84; 95% CI, 0.82-0.87) and PET (OR, 0.12; 95% CI, 0.10-0.13) were less likely to be scored than CT (; P < .001). After AI tool deployment, 72,083 orders (47.8%) remained unscored, 45,186 (62.7%) with free-text-only indications.

CONCLUSIONS

Embedding AI assistance within imaging clinical decision support was associated with increased structured indication orders and independently predicted a higher likelihood of scored orders. However, 48% of orders remained unscored, driven by both provider behavior and infrastructure-related barriers.

Post-implementation optimization of medication alerts in hospital computerized provider order entry systems: a scoping review

OBJECTIVES

A scoping review identified interventions for optimizing hospital medication alerts post-implementation, and characterized the methods used, the populations studied, and any effects of optimization.

MATERIALS AND METHODS

A structured search was undertaken in the MEDLINE and Embase databases, from inception to August 2023. Articles providing sufficient information to determine whether an intervention was conducted to optimize alerts were included in the analysis. Snowball analysis was conducted to identify additional studies.

RESULTS

Sixteen studies were identified. Most were based in the United States and used a wide range of clinical software. Many studies used inpatient cohorts and conducted more than one intervention during the trial period. Alert types studied included drug-drug interactions, drug dosage alerts, and drug allergy alerts. Six types of interventions were identified: alert inactivation, alert severity reclassification, information provision, use of contextual information, threshold adjustment, and encounter suppression. The majority of interventions decreased alert quantity and enhanced alert acceptance. Alert quantity decreased with alert inactivation by 1%-25.3%, and with alert severity reclassification by 1%-16.5% in 6 of 7 studies. Alert severity reclassification increased alert acceptance by 4.2%-50.2% and was associated with a 100% acceptance rate for high-severity alerts when implemented. Clinical errors reported in 4 studies were seen to remain stable or decrease.

DISCUSSION

Post-implementation medication optimization interventions have positive effects for clinicians when applied in a variety of settings. Less well reported are the impacts of these interventions on the clinical care of patients, and how endpoints such as alert quantity contribute to changes in clinician and pharmacist perceptions of alert fatigue.

CONCLUSION

Well conducted alert optimization can reduce alert fatigue by reducing overall alert quantity, improving clinical acceptance, and enhancing clinical utility.

A Data-Driven Approach to Evaluate Barcode-Assisted Medication Preparation Alerts at a Large Academic Medical Center

BACKGROUND

The purpose of this study was to develop a data-driven process to analyze barcode-assisted medication preparation alert data with a goal of minimizing inaccurate alerts.

METHODS

Medication preparation data for the prior three-month period was obtained from an electronic health record system. A dashboard was developed to identify recurrent, high-volume alerts and associated medication records. A randomization tool was used to obtain a prespecified proportion of the alerts to review for appropriateness. Alert root causes were identified by chart review. Depending on the alert's cause(s), targeted informatics build changes, workflow and purchasing changes, and/or staff education were implemented. The rate of alerts was measured postintervention for select drugs.

RESULTS

The institution averaged 31,000 medication preparation alerts per month. The "barcode not recognized" alert (13,000) was the highest volume over the study period. Eighty-five medication records were identified as contributing to a high volume of alerts (5,200/31,000), representing 49 unique drugs. Of the 85 medication records triggering alerts, 36 required staff education, 22 required informatics build changes, and 8 required workflow changes. Targeted interventions for 2 medications, resulted in reducing the rate of the "barcode not recognized" alert from 26.6% to 1.3% for polyethylene glycol and from 48.7% to 0% for cyproheptadine.

CONCLUSION

This quality improvement project highlighted opportunities to improve medication purchasing, storage, and preparation through development of a standard process to evaluate barcode-assisted medication preparation alert data. A data-driven approach can help identify and minimize inaccurate alerts ("noise") and promote medication safety.