Evaluating the Impact of Interruptive Alerts within a Health System: Use, Response Time, and Cumulative Time Burden

Clinician use of the Statin Choice Shared Decision-making Encounter Tool in a Major Health System

BACKGROUND

Effective shared decision-making (SDM) tools for use during clinical encounters are available, but, outside of study settings, little is known about clinician use of these tools in practice.

OBJECTIVE

To describe real-world use of an SDM encounter tool for statin prescribing, Statin Choice, embedded into the workflow of an electronic health record.

DESIGN

Cross-sectional study.

PARTICIPANTS

Clinicians and their statin-eligible patients who had outpatient encounters between January 2020 and June 2021 in Cleveland Clinic Health System.

MAIN MEASURES

Clinician use of Statin Choice was recorded within the Epic record system. We categorized each patient's 10-year atherosclerotic cardiovascular disease risk into low (< 5%), borderline (5-7.5%), intermediate (7.5-20%), and high (≥ 20%). Other patient factors included age, sex, insurance, and race. We used mixed effects logistic regression to assess the odds of using Statin Choice for statin-eligible patients, accounting for clustering by clinician and site. We generated a residual intraclass correlation coefficient (ICC) to characterize the impact of the clinician on Statin Choice use.

KEY RESULTS

Statin Choice was used in 7% of 68,505 eligible patients. Of 1047 clinicians, 48% used Statin Choice with ≥ 1 patient, and these clinicians used it with a median 9% of their patients (interquartile range: 3-22%). In the mixed effects logistic regression model, patient age (adjusted OR per year: 1.04; 95%CI 1.03-1.04) and 10-year ASVCD risk (aOR for 5-7.5% versus < 5% risk: 1.28; 95%CI: 1.14-1.44) were associated with use of Statin Choice. Black versus White race was associated with a lower odds of Statin Choice use (aOR: 0.83; 95%CI: 0.73-0.95), as was female versus male sex (aOR: 0.83; 95%CI: 0.76-0.90). The model ICC demonstrated that 53% of the variation in use of Statin Choice was clinician-driven.

CONCLUSIONS

Patient factors, including race and sex, were associated with clinician use of Statin Choice; half the variation in use was attributable to individual clinicians.

Do Decision Support Tools Decrease the Prevalence of Hospital-Acquired Venous Thromboembolisms When Compared to Clinical Judgement? A Single-Center Pre-Post Study

Introduction: Hospital-acquired venous thromboembolisms (HA-VTEs) carry a significant health burden on patients and a financial burden on hospitals due to reimbursement penalties. VTE prophylaxis at our institute was performed through utilizing an order set based on healthcare professionals' perceived level of risk. However, the use of standardized risk assessment models is recommended by multiple professional societies. Furthermore, integrating decision support tools (DST) based on the standardized risk assessment models has been shown to increase the administration of appropriate deep vein thrombosis (DVT) prophylaxis. Nonetheless, such scoring systems are not inherently flawless and their integration into EMR as a mandatory step can come at the risk of healthcare professional fatigue and burnout. We conducted a study to evaluate the incidence of HA-VTE and length of stay pre- and post implementation of a DST. Methods: We conducted a retrospective, pre-post-implementation observational study at a tertiary medical center after implementing a mandatory DST. The DST used Padua scores for medical patients and Caprini scores for surgical patients. Patients were identified through ICD-10 codes and outcomes were collected from electronic charts. Healthcare professionals were surveyed through an anonymous survey and stored securely. Statistical analysis was conducted by using R (version 3.4.3). Results: A total of 343 patients developed HA-VTE during the study period. Of these, 170 patients developed HA-VTE in the 9 months following the implementation of the DST, while 173 patients were identified in the 9 months preceding the implementation. There was no statistically significant difference in mean HA-VTE/1000 discharge/month pre- and post implementation (4.4 (SD 1.6) compared to 4.6 (SD 1.2), confidence interval [CI] -1.6 to 1.2, p = 0.8). The DST was used in 73% of all HA-VTE cases over the first 6 months of implementation. The hospital length of stay (LOS) was 14.2 (SD 1.9) days prior to implementation and 14.1 (SD 1.6) days afterwards. No statistically significant change in readmission rates was noted (8.8% (SD 2.6) prior to implementation and 15.53% (SD 9.6) afterwards, CI -14.27 to 0.74, p = 0.07). Of the 56 healthcare professionals who answered the survey, 84% (n = 47) reported to be dissatisfied or extremely dissatisfied with the DST, while 91% (n = 51) reported that it slowed them down. Conclusions: There were no apparent changes in the prevalence of HA-VTE, length of stay, or readmission rates when VTE prophylaxis was mandated through DST compared to a prior model which used order sets based on perceived risk. Further studies are needed to further evaluate the current risk assessment models and improve healthcare professionals' satisfaction with DST.

Optimising interruptive clinical decision support alerts for antithrombotic duplicate prescribing in hospital

INTRODUCTION

Duplicate prescribing clinical decision support alerts can prevent important prescribing errors but are frequently the cause of much alert fatigue. Stat dose prescriptions are a known reason for overriding these alerts. This study aimed to evaluate the effect of excluding stat dose prescriptions from duplicate prescribing alerts for antithrombotic medicines on alert burden, prescriber adherence, and prescribing.

MATERIALS AND METHODS

A before (January 1st, 2017 to August 31st, 2022) and after (October 5th, 2022 to September 30th, 2023) study was undertaken of antithrombotic duplicate prescribing alerts and prescribing following a change in alert settings. Alert and prescribing data for antithrombotic medicines were joined, processed, and analysed to compare alert rates, adherence, and prescribing. Alert burden was assessed as alerts per 100 prescriptions. Adherence was measured at the point of the alert as whether the prescriber accepted the alert and following the alert as whether a relevant prescription was ceased within an hour. Co-prescribing of antithrombotic stat dose prescriptions was assessed pre- and post-alert reconfiguration.

RESULTS

Reconfiguration of the alerts reduced the alert rate by 29 % (p < 0.001). The proportion of alerts associated with cessation of antithrombotic duplication significantly increased (32.8 % to 44.5 %, p < 0.001). Adherence at the point of the alert increased 1.2 % (4.8 % to 6.0 %, p = 0.012) and 11.5 % (29.4 % to 40.9 %, p < 0.001) within one hour of the alert. When ceased after the alert over 80 % of duplicate prescriptions were ceased within 2 min of overriding. Antithrombotic stat dose co-prescribing was unchanged for 4 out of 5 antithrombotic duplication alert rules.

CONCLUSION

By reconfiguring our antithrombotic duplicate prescribing alerts, we reduced alert burden and increased alert adherence. Many prescribers ceased duplicate prescribing within 2 min of alert override highlighting the importance of incorporating post-alert measures in accurately determining prescriber alert adherence.

Randomized Comparison of Electronic Health Record Alert Types in Eliciting Responses about Prognosis in Gynecologic Oncology Patients

OBJECTIVES

 To compare the ability of different electronic health record alert types to elicit responses from users caring for cancer patients benefiting from goals of care (GOC) conversations.

METHODS

 A validated question asking if the user would be surprised by the patient's 6-month mortality was built as an Epic BestPractice Advisory (BPA) alert in three versions-(1) Required on Open chart (pop-up BPA), (2) Required on Close chart (navigator BPA), and (3) Optional Persistent (Storyboard BPA)-randomized using patient medical record number. Meaningful responses were defined as "Yes" or "No," rather than deferral. Data were extracted over 6 months.

RESULTS

 Alerts appeared for 685 patients during 1,786 outpatient encounters. Measuring encounters where a meaningful response was elicited, rates were highest for Required on Open (94.8% of encounters), compared with Required on Close (90.1%) and Optional Persistent (19.7%) (p < 0.001). Measuring individual alerts to which responses were given, they were most likely meaningful with Optional Persistent (98.3% of responses) and least likely with Required on Open (68.0%) (p < 0.001). Responses of "No," suggesting poor prognosis and prompting GOC, were more likely with Optional Persistent (13.6%) and Required on Open (10.3%) than with Required on Close (7.0%) (p = 0.028).

CONCLUSION

 Required alerts had response rates almost five times higher than optional alerts. Timing of alerts affects rates of meaningful responses and possibly the response itself. The alert with the most meaningful responses was also associated with the most interruptions and deferral responses. Considering tradeoffs in these metrics is important in designing clinical decision support to maximize success.

Addressing Alert Fatigue by Replacing a Burdensome Interruptive Alert with Passive Clinical Decision Support

BACKGROUND

Recognizing that alert fatigue poses risks to patient safety and clinician wellness, there is a growing emphasis on evaluation and governance of electronic health record clinical decision support (CDS). This is particularly critical for interruptive alerts to ensure that they achieve desired clinical outcomes while minimizing the burden on clinicians. This study describes an improvement effort to address a problematic interruptive alert intended to notify clinicians about patients needing coronavirus disease 2019 (COVID) precautions and how we collaborated with operational leaders to develop an alternative passive CDS system in acute care areas.

OBJECTIVES

Our dual aim was to reduce the alert burden by redesigning the CDS to adhere to best practices for decision support while also improving the percent of admitted patients with symptoms of possible COVID who had appropriate and timely infection precautions orders.

METHODS

Iterative changes to CDS design included adjustment to alert triggers and acknowledgment reasons and development of a noninterruptive rule-based order panel for acute care areas. Data on alert burden and appropriate precautions orders on symptomatic admitted patients were followed over time on run and attribute (p) and individuals-moving range control charts.

RESULTS

At baseline, the COVID alert fired on average 8,206 times per week with an alert per encounter rate of 0.36. After our interventions, the alerts per week decreased to 1,449 and alerts per encounter to 0.07 equating to an 80% reduction for both metrics. Concurrently, the percentage of symptomatic admitted patients with COVID precautions ordered increased from 23 to 61% with a reduction in the mean time between COVID test and precautions orders from 19.7 to -1.3 minutes.

CONCLUSION

CDS governance, partnering with operational stakeholders, and iterative design led to successful replacement of a frequently firing interruptive alert with less burdensome passive CDS that improved timely ordering of COVID precautions.

Clinical Informatics in Critical Care Medicine

Continuous monitoring and treatment of patients in intensive care units generates vast amounts of data. Critical Care Medicine clinicians incorporate this continuously evolving data to make split-second, life or death decisions for management of these patients. Despite the abundance of data, it can be challenging to consider every accessible data point when making the quick decisions necessary at the point of care. Consequently, Clinical Informatics offers a natural partnership to improve the care for critically ill patients. The last two decades have seen a significant evolution in the role of Clinical Informatics in Critical Care Medicine. In this review, we will discuss how Clinical Informatics improves the care of critically ill patients by enhancing not only data collection and visualization but also bedside medical decision making. We will further discuss the evolving role of machine learning algorithms in Clinical Informatics as it pertains to Critical Care Medicine.

Meaningful time-related aspects of alerts in Clinical Decision Support Systems. A unified framework

Alerts are a common functionality of clinical decision support systems (CDSSs). Although they have proven to be useful in clinical practice, the alert burden can lead to alert fatigue and significantly reduce their usability and acceptance. Based on a literature review, we propose a unified framework consisting of a set of meaningful timestamps that allows the use of state-of-the-art measures for alert burden, such as alert dwell time, alert think time, and response time. In addition, it can be used to investigate other measures that could be relevant as regards dealing with this problem. Furthermore, we provide a case study concerning three different types of alerts to which the framework was successfully applied. We consider that our framework can easily be adapted to other CDSSs and that it could be useful for dealing with alert burden measurement thus contributing to its appropriate management.

Use of Clinical Decision Support to Improve the Laboratory Evaluation of Monoclonal Gammopathies

OBJECTIVES

There is considerable variation in ordering practices for the initial laboratory evaluation of monoclonal gammopathies (MGs) despite clear society guidelines to include serum free light chain (sFLC) testing. We assessed the ability of a clinical decision support (CDS) alert to improve guideline compliance and analyzed its clinical impact.

METHODS

We designed and deployed a targeted CDS alert to educate and prompt providers to order an sFLC assay when ordering serum protein electrophoresis (SPEP) testing.

RESULTS

The alert was highly effective at increasing the co-ordering of SPEP and sFLC testing. Preimplementation, 62.8% of all SPEP evaluations included sFLC testing, while nearly 90% of evaluations included an sFLC assay postimplementation. In patients with no prior sFLC testing, analysis of sFLC orders prompted by the alert led to the determination that 28.9% (800/2,769) of these patients had an abnormal κ/λ ratio. In 452 of these patients, the sFLC assay provided the only laboratory evidence of a monoclonal protein. Moreover, within this population, there were numerous instances of new diagnoses of multiple myeloma and other MGs.

CONCLUSIONS

The CDS alert increased compliance with society guidelines and improved the diagnostic evaluation of patients with suspected MGs.

Acceptance of Acute Kidney Injury Alert by Providers in Cardiac Surgery Intensive Care Unit

BACKGROUND

Acute kidney injury (AKI) is a common complication after cardiac surgery and is associated with worse outcomes. Its management relies on early diagnosis, and therefore, electronic alerts have been used to alert clinicians for development of AKI. Electronic alerts are, however, associated with high rates of alert fatigue.

OBJECTIVES

We designed this study to assess the acceptance of user-centered electronic AKI alert by clinicians.

METHODS

We developed a user-centered electronic AKI alert that alerted clinicians of development of AKI in a persistent yet noninterruptive fashion. As the goal of the alert was to alert toward new or worsening AKI, it disappeared 48 hours after being activated. We assessed the acceptance of the alert using surveys at 6 and 12 months after the alert went live.

RESULTS

At 6 months after their implementation, 38.9% providers reported that they would not have recognized AKI as early as they did without this alert. This number increased to 66.7% by 12 months of survey. Most providers also shared that they re-dosed or discontinued medications earlier, provided earlier management of volume status, avoided intravenous contrast use, and evaluated patients by using point-of-care ultrasounds more due to the alert. Overall, 83.3% respondents reported satisfaction with the electronic AKI alerts at 6 months and 94.4% at 12 months.

CONCLUSION

This study showed high rates of acceptance of a user-centered electronic AKI alert over time by clinicians taking care of patients with AKI.

Clinician Perspectives on Barriers and Enablers to Implementing an Inpatient Oncology Early Warning System: A Mixed-Methods Study

PURPOSE

To elicit end-user and stakeholder perceptions regarding design and implementation of an inpatient clinical deterioration early warning system (EWS) for oncology patients to better fit routine clinical practices and enhance clinical impact.

METHODS

In an explanatory-sequential mixed-methods study, we evaluated a stakeholder-informed oncology early warning system (OncEWS) using surveys and semistructured interviews. Stakeholders were physicians, advanced practice providers (APPs), and nurses. For qualitative data, we used grounded theory and thematic content analysis via the constant comparative method to identify determinants of OncEWS implementation.

RESULTS

Survey respondents generally agreed that an oncology-focused EWS could add value beyond clinical judgment, with nurses endorsing this notion significantly more strongly than other clinicians (nurse: median 5 on a 6-point scale [6 = strongly agree], interquartile range 4-5; doctors/advanced practice providers: 4 [4-5]; P = .005). However, some respondents would not trust an EWS to identify risk accurately (n = 36 [42%] somewhat or very concerned), while others were concerned that institutional culture would not embrace such an EWS (n = 17 [28%]).Interviews highlighted important aspects of the EWS and the local context that might facilitate implementation, including (1) a model tailored to the subtleties of oncology patients, (2) transparent model information, and (3) nursing-centric workflows. Interviewees raised the importance of sepsis as a common and high-risk deterioration syndrome.

CONCLUSION

Stakeholders prioritized maximizing the degree to which the OncEWS is understandable, informative, actionable, and workflow-complementary, and perceived these factors to be key for translation into clinical benefit.

The value of incorporating patient-consulted medication reconciliation in influencing drug-related actions in the outpatient rheumatology setting

Background

Unintentional changes to patients’ medicine regimens and drug non-adherence are discovered by medication reconciliation. High numbers of outpatient visits and medication reconciliation being time-consuming, make it challenging to perform medication reconciliation for all outpatients. Therefore, we aimed to get insight into the proportion of outpatient visits in which information obtained with medication reconciliation led to additional drug-related actions.

Methods

In October and November 2018, we performed a cross-sectional observational study at the rheumatology outpatient clinic. Based on a standardized data collection form, outpatient visits were observed by a pharmacy technician trained to observe and report all drug-related actions made by the rheumatologist. Afterwards, the nine observed rheumatologists and an expert panel, consisting of two rheumatologists and two pharmacists, were individually asked which drug information reported on the drug list composed by medication reconciliation was required to perform the drug-related actions. The four members of the expert panel discussed until consensus was reached about their assessment of the required information. Subsequently, a researcher determined if the required information was available in digital sources: electronic medical record (electronic prescribing system plus physician’s medical notes) or Dutch Nationwide Medication Record System.

Results

Of the 114 selected patients, 83 (73%) patients were included. If both digital drug sources were available, patient’s input during medication reconciliation resulted in additional information to perform drug-related actions according to the rheumatologist in 0% of the visits and according to the expert panel in 14%. If there was only access to the electronic medical record, the proportions were 8 and 29%, respectively. Patient’s input was especially required for starting a new drug and discussing drug-related problems.

Conclusions

If rheumatologists only had access to the electronic medical record, in 1 out of 3 visits the patient provided additional information during medication reconciliation which was required to perform a drug-related action. When rheumatologists had access to two digital sources, patient’s additional input during medication reconciliation was at most 14%. As the added value of patient’s input was highest when rheumatologists prescribe a new drug and/or discuss a drug-related problem, it may be considered that rheumatologists only perform medication reconciliation during the visit when performing one of these actions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-08391-7.

Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts

Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.

Impact of Kidney Failure Risk Prediction Clinical Decision Support on Monitoring and Referral in Primary Care Management of CKD: A Randomized Pragmatic Clinical Trial

Rationale & Objective

To design and implement clinical decision support incorporating a validated risk prediction estimate of kidney failure in primary care clinics and to evaluate the impact on stage-appropriate monitoring and referral.

Study Design

Block-randomized, pragmatic clinical trial.

Setting & Participants

Ten primary care clinics in the greater Boston area. Patients with stage 3-5 chronic kidney disease (CKD) were included. Patients were randomized within each primary care physician panel through a block randomization approach. The trial occurred between December 4, 2015, and December 3, 2016.

Intervention

Point-of-care noninterruptive clinical decision support that delivered the 5-year kidney failure risk equation as well as recommendations for stage-appropriate monitoring and referral to nephrology.

Outcomes

The primary outcome was as follows: Urine and serum laboratory monitoring test findings measured at one timepoint 6 months after the initial primary care visit and analyzed only in patients who had not undergone the recommended monitoring test in the preceding 12 months. The secondary outcome was nephrology referral in patients with a calculated kidney failure risk equation value of >10% measured at one timepoint 6 months after the initial primary care visit.

Results

The clinical decision support application requested and processed 569,533 Continuity of Care Documents during the study period. Of these, 41,842 (7.3%) documents led to a diagnosis of stage 3, 4, or 5 CKD by the clinical decision support application. A total of 5,590 patients with stage 3, 4, or 5 CKD were randomized and included in the study. The link to the clinical decision support application was clicked 122 times by 57 primary care physicians. There was no association between the clinical decision support intervention and the primary outcome. There was a small but statistically significant difference in nephrology referral, with a higher rate of referral in the control arm.

Limitations

Contamination within provider and clinic may have attenuated the impact of the intervention and may have biased the result toward null.

Conclusions

The noninterruptive design of the clinical decision support was selected to prevent cognitive overload; however, the design led to a very low rate of use and ultimately did not improve stage-appropriate monitoring.

Funding

Research reported in this publication was supported by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health under award K23DK097187.

Trial Registration

ClinicalTrials.gov Identifier: NCT02990897.

Modulators Influencing Medication Alert Acceptance: An Explorative Review

OBJECTIVES

Clinical decision support systems (CDSSs) use alerts to enhance medication safety and reduce medication error rates. A major challenge of medication alerts is their low acceptance rate, limiting their potential benefit. A structured overview about modulators influencing alert acceptance is lacking. Therefore, we aimed to review and compile qualitative and quantitative modulators of alert acceptance and organize them in a comprehensive model.

METHODS

In accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline, a literature search in PubMed was started in February 2018 and continued until October 2021. From all included articles, qualitative and quantitative parameters and their impact on alert acceptance were extracted. Related parameters were then grouped into factors, allocated to superordinate determinants, and subsequently further allocated into five categories that were already known to influence alert acceptance.

RESULTS

Out of 539 articles, 60 were included. A total of 391 single parameters were extracted (e.g., patients' comorbidity) and grouped into 75 factors (e.g., comorbidity), and 25 determinants (e.g., complexity) were consequently assigned to the predefined five categories, i.e., CDSS, care provider, patient, setting, and involved drug. More than half of all factors were qualitatively assessed (n = 21) or quantitatively inconclusive (n = 19). Furthermore, 33 quantitative factors clearly influenced alert acceptance (positive correlation: e.g., alert type, patients' comorbidity; negative correlation: e.g., number of alerts per care provider, moment of alert display in the workflow). Two factors (alert frequency, laboratory value) showed contradictory effects, meaning that acceptance was significantly influenced both positively and negatively by these factors, depending on the study. Interventional studies have been performed for only 12 factors while all other factors were evaluated descriptively.

CONCLUSION

This review compiles modulators of alert acceptance distinguished by being studied quantitatively or qualitatively and indicates their effect magnitude whenever possible. Additionally, it describes how further research should be designed to comprehensively quantify the effect of alert modulators.

Associations of physicians' prescribing experience, work hours, and workload with prescription errors

OBJECTIVE

We aimed to assess associations of physician's work overload, successive work shifts, and work experience with physicians' risk to err.

MATERIALS AND METHODS

This large-scale study included physicians who prescribed at least 100 systemic medications at Sheba Medical Center during 2012-2017 in all acute care departments, excluding intensive care units. Presumed medication errors were flagged by a high-accuracy computerized decision support system that uses machine-learning algorithms to detect potential medication prescription errors. Physicians' successive work shifts (first or only shift, second, and third shifts), workload (assessed by the number of prescriptions during a shift) and work-experience, as well as a novel measurement of physicians' prescribing experience with a specific drug, were assessed per prescription. The risk to err was determined for various work conditions.

RESULTS

1 652 896 medical orders were prescribed by 1066 physicians; The system flagged 3738 (0.23%) prescriptions as erroneous. Physicians were 8.2 times more likely to err during high than normal-low workload shifts (5.19% vs 0.63%, P < .0001). Physicians on their third or second successive shift (compared to a first or single shift) were more likely to err (2.1%, 1.8%, and 0.88%, respectively, P < .001). Lack of experience in prescribing a specific medication was associated with higher error rate (0.37% for the first 5 prescriptions vs 0.13% after over 40, P < .001).

DISCUSSION

Longer hours and less experience in prescribing a specific medication increase risk of erroneous prescribing.

CONCLUSION

Restricting successive shifts, reducing workload, increasing training and supervision, and implementing smart clinical decision support systems may help reduce prescription errors.