Optimization of drug-drug interaction alert rules in a pediatric hospital's electronic health record system using a visual analytics dashboard

A Geospatial Drug Abuse Risk Assessment and Monitoring Dashboard Tailored for School Students: Development Study With Requirement Analysis and Acceptance Evaluation

BACKGROUND

The enormous consequences of drugs include suicides, traffic accidents, and violence, affecting the individual, family, society, and country. Therefore, it is necessary to constantly identify and monitor the drug abuse rate among school-going youth. A geospatial dashboard is vital for the monitoring of drug abuse and related crime incidence in a decision support system.

OBJECTIVE

This paper mainly focuses on developing MyAsriGeo, a geospatial drug abuse risk assessment and monitoring dashboard tailored for school students. It introduces innovative functionality, seamlessly orchestrating the assessment of drug abuse usage patterns and risks using multivariate student data.

METHODS

A geospatial drug abuse dashboard for monitoring and analysis was designed and developed in this study based on agile methodology and prototyping. Using focus group and interviews, we first examined and gathered the requirements, feedback, and user approval of the MyAsriGeo dashboard. Experts and stakeholders such as the National Anti-Drugs Agency, police, the Federal Department of Town and Country Planning, school instructors, students, and researchers were among those who responded. A total of 20 specialists were involved in the requirement analysis and acceptance evaluation of the pilot and final version of the dashboard. The evaluation sought to identify various user acceptance aspects, such as ease of use and usefulness, for both the pilot and final versions, and 2 additional factors based on the Post-Study System Usability Questionnaire and Task-Technology Fit models were enlisted to assess the interface quality and dashboard sufficiency for the final version.

RESULTS

The MyAsriGeo geospatial dashboard was designed to meet the needs of all user types, as identified through a requirement gathering process. It includes several key functions, such as a geospatial map that shows the locations of high-risk areas for drug abuse, data on drug abuse among students, tools for assessing the risk of drug abuse in different areas, demographic information, and a self-problem test. It also includes the Alcohol, Smoking, and Substance Involvement Screening Test and its risk assessment to help users understand and interpret the results of student risk. The initial prototype and final version of the dashboard were evaluated by 20 experts, which revealed a significant improvement in the ease of use (P=.047) and usefulness (P=.02) factors and showed a high acceptance mean scores for ease of use (4.2), usefulness (4.46), interface quality (4.29), and sufficiency (4.13).

CONCLUSIONS

The MyAsriGeo geospatial dashboard is useful for monitoring and analyzing drug abuse among school-going youth in Malaysia. It was developed based on the needs of various stakeholders and includes a range of functions. The dashboard was evaluated by a group of experts. Overall, the MyAsriGeo geospatial dashboard is a valuable resource for helping stakeholders understand and respond to the issue of drug abuse among youth.

Interprofessional Evaluation of a Medication Clinical Decision Support System Prior to Implementation

BACKGROUND

 Computerized physician order entry (CPOE) and clinical decision support systems (CDSS) are widespread due to increasing digitalization of hospitals. They can be associated with reduced medication errors and improved patient safety, but also with well-known risks (e.g., overalerting, nonadoption).

OBJECTIVES

 Therefore, we aimed to evaluate a commonly used CDSS containing Medication-Safety-Validators (e.g., drug-drug interactions), which can be locally activated or deactivated, to identify limitations and thereby potentially optimize the use of the CDSS in clinical routine.

METHODS

 Within the implementation process of Meona (commercial CPOE/CDSS) at a German University hospital, we conducted an interprofessional evaluation of the CDSS and its included Medication-Safety-Validators following a defined algorithm: (1) general evaluation, (2) systematic technical and content-related validation, (3) decision of activation or deactivation, and possibly (4) choosing the activation mode (interruptive or passive). We completed the in-depth evaluation for exemplarily chosen Medication-Safety-Validators. Moreover, we performed a survey among 12 German University hospitals using Meona to compare their configurations.

RESULTS

 Based on the evaluation, we deactivated 3 of 10 Medication-Safety-Validators due to technical or content-related limitations. For the seven activated Medication-Safety-Validators, we chose the interruptive option ["PUSH-(&PULL)-modus"] four times (4/7), and a new, on-demand option ["only-PULL-modus"] three times (3/7). The site-specific configuration (activation or deactivation) differed across all participating hospitals in the survey and led to varying medication safety alerts for identical patient cases.

CONCLUSION

 An interprofessional evaluation of CPOE and CDSS prior to implementation in clinical routine is crucial to detect limitations. This can contribute to a sustainable utilization and thereby possibly increase medication safety.

Post-implementation optimization of medication alerts in hospital computerized provider order entry systems: a scoping review

OBJECTIVES

A scoping review identified interventions for optimizing hospital medication alerts post-implementation, and characterized the methods used, the populations studied, and any effects of optimization.

MATERIALS AND METHODS

A structured search was undertaken in the MEDLINE and Embase databases, from inception to August 2023. Articles providing sufficient information to determine whether an intervention was conducted to optimize alerts were included in the analysis. Snowball analysis was conducted to identify additional studies.

RESULTS

Sixteen studies were identified. Most were based in the United States and used a wide range of clinical software. Many studies used inpatient cohorts and conducted more than one intervention during the trial period. Alert types studied included drug-drug interactions, drug dosage alerts, and drug allergy alerts. Six types of interventions were identified: alert inactivation, alert severity reclassification, information provision, use of contextual information, threshold adjustment, and encounter suppression. The majority of interventions decreased alert quantity and enhanced alert acceptance. Alert quantity decreased with alert inactivation by 1%-25.3%, and with alert severity reclassification by 1%-16.5% in 6 of 7 studies. Alert severity reclassification increased alert acceptance by 4.2%-50.2% and was associated with a 100% acceptance rate for high-severity alerts when implemented. Clinical errors reported in 4 studies were seen to remain stable or decrease.

DISCUSSION

Post-implementation medication optimization interventions have positive effects for clinicians when applied in a variety of settings. Less well reported are the impacts of these interventions on the clinical care of patients, and how endpoints such as alert quantity contribute to changes in clinician and pharmacist perceptions of alert fatigue.

CONCLUSION

Well conducted alert optimization can reduce alert fatigue by reducing overall alert quantity, improving clinical acceptance, and enhancing clinical utility.

Evaluating the accuracy and efficacy of an electronic medical record alert to identify paediatric patients with low-risk febrile neutropenia

BACKGROUND

Point-of-care decision support, embedded into electronic medical record (EMR) workflows, has the potential to improve efficiency, reduce unwarranted variation and improve patient outcomes. A clinical-facing best practice advisory (BPA) in the Epic EMR system was developed to identify children admitted with low-risk febrile neutropenia (FN) who should be considered for treatment at home after a brief inpatient stay. We evaluated the accuracy and impact of this BPA and identify areas for improvement.

METHODS

The low-risk FN BPA was co-designed with key-stakeholders and implemented after a one-month testing phase. Mixed methodology was used to collect and analyse data. The sensitivity and positive predictive value of the BPA was calculated using FN episodes captured in a prospectively collected database. Overall effectiveness was defined as the proportion of alerts resulting in completion of a FN risk assessment flowsheet.

RESULTS

Over the 12-month period 176 FN episodes were admitted. Overall, the alert had poor sensitivity (58%) and positive predictive value (75%), failing to trigger in 62 (35%) episodes. In the episodes where the alert did trigger, the alert was frequently dismissed by clinicians (76%) and the overall effectiveness was extremely low (3%). Manual review of each FN episode without a BPA identified important design limitations and incorrect workflow assumptions.

DISCUSSION

Given the poor sensitivity and limited impact on clinician behaviour the low-risk BPA, in its current form, has not been an effective intervention at this site. While work is ongoing to enhance the accuracy of the BPA, alternative EMR workflows are likely required to improve the clinical impact.

Optimizing Best Practice Advisory alerts in electronic medical records with a multi-pronged strategy at a tertiary care hospital in Singapore

Objective

Clinical decision support (CDS) alerts can aid in improving patient care. One CDS functionality is the Best Practice Advisory (BPA) alert notification system, wherein BPA alerts are automated alerts embedded in the hospital's electronic medical records (EMR). However, excessive alerts can change clinician behavior; redundant and repetitive alerts can contribute to alert fatigue. Alerts can be optimized through a multipronged strategy. Our study aims to describe these strategies adopted and evaluate the resultant BPA alert optimization outcomes.

Materials and Methods

This retrospective single-center study was done at Jurong Health Campus. Aggregated, anonymized data on patient demographics and alert statistics were collected from January 1, 2018 to December 31, 2021. "Preintervention" period was January 1-December 31, 2018, and "postintervention" period was January 1-December 31, 2021. The intervention period was the intervening period. Categorical variables were reported as frequencies and proportions and compared using the chi-square test. Continuous data were reported as median (interquartile range, IQR) and compared using the Wilcoxon rank-sum test. Statistical significance was defined at P < .05.

Results

There was a significant reduction of 59.6% in the total number of interruptive BPA alerts, despite an increase in the number of unique BPAs from 54 to 360 from pre- to postintervention. There was a 74% reduction in the number of alerts from the 7 BPAs that were optimized from the pre- to postintervention period. There was a significant increase in percentage of overall interruptive BPA alerts with action taken (8 [IQR 7.7-8.4] to 54.7 [IQR 52.5-58.9], P-value < .05) and optimized BPAs with action taken (32.6 [IQR 32.3-32.9] to 72.6 [IQR 64.3-73.4], P-value < .05). We estimate that the reduction in alerts saved 3600 h of providers' time per year.

Conclusions

A significant reduction in interruptive alert volume, and a significant increase in action taken rates despite manifold increase in the number of unique BPAs could be achieved through concentrated efforts focusing on governance, data review, and visualization using a system-embedded tool, combined with the CDS Five Rights framework, to optimize alerts. Improved alert compliance was likely multifactorial-due to decreased repeated alert firing for the same patient; better awareness due to stakeholders' involvement; and less fatigue since unnecessary alerts were removed. Future studies should prospectively focus on patients' clinical chart reviews to assess downstream effects of various actions taken, identify any possibility of harm, and collect end-user feedback regarding the utility of alerts.

Refining Clinical Phenotypes to Improve Clinical Decision Support and Reduce Alert Fatigue: A Feasibility Study

BACKGROUND

Chronic kidney disease (CKD) is common and associated with adverse clinical outcomes. Most care for early CKD is provided in primary care, including hypertension (HTN) management. Computerized clinical decision support (CDS) can improve the quality of care for CKD but can also cause alert fatigue for primary care physicians (PCPs). Computable phenotypes (CPs) are algorithms to identify disease populations using, for example, specific laboratory data criteria.

OBJECTIVES

Our objective was to determine the feasibility of implementation of CDS alerts by developing CPs and estimating potential alert burden.

METHODS

We utilized clinical guidelines to develop a set of five CPs for patients with stage 3 to 4 CKD, uncontrolled HTN, and indications for initiation or titration of guideline-recommended antihypertensive agents. We then conducted an iterative data analytic process consisting of database queries, data validation, and subject matter expert discussion, to make iterative changes to the CPs. We estimated the potential alert burden to make final decisions about the scope of the CDS alerts. Specifically, the number of times that each alert could fire was limited to once per patient.

RESULTS

In our primary care network, there were 239,339 encounters for 105,992 primary care patients between April 1, 2018 and April 1, 2019. Of these patients, 9,081 (8.6%) had stage 3 and 4 CKD. Almost half of the CKD patients, 4,191 patients, also had uncontrolled HTN. The majority of CKD patients were female, elderly, white, and English-speaking. We estimated that 5,369 alerts would fire if alerts were triggered multiple times per patient, with a mean number of alerts shown to each PCP ranging from 0.07-to 0.17 alerts per week.

CONCLUSION

Development of CPs and estimation of alert burden allows researchers to iteratively fine-tune CDS prior to implementation. This method of assessment can help organizations balance the tradeoff between standardization of care and alert fatigue.

Development of a risk-tailored approach and dashboard for efficient management and monitoring of investigator-initiated trials

BACKGROUND

Most randomized controlled trials (RCTs) in the academic setting have limited resources for clinical trial management and monitoring. Inefficient conduct of trials was identified as an important source of waste even in well-designed studies. Thoroughly identifying trial-specific risks to enable focussing of monitoring and management efforts on these critical areas during trial conduct may allow for the timely initiation of corrective action and to improve the efficiency of trial conduct. We developed a risk-tailored approach with an initial risk assessment of an individual trial that informs the compilation of monitoring and management procedures in a trial dashboard.

METHODS

We performed a literature review to identify risk indicators and trial monitoring approaches followed by a contextual analysis involving local, national and international stakeholders. Based on this work we developed a risk-tailored management approach with integrated monitoring for RCTs and including a visualizing trial dashboard. We piloted the approach and refined it in an iterative process based on feedback from stakeholders and performed formal user testing with investigators and staff of two clinical trials.

RESULTS

The developed risk assessment comprises four domains (patient safety and rights, overall trial management, intervention management, trial data). An accompanying manual provides rationales and detailed instructions for the risk assessment. We programmed two trial dashboards tailored to one medical and one surgical RCT to manage identified trial risks based on daily exports of accumulating trial data. We made the code for a generic dashboard available on GitHub that can be adapted to individual trials.

CONCLUSIONS

The presented trial management approach with integrated monitoring enables user-friendly, continuous checking of critical elements of trial conduct to support trial teams in the academic setting. Further work is needed in order to show effectiveness of the dashboard in terms of safe trial conduct and successful completion of clinical trials.

Use of electronic health record best practice alerts to improve adherence to American Urological Association vesicoureteral reflux guidelines

PURPOSE

To evaluate the effect of a guidelines-based best practice alerts (BPA) in the electronic health record (EHR) on adherence to American Urological Association (AUA) vesicoureteral reflux (VUR) guidelines.

METHODS

Retrospective cohort study of patients aged 0-17 years old with primary VUR with an initial urology clinic visit the year before or year after BPA implementation was done. Primary outcomes include obtaining vital signs, urinalysis, and ultrasound at initial and 1-year follow-up visit.

RESULTS

We identified 123 patients with initial visits during the study period, 58 of whom returned for 1-year follow-up visits. Patients seen post-BPA were more likely to have height measured at initial visit than those seen pre-BPA (47.3% vs. 11.8%, p < 0.001). The majority of patients were screened with weight (98.3%) and ultrasound (87.9%) at 1-year follow-up both before and after BPA implementation. Neither blood pressure measurements (59.1% vs. 55.6%, p > 0.5) nor urinalysis orders (23.8% vs. 19.4%, p > 0.05) significantly increased post-BPA.

CONCLUSION

The use of an EHR-based BPA increased the likelihood of obtaining height measurements by clinic intake staff but did not significantly affect provider adherence to other practice guideline recommendations. Our findings suggest that BPA implementation alone is not sufficient to impact provider uptake of VUR guideline recommendations.

i-CLIMATE: a "clinical climate informatics" action framework to reduce environmental pollution from healthcare

Addressing environmental pollution and climate change is one of the biggest sociotechnical challenges of our time. While information technology has led to improvements in healthcare, it has also contributed to increased energy usage, destructive natural resource extraction, piles of e-waste, and increased greenhouse gases. We introduce a framework "Information technology-enabled Clinical cLimate InforMAtics acTions for the Environment" (i-CLIMATE) to illustrate how clinical informatics can help reduce healthcare's environmental pollution and climate-related impacts using 5 actionable components: (1) create a circular economy for health IT, (2) reduce energy consumption through smarter use of health IT, (3) support more environmentally friendly decision-making by clinicians and health administrators, (4) mobilize healthcare workforce environmental stewardship through informatics, and (5) Inform policies and regulations for change. We define Clinical Climate Informatics as a field that applies data, information, and knowledge management principles to operationalize components of the i-CLIMATE Framework.

Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts

Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.

Clinician collaboration to improve clinical decision support: the Clickbusters initiative

OBJECTIVE

We describe the Clickbusters initiative implemented at Vanderbilt University Medical Center (VUMC), which was designed to improve safety and quality and reduce burnout through the optimization of clinical decision support (CDS) alerts.

MATERIALS AND METHODS

We developed a 10-step Clickbusting process and implemented a program that included a curriculum, CDS alert inventory, oversight process, and gamification. We carried out two 3-month rounds of the Clickbusters program at VUMC. We completed descriptive analyses of the changes made to alerts during the process, and of alert firing rates before and after the program.

RESULTS

Prior to Clickbusters, VUMC had 419 CDS alerts in production, with 488 425 firings (42 982 interruptive) each week. After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%). In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS.

CONCLUSIONS

At VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks. The program also involved more users in the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.

A Framework for Evaluating Dashboards in Healthcare

In the era of 'information overload', effective information provision is essential for enabling rapid response and critical decision making. In making sense of diverse information sources, dashboards have become an indispensable tool, providing fast, effective, adaptable, and personalized access to information for professionals and the general public alike. However, these objectives place heavy requirements on dashboards as information systems in usability and effective design. Understanding these issues is challenging given the absence of consistent and comprehensive approaches to dashboard evaluation. In this article we systematically review literature on dashboard implementation in healthcare, where dashboards have been employed widely, and where there is widespread interest for improving the current state of the art, and subsequently analyse approaches taken towards evaluation. We draw upon consolidated dashboard literature and our own observations to introduce a general definition of dashboards which is more relevant to current trends, together with seven evaluation scenarios - task performance, behaviour change, interaction workflow, perceived engagement, potential utility, algorithm performance and system implementation. These scenarios distinguish different evaluation purposes which we illustrate through measurements, example studies, and common challenges in evaluation study design. We provide a breakdown of each evaluation scenario, and highlight some of the more subtle questions. We demonstrate the use of the proposed framework by a design study guided by this framework. We conclude by comparing this framework with existing literature, outlining a number of active discussion points and a set of dashboard evaluation best practices for the academic, clinical and software development communities alike.

Modulators Influencing Medication Alert Acceptance: An Explorative Review

OBJECTIVES

Clinical decision support systems (CDSSs) use alerts to enhance medication safety and reduce medication error rates. A major challenge of medication alerts is their low acceptance rate, limiting their potential benefit. A structured overview about modulators influencing alert acceptance is lacking. Therefore, we aimed to review and compile qualitative and quantitative modulators of alert acceptance and organize them in a comprehensive model.

METHODS

In accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline, a literature search in PubMed was started in February 2018 and continued until October 2021. From all included articles, qualitative and quantitative parameters and their impact on alert acceptance were extracted. Related parameters were then grouped into factors, allocated to superordinate determinants, and subsequently further allocated into five categories that were already known to influence alert acceptance.

RESULTS

Out of 539 articles, 60 were included. A total of 391 single parameters were extracted (e.g., patients' comorbidity) and grouped into 75 factors (e.g., comorbidity), and 25 determinants (e.g., complexity) were consequently assigned to the predefined five categories, i.e., CDSS, care provider, patient, setting, and involved drug. More than half of all factors were qualitatively assessed (n = 21) or quantitatively inconclusive (n = 19). Furthermore, 33 quantitative factors clearly influenced alert acceptance (positive correlation: e.g., alert type, patients' comorbidity; negative correlation: e.g., number of alerts per care provider, moment of alert display in the workflow). Two factors (alert frequency, laboratory value) showed contradictory effects, meaning that acceptance was significantly influenced both positively and negatively by these factors, depending on the study. Interventional studies have been performed for only 12 factors while all other factors were evaluated descriptively.

CONCLUSION

This review compiles modulators of alert acceptance distinguished by being studied quantitatively or qualitatively and indicates their effect magnitude whenever possible. Additionally, it describes how further research should be designed to comprehensively quantify the effect of alert modulators.

OUP accepted manuscript

Objectives

(1) Systematically review the literature on computerized audit and feedback (e-A&F) systems in healthcare. (2) Compare features of current systems against e-A&F best practices. (3) Generate hypotheses on how e-A&F systems may impact patient care and outcomes.

Methods

We searched MEDLINE (Ovid), EMBASE (Ovid), and CINAHL (Ebsco) databases to December 31, 2020. Two reviewers independently performed selection, extraction, and quality appraisal (Mixed Methods Appraisal Tool). System features were compared with 18 best practices derived from Clinical Performance Feedback Intervention Theory. We then used realist concepts to generate hypotheses on mechanisms of e-A&F impact. Results are reported in accordance with the PRISMA statement.

Results

Our search yielded 4301 unique articles. We included 88 studies evaluating 65 e-A&F systems, spanning a diverse range of clinical areas, including medical, surgical, general practice, etc. Systems adopted a median of 8 best practices (interquartile range 6–10), with 32 systems providing near real-time feedback data and 20 systems incorporating action planning. High-confidence hypotheses suggested that favorable e-A&F systems prompted specific actions, particularly enabled by timely and role-specific feedback (including patient lists and individual performance data) and embedded action plans, in order to improve system usage, care quality, and patient outcomes.

Conclusions

e-A&F systems continue to be developed for many clinical applications. Yet, several systems still lack basic features recommended by best practice, such as timely feedback and action planning. Systems should focus on actionability, by providing real-time data for feedback that is specific to user roles, with embedded action plans.

Protocol Registration

PROSPERO CRD42016048695.

Alert burden in pediatric hospitals: a cross-sectional analysis of six academic pediatric health systems using novel metrics

BACKGROUND

Excessive electronic health record (EHR) alerts reduce the salience of actionable alerts. Little is known about the frequency of interruptive alerts across health systems and how the choice of metric affects which users appear to have the highest alert burden.

OBJECTIVE

(1) Analyze alert burden by alert type, care setting, provider type, and individual provider across 6 pediatric health systems. (2) Compare alert burden using different metrics.

MATERIALS AND METHODS

We analyzed interruptive alert firings logged in EHR databases at 6 pediatric health systems from 2016-2019 using 4 metrics: (1) alerts per patient encounter, (2) alerts per inpatient-day, (3) alerts per 100 orders, and (4) alerts per unique clinician days (calendar days with at least 1 EHR log in the system). We assessed intra- and interinstitutional variation and how alert burden rankings differed based on the chosen metric.

RESULTS

Alert burden varied widely across institutions, ranging from 0.06 to 0.76 firings per encounter, 0.22 to 1.06 firings per inpatient-day, 0.98 to 17.42 per 100 orders, and 0.08 to 3.34 firings per clinician day logged in the EHR. Custom alerts accounted for the greatest burden at all 6 sites. The rank order of institutions by alert burden was similar regardless of which alert burden metric was chosen. Within institutions, the alert burden metric choice substantially affected which provider types and care settings appeared to experience the highest alert burden.

CONCLUSION

Estimates of the clinical areas with highest alert burden varied substantially by institution and based on the metric used.

Visual Analytics Dashboard Promises to Improve Hypertension Guideline Implementation

BACKGROUND

Primary care management of hypertension under new guidelines incorporates assessment of cardiovascular disease risk and commonly requires review of electronic health record (EHR) data. Visual analytics can streamline the review of complex data and may lessen the burden clinicians face using the EHR. This study sought to assess the utility of a visual analytics dashboard in addition to EHR in managing hypertension in a primary care setting.

METHODS

Primary care physicians within an urban, academic internal medicine clinic were tasked with performing 2 simulated patient encounters for hypertension management: the first using standard EHR, and the second using EHR paired with a visual dashboard. The dashboard included graphical blood pressure trends with guideline-directed targets, calculated atherosclerotic cardiovascular disease risk score, and relevant medications. Guideline-appropriate antihypertensive prescribing, correct target blood pressure goal, and total encounter time were assessed.

RESULTS

We evaluated 70 case simulations. Use of the dashboard with the EHR compared with use of the EHR alone was associated with greater adherence to prescribing guidelines (95% vs. 62%, P < 0.001) and more correct identification of blood pressure target (95% vs. 57%, P < 0.01). Total encounter time fell an average of 121 seconds (95% confidence interval 69-157 seconds, P < 0.001) in encounters that used the dashboard combined with the EHR.

CONCLUSIONS

The integration of a hypertension-specific visual analytics dashboard with EHR demonstrates the potential to reduce time and improve hypertension guideline implementation. Further widespread testing in clinical practice is warranted.

VISEMURE: A Visual Analytics System for Making Sense of Multimorbidity Using Electronic Medical Record Data

Multimorbidity is a growing healthcare problem, especially for aging populations. Traditional single disease-centric approaches are not suitable for multimorbidity, and a holistic framework is required for health research and for enhancing patient care. Patterns of multimorbidity within populations are complex and difficult to communicate with static visualization techniques such as tables and charts. We designed a visual analytics system called VISEMURE that facilitates making sense of data collected from patients with multimorbidity. With VISEMURE, users can interactively create different subsets of electronic medical record data to investigate multimorbidity within different subsets of patients with pre-existing chronic diseases. It also allows the creation of groups of patients based on age, gender, and socioeconomic status for investigation. VISEMURE can use a range of statistical and machine learning techniques and can integrate them seamlessly to compute prevalence and correlation estimates for selected diseases. It presents results using interactive visualizations to help healthcare researchers in making sense of multimorbidity. Using a case study, we demonstrate how VISEMURE can be used to explore the high-dimensional joint distribution of random variables that describes the multimorbidity present in a patient population.

A Tool for Evaluating Medication Alerting Systems: Development and Initial Assessment

BACKGROUND

It is well known that recommendations from electronic medication alerts are seldom accepted or acted on by users. Key factors affecting the effectiveness of medication alerts include system usability and alert design. Thus, human factors principles that apply knowledge of human capabilities and limitations are increasingly used in the design of health technology to improve the usability of systems.

OBJECTIVE

This study aims to evaluate a newly developed evidence-based self-assessment tool that allows the valid and reliable evaluation of computerized medication alerting systems. This tool was developed to be used by hospital staff with detailed knowledge of their hospital's computerized provider order entry system and alerts to identify and address potential system deficiencies. In this initial assessment, we aim to determine whether the items in the tool can measure compliance of medication alerting systems with human factors principles of design, the tool can be consistently used by multiple users to assess the same system, and the items are easy to understand and perceived to be useful for assessing medication alerting systems.

METHODS

The Tool for Evaluating Medication Alerting Systems (TEMAS) was developed based on human factors design principles and consisted of 66 items. In total, 18 staff members recruited across 6 hospitals used the TEMAS to assess their medication alerting systems. Data collected from participant assessments were used to evaluate the validity, reliability, and usability of the TEMAS. Validity was assessed by comparing the results of the TEMAS with those of prior in-house evaluations. Reliability was measured using Krippendorff α to determine agreement among assessors. A 7-item survey was used to determine usability.

RESULTS

The participants reported mostly negative (n=8) and neutral (n=7) perceptions of alerts in their medication alerting system. However, the validity of the TEMAS could not be directly tested, as participants were unaware of any results from prior in-house evaluations. The reliability of the TEMAS, as measured by Krippendorff α, was low to moderate (range 0.26-0.46); however, participant feedback suggests that individuals' knowledge of the system varied according to their professional background. In terms of usability, 61% (11/18) of participants reported that the TEMAS items were generally easy to understand; however, participants suggested the revision of 22 items to improve clarity.

CONCLUSIONS

This initial assessment of the TEMAS allowed the identification of its components that required modification to improve usability and usefulness. It also revealed that for the TEMAS to be effective in facilitating a comprehensive assessment of a medication alerting system, it should be completed by a multidisciplinary team of hospital staff from both clinical and technical backgrounds to maximize their knowledge of systems.

Visual Analytics for Electronic Health Records: A Review

The increasing use of electronic health record (EHR)-based systems has led to the generation of clinical data at an unprecedented rate, which produces an untapped resource for healthcare experts to improve the quality of care. Despite the growing demand for adopting EHRs, the large amount of clinical data has made some analytical and cognitive processes more challenging. The emergence of a type of computational system called visual analytics has the potential to handle information overload challenges in EHRs by integrating analytics techniques with interactive visualizations. In recent years, several EHR-based visual analytics systems have been developed to fulfill healthcare experts’ computational and cognitive demands. In this paper, we conduct a systematic literature review to present the research papers that describe the design of EHR-based visual analytics systems and provide a brief overview of 22 systems that met the selection criteria. We identify and explain the key dimensions of the EHR-based visual analytics design space, including visual analytics tasks, analytics, visualizations, and interactions. We evaluate the systems using the selected dimensions and identify the gaps and areas with little prior work.

Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review

OBJECTIVE

Alert fatigue limits the effectiveness of medication safety alerts, a type of computerized clinical decision support (CDS). Researchers have suggested alternative interactive designs, as well as tailoring alerts to clinical roles. As examples, alerts may be tiered to convey risk, and certain alerts may be sent to pharmacists. We aimed to evaluate which variants elicit less alert fatigue.

MATERIALS AND METHODS

We searched for articles published between 2007 and 2017 using the PubMed, Embase, CINAHL, and Cochrane databases. We included articles documenting peer-reviewed empirical research that described the interactive design of a CDS system, to which clinical role it was presented, and how often prescribers accepted the resultant advice. Next, we compared the acceptance rates of conventional CDS-presenting prescribers with interruptive modal dialogs (ie, "pop-ups")-with alternative designs, such as role-tailored alerts.

RESULTS

Of 1011 articles returned by the search, we included 39. We found different methods for measuring acceptance rates; these produced incomparable results. The most common type of CDS-in which modals interrupted prescribers-was accepted the least often. Tiering by risk, providing shortcuts for common corrections, requiring a reason to override, and tailoring CDS to match the roles of pharmacists and prescribers were the most common alternatives. Only 1 alternative appeared to increase prescriber acceptance: role tailoring. Possible reasons include the importance of etiquette in delivering advice, the cognitive benefits of delegation, and the difficulties of computing "relevance."

CONCLUSIONS

Alert fatigue may be mitigated by redesigning the interactive behavior of CDS and tailoring CDS to clinical roles. Further research is needed to develop alternative designs, and to standardize measurement methods to enable meta-analyses.

Giving Your Electronic Health Record a Checkup After COVID-19: A Practical Framework for Reviewing Clinical Decision Support in Light of the Telemedicine Expansion

BACKGROUND

The transformation of health care during COVID-19, with the rapid expansion of telemedicine visits, presents new challenges to chronic care and preventive health providers. Clinical decision support (CDS) is critically important to chronic care providers, and CDS malfunction is common during times of change. It is essential to regularly reassess an organization's ambulatory CDS program to maintain care quality. This is especially true after an immense change, like the COVID-19 telemedicine expansion.

OBJECTIVE

Our objective is to reassess the ambulatory CDS program at a large academic medical center in light of telemedicine's expansion in response to the COVID-19 pandemic.

METHODS

Our clinical informatics team devised a practical framework for an intrapandemic ambulatory CDS assessment focused on the impact of the telemedicine expansion. This assessment began with a quantitative analysis comparing CDS alert performance in the context of in-person and telemedicine visits. Board-certified physician informaticists then completed a formal workflow review of alerts with inferior performance in telemedicine visits. Informaticists then reported on themes and optimization opportunities through the existing CDS governance structure.

RESULTS

Our assessment revealed that 10 of our top 40 alerts by volume were not firing as expected in telemedicine visits. In 3 of the top 5 alerts, providers were significantly less likely to take action in telemedicine when compared to office visits. Cumulatively, alerts in telemedicine encounters had an action taken rate of 5.3% (3257/64,938) compared to 8.3% (19,427/233,636) for office visits. Observations from a clinical informaticist workflow review included the following: (1) Telemedicine visits have different workflows than office visits. Some alerts developed for the office were not appearing at the optimal time in the telemedicine workflow. (2) Missing clinical data is a common reason for the decreased alert firing seen in telemedicine visits. (3) Remote patient monitoring and patient-reported clinical data entered through the portal could replace data collection usually completed in the office by a medical assistant or registered nurse.

CONCLUSIONS

In a large academic medical center at the pandemic epicenter, an intrapandemic ambulatory CDS assessment revealed clinically significant CDS malfunctions that highlight the importance of reassessing ambulatory CDS performance after the telemedicine expansion.

Optimizing clinical decision support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals

OBJECTIVE

To identify and summarize the current internal governance processes adopted by hospitals, as reported in the literature, for selecting, optimizing, and evaluating clinical decision support (CDS) alerts in order to identify effective approaches.

MATERIALS AND METHODS

Databases (Medline, Embase, CINAHL, Scopus, Web of Science, IEEE Xplore Digital Library, CADTH, and WorldCat) were searched to identify relevant papers published from January 2010 to April 2020. All paper types published in English that reported governance processes for selecting and/or optimizing CDS alerts in hospitals were included.

RESULTS

Eight papers were included in the review. Seven papers focused specifically on medication-related CDS alerts. All papers described the use of a multidisciplinary committee to optimize alerts. Other strategies included the use of clinician feedback, alert data, literature and drug references, and a visual dashboard. Six of the 8 papers reported evaluations of their CDS alert modifications following the adoption of optimization strategies, and of these, 5 reported a reduction in alert rate.

CONCLUSIONS

A multidisciplinary committee, often in combination with other approaches, was the most frequent strategy reported by hospitals to optimize their CDS alerts. Due to the limited number of published processes, variation in system changes, and evaluation results, we were unable to compare the effectiveness of different strategies, although employing multiple strategies appears to be an effective approach for reducing CDS alert numbers. We recommend hospitals report on descriptions and evaluations of governance processes to enable identification of effective strategies for optimization of CDS alerts in hospitals.

Forecasting the future of library and information science and its sub-fields

Forecasting is one of the methods applied in many studies in the library and information science (LIS) field for numerous purposes, from making predictions of the next Nobel laureates to potential technological developments. This study sought to draw a picture for the future of the LIS field and its sub-fields by analysing 97 years of publication and citation patterns. The core Web of Science indexes were used as the data source, and 123,742 articles were examined in-depth for time series analysis. The social network analysis method was used for sub-field classification. The field was divided into four sub-fields: (1) librarianship and law librarianship, (2) health information in LIS, (3) scientometrics and information retrieval and (4) management and information systems. The results of the study show that the LIS sub-fields are completely different from each other in terms of their publication and citation patterns, and all the sub-fields have different dynamics. Furthermore, the number of publications, references and citations will increase significantly in the future. It is expected that more scholars will work together. The future subjects of the LIS field show astonishing diversity from fake news to predatory journals, open government, e-learning and electronic health records. However, the findings prove that publish or perish culture will shape the field. Therefore, it is important to go beyond numbers. It can only be achieved by understanding publication and citation patterns of the field and developing research policies accordingly.

Reducing Alert Fatigue by Sharing Low-Level Alerts With Patients and Enhancing Collaborative Decision Making Using Blockchain Technology: Scoping Review and Proposed Framework (MedAlert)

Background

Clinical decision support (CDS) is a tool that helps clinicians in decision making by generating clinical alerts to supplement their previous knowledge and experience. However, CDS generates a high volume of irrelevant alerts, resulting in alert fatigue among clinicians. Alert fatigue is the mental state of alerts consuming too much time and mental energy, which often results in relevant alerts being overridden unjustifiably, along with clinically irrelevant ones. Consequently, clinicians become less responsive to important alerts, which opens the door to medication errors.

Objective

This study aims to explore how a blockchain-based solution can reduce alert fatigue through collaborative alert sharing in the health sector, thus improving overall health care quality for both patients and clinicians.

Methods

We have designed a 4-step approach to answer this research question. First, we identified five potential challenges based on the published literature through a scoping review. Second, a framework is designed to reduce alert fatigue by addressing the identified challenges with different digital components. Third, an evaluation is made by comparing MedAlert with other proposed solutions. Finally, the limitations and future work are also discussed.

Results

Of the 341 academic papers collected, 8 were selected and analyzed. MedAlert securely distributes low-level (nonlife-threatening) clinical alerts to patients, enabling a collaborative clinical decision. Among the solutions in our framework, Hyperledger (private permissioned blockchain) and BankID (federated digital identity management) have been selected to overcome challenges such as data integrity, user identity, and privacy issues.

Conclusions

MedAlert can reduce alert fatigue by attracting the attention of patients and clinicians, instead of solely reducing the total number of alerts. MedAlert offers other advantages, such as ensuring a higher degree of patient privacy and faster transaction times compared with other frameworks. This framework may not be suitable for elderly patients who are not technology savvy or in-patients. Future work in validating this framework based on real health care scenarios is needed to provide the performance evaluations of MedAlert and thus gain support for the better development of this idea.

Utilizing open-source platforms to build and deploy interactive patient-reported quality of life tracking tools for monitoring protocol adherence

PURPOSE

Tracking patient-reported outcomes (PROs) and quality-of-life response rates is essential for clinical trials. Historically, rates are monitored through scheduled reports, which can require gathering, merging, and cleaning data from multiple databases. At the end of this process, if gaps are found, new data are entered and the cycle repeats, leaving a trail of reports that are not up-to-date or immediately accessible to the investigator. The financial and person-hour cost of utilizing clinical research staff for this purpose is impractical. Online dashboards are continuously updated to monitor data, providing on-demand access to promote successful research.

METHODS

Dashboard implementation utilizes R, an open-source statistical programming language, RMarkdown, a markup language, Flexdashboard, which creates structural elements, and Shiny, allowing investigators the ability to interact with data within the dashboard. By leveraging these four elements, powerful, cost-effective interactive dashboards can be built.

RESULTS

Numerous dashboards have been utilized to identify potentially missing data and increase protocol adherence. Immediate patient consultation can occur to retrieve protocol-related forms, reducing research staff and patient burden while improving trial effectiveness. Dashboards can monitor PROs, enrollment, demographics, toxicity, and biomarker data, clinical outcomes, and implemented predictive models, creating a single hub for on-demand clinical trial monitoring.

CONCLUSION

By employing a set of freely available tools, the burden of utilizing study staff to continuously monitor trials is greatly reduced. These tools allow users to rapidly build and deploy dynamic dashboards capable of meeting the research needs of any investigator while limiting missing data through simplified monitoring of protocol adherence.

Feedback at the Point of Care to Decrease Medication Alert Rates in an Electronic Health Record

Frequently overridden alerts in the electronic health record can highlight alerts that may need revision. This method is a way of fine-tuning clinical decision support. We evaluated the feasibility of a complementary, yet different method that directly involved pediatric emergency department (PED) providers in identifying additional medication alerts that were potentially incorrect or intrusive. We then evaluated the effect subsequent resulting modifications had on alert salience.

METHODS

We performed a prospective, interventional study over 34 months (March 6, 2014, to December 31, 2016) in the PED. We implemented a passive alert feedback mechanism by enhancing the native electronic health record functionality on alert reviews. End-users flagged potentially incorrect/bothersome alerts for review by the study's team. The alerts were updated when clinically appropriate and trends of the impact were evaluated.

RESULTS

More than 200 alerts were reported from both inside and outside the PED, suggesting an intuitive approach. On average, we processed 4 reviews per week from the PED, with attending physicians as major contributors. The general trend of the impact of these changes seems favorable.

DISCUSSION

The implementation of the review mechanism for user-selected alerts was intuitive and sustainable and seems to be able to detect alerts that are bothersome to the end-users. The method should be run in parallel with the traditional data-driven approach to support capturing of inaccurate alerts.

CONCLUSIONS

User-centered, context-specific alert feedback can be used for selecting suboptimal, interruptive medication alerts.

Visual Analytics for Dimension Reduction and Cluster Analysis of High Dimensional Electronic Health Records

Recent advancement in EHR-based (Electronic Health Record) systems has resulted in producing data at an unprecedented rate. The complex, growing, and high-dimensional data available in EHRs creates great opportunities for machine learning techniques such as clustering. Cluster analysis often requires dimension reduction to achieve efficient processing time and mitigate the curse of dimensionality. Given a wide range of techniques for dimension reduction and cluster analysis, it is not straightforward to identify which combination of techniques from both families leads to the desired result. The ability to derive useful and precise insights from EHRs requires a deeper understanding of the data, intermediary results, configuration parameters, and analysis processes. Although these tasks are often tackled separately in existing studies, we present a visual analytics (VA) system, called Visual Analytics for Cluster Analysis and Dimension Reduction of High Dimensional Electronic Health Records (VALENCIA), to address the challenges of high-dimensional EHRs in a single system. VALENCIA brings a wide range of cluster analysis and dimension reduction techniques, integrate them seamlessly, and make them accessible to users through interactive visualizations. It offers a balanced distribution of processing load between users and the system to facilitate the performance of high-level cognitive tasks in such a way that would be difficult without the aid of a VA system. Through a real case study, we have demonstrated how VALENCIA can be used to analyze the healthcare administrative dataset stored at ICES. This research also highlights what needs to be considered in the future when developing VA systems that are designed to derive deep and novel insights into EHRs.

Development, Implementation, and Evaluation of a Personalized Machine Learning Algorithm for Clinical Decision Support: Case Study With Shingles Vaccination

BACKGROUND

Although clinical decision support (CDS) alerts are effective reminders of best practices, their effectiveness is blunted by clinicians who fail to respond to an overabundance of inappropriate alerts. An electronic health record (EHR)-integrated machine learning (ML) algorithm is a potentially powerful tool to increase the signal-to-noise ratio of CDS alerts and positively impact the clinician's interaction with these alerts in general.

OBJECTIVE

This study aimed to describe the development and implementation of an ML-based signal-to-noise optimization system (SmartCDS) to increase the signal of alerts by decreasing the volume of low-value herpes zoster (shingles) vaccination alerts.

METHODS

We built and deployed SmartCDS, which builds personalized user activity profiles to suppress shingles vaccination alerts unlikely to yield a clinician's interaction. We extracted all records of shingles alerts from January 2017 to March 2019 from our EHR system, including 327,737 encounters, 780 providers, and 144,438 patients.

RESULTS

During the 6 weeks of pilot deployment, the SmartCDS system suppressed an average of 43.67% (15,425/35,315) potential shingles alerts (appointments) and maintained stable counts of weekly shingles vaccination orders (326.3 with system active vs 331.3 in the control group; P=.38) and weekly user-alert interactions (1118.3 with system active vs 1166.3 in the control group; P=.20).

CONCLUSIONS

All key statistics remained stable while the system was turned on. Although the results are promising, the characteristics of the system can be subject to future data shifts, which require automated logging and monitoring. We demonstrated that an automated, ML-based method and data architecture to suppress alerts are feasible without detriment to overall order rates. This work is the first alert suppression ML-based model deployed in practice and serves as foundational work in encounter-level customization of alert display to maximize effectiveness.

Multiple Regression Analysis and Frequent Itemset Mining of Electronic Medical Records: A Visual Analytics Approach Using VISA_M3R3

Medication-induced acute kidney injury (AKI) is a well-known problem in clinical medicine. This paper reports the first development of a visual analytics (VA) system that examines how different medications associate with AKI. In this paper, we introduce and describe VISA_M3R3, a VA system designed to assist healthcare researchers in identifying medications and medication combinations that associate with a higher risk of AKI using electronic medical records (EMRs). By integrating multiple regression models, frequent itemset mining, data visualization, and human-data interaction mechanisms, VISA_M3R3 allows users to explore complex relationships between medications and AKI in such a way that would be difficult or sometimes even impossible without the help of a VA system. Through an analysis of 595 medications using VISA_M3R3, we have identified 55 AKI-inducing medications, 24,212 frequent medication groups, and 78 medication groups that are associated with AKI. The purpose of this paper is to demonstrate the usefulness of VISA_M3R3 in the investigation of medication-induced AKI in particular and other clinical problems in general. Furthermore, this research highlights what needs to be considered in the future when designing VA systems that are intended to support gaining novel and deep insights into massive existing EMRs.

Reducing Interruptive Alert Burden Using Quality Improvement Methodology

BACKGROUND

Increased adoption of electronic health records (EHR) with integrated clinical decision support (CDS) systems has reduced some sources of error but has led to unintended consequences including alert fatigue. The "pop-up" or interruptive alert is often employed as it requires providers to acknowledge receipt of an alert by taking an action despite the potential negative effects of workflow interruption. We noted a persistent upward trend of interruptive alerts at our institution and increasing requests for new interruptive alerts.

OBJECTIVES

Using Institute for Healthcare Improvement (IHI) quality improvement (QI) methodology, the primary objective was to reduce the total volume of interruptive alerts received by providers.

METHODS

We created an interactive dashboard for baseline alert data and to monitor frequency and outcomes of alerts as well as to prioritize interventions. A key driver diagram was developed with a specific aim to decrease the number of interruptive alerts from a baseline of 7,250 to 4,700 per week (35%) over 6 months. Interventions focused on the following key drivers: appropriate alert display within workflow, clear alert content, alert governance and standardization, user feedback regarding overrides, and respect for user knowledge.

RESULTS

A total of 25 unique alerts accounted for 90% of the total interruptive alert volume. By focusing on these 25 alerts, we reduced interruptive alerts from 7,250 to 4,400 per week.

CONCLUSION

Systematic and structured improvements to interruptive alerts can lead to overall reduced interruptive alert burden. Using QI methods to prioritize our interventions allowed us to maximize our impact. Further evaluation should be done on the effects of reduced interruptive alerts on patient care outcomes, usability heuristics on cognitive burden, and direct feedback mechanisms on alert utility.

Reducing Alert Burden in Electronic Health Records: State of the Art Recommendations from Four Health Systems

BACKGROUND

Electronic health record (EHR) alert fatigue, while widely recognized as a concern nationally, lacks a corresponding comprehensive mitigation plan.

OBJECTIVES

The goal of this manuscript is to provide practical guidance to clinical informaticists and other health care leaders who are considering creating a program to manage EHR alerts.

METHODS

This manuscript synthesizes several approaches and recommendations for better alert management derived from four U.S. health care institutions that presented their experiences and recommendations at the American Medical Informatics Association 2019 Clinical Informatics Conference in Atlanta, Georgia, United States. The assembled health care institution leaders represent academic, pediatric, community, and specialized care domains. We describe governance and management, structural concepts and components, and human-computer interactions with alerts, and make recommendations regarding these domains based on our experience supplemented with literature review. This paper focuses on alerts that impact bedside clinicians.

RESULTS

The manuscript addresses the range of considerations relevant to alert management including a summary of the background literature about alerts, alert governance, alert metrics, starting an alert management program, approaches to evaluating alerts prior to deployment, and optimization of existing alerts. The manuscript includes examples of alert optimization successes at two of the represented institutions. In addition, we review limitations on the ability to evaluate alerts in the current state and identify opportunities for further scholarship.

CONCLUSION

Ultimately, alert management programs must strive to meet common goals of improving patient care, while at the same time decreasing the alert burden on clinicians. In so doing, organizations have an opportunity to promote the wellness of patients, clinicians, and EHRs themselves.

Modelling the interactive behaviour of users with a medication safety dashboard in a primary care setting

OBJECTIVE

To characterise the use of an electronic medication safety dashboard by exploring and contrasting interactions from primary users (i.e. pharmacists) who were leading the intervention and secondary users (i.e. non-pharmacist staff) who used the dashboard to engage in safe prescribing practices.

MATERIALS AND METHODS

We conducted a 10-month observational study in which 35 health professionals used an instrumented medication safety dashboard for audit and feedback purposes in clinical practice as part of a wider intervention study. We modelled user interaction by computing features representing exploration and dwell time through user interface events that were logged on a remote database. We applied supervised learning algorithms to classify primary against secondary users.

RESULTS

We observed values for accuracy above 0.8, indicating that 80% of the time we were able to distinguish a primary user from a secondary user. In particular, the Multilayer Perceptron (MLP) yielded the highest values of precision (0.88), recall (0.86) and F-measure (0.86). The behaviour of primary users was distinctive in that they spent less time between mouse clicks (lower dwell time) on the screens showing the overview of the practice and trends. Secondary users exhibited a higher dwell time and more visual search activity (higher exploration) on the screens displaying patients at risk and visualisations.

DISCUSSION AND CONCLUSION

We were able to distinguish the interactive behaviour of primary and secondary users of a medication safety dashboard in primary care using timestamped mouse events. Primary users were more competent on population health monitoring activities, while secondary users struggled on activities involving a detailed breakdown of the safety of patients. Informed by these findings, we propose workflows that group these activities and adaptive nudges to increase user engagement.

Automated analysis of free-text comments and dashboard representations in patient experience surveys: a multimethod co-design study

Background

Patient experience surveys (PESs) often include informative free-text comments, but with no way of systematically, efficiently and usefully analysing and reporting these. The National Cancer Patient Experience Survey (CPES), used to model the approach reported here, generates > 70,000 free-text comments annually.

Main aim

To improve the use and usefulness of PES free-text comments in driving health service changes that improve the patient experience.

Secondary aims

(1) To structure CPES free-text comments using rule-based information retrieval (IR) (‘text engineering’), drawing on health-care domain-specific gazetteers of terms, with in-built transferability to other surveys and conditions; (2) to display the results usefully for health-care professionals, in a digital toolkit dashboard display that drills down to the original free text; (3) to explore the usefulness of interdisciplinary mixed stakeholder co-design and consensus-forming approaches in technology development, ensuring that outputs have meaning for all; and (4) to explore the usefulness of Normalisation Process Theory (NPT) in structuring outputs for implementation and sustainability.

Design

A scoping review, rapid review and surveys with stakeholders in health care (patients, carers, health-care providers, commissioners, policy-makers and charities) explored clinical dashboard design/patient experience themes. The findings informed the rules for the draft rule-based IR [developed using half of the 2013 Wales CPES (WCPES) data set] and prototype toolkit dashboards summarising PES data. These were refined following mixed stakeholder, concept-mapping workshops and interviews, which were structured to enable consensus-forming ‘co-design’ work. IR validation used the second half of the WCPES, with comparison against its manual analysis; transferability was tested using further health-care data sets. A discrete choice experiment (DCE) explored which toolkit features were preferred by health-care professionals, with a simple cost–benefit analysis. Structured walk-throughs with NHS managers in Wessex, London and Leeds explored usability and general implementation into practice.

Key outcomes

A taxonomy of ranked PES themes, a checklist of key features recommended for digital clinical toolkits, rule-based IR validation and transferability scores, usability, and goal-oriented, cost–benefit and marketability results. The secondary outputs were a survey, scoping and rapid review findings, and concordance and discordance between stakeholders and methods.

Results

(1) The surveys, rapid review and workshops showed that stakeholders differed in their understandings of the patient experience and priorities for change, but that they reached consensus on a shortlist of 19 themes; six were considered to be core; (2) the scoping review and one survey explored the clinical toolkit design, emphasising that such toolkits should be quick and easy to use, and embedded in workflows; the workshop discussions, the DCE and the walk-throughs confirmed this and foregrounded other features to form the toolkit design checklist; and (3) the rule-based IR, developed using noun and verb phrases and lookup gazetteers, was 86% accurate on the WCPES, but needs modification to improve this and to be accurate with other data sets. The DCE and the walk-through suggest that the toolkit would be well accepted, with a favourable cost–benefit ratio, if implemented into practice with appropriate infrastructure support.

Limitations

Small participant numbers and sampling bias across component studies. The scoping review studies mostly used top-down approaches and focused on professional dashboards. The rapid review of themes had limited scope, with no second reviewer. The IR needs further refinement, especially for transferability. New governance restrictions further limit immediate use.

Conclusions

Using a multidisciplinary, mixed stakeholder, use of co-design, proof of concept was shown for an automated display of patient experience free-text comments in a way that could drive health-care improvements in real time. The approach is easily modified for transferable application.

Future work

Further exploration is needed of implementation into practice, transferable uses and technology development co-design approaches.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Optimizing Drug-Drug Interaction Alerts Using a Multidimensional Approach

OBJECTIVES

Excessive alerts are a common concern associated with clinical decision support systems that monitor drug-drug interactions (DDIs). To reduce the number of low-value interruptive DDI alerts at our hospital, we implemented an iterative, multidimensional quality improvement effort, which included an interdisciplinary advisory group, alert metrics, and measurement of perceived clinical value.

METHODS

Alert data analysis indicated that DDIs were the most common interruptive medication alert. An interdisciplinary alert advisory group was formed to provide expert advice and oversight for alert refinement and ongoing review of alert data. Alert data were categorized into drug classes and analyzed to identify DDI alerts for refinement. Refinement strategies included alert suppression and modification of alerts to be contextually aware.

RESULTS

On the basis of historical analysis of classified DDI alerts, 26 alert refinements were implemented, representing 47% of all alerts. Alert refinement efforts resulted in the following substantial decreases in the number of interruptive DDI alerts: 40% for all clinicians (22.9-14 per 100 orders) and as high as 82% for attending physicians (6.5-1.2 per 100 orders). Two patient safety events related to alert refinements were reported during the project period.

CONCLUSIONS

Our quality improvement effort refined 47% of all DDI alerts that were firing during historical analysis, significantly reduced the number of DDI alerts in a 54-week period, and established a model for sustained alert refinements.

Timely Interventions for Children with ADHD through Web-Based Monitoring Algorithms

The aim of this study was to evaluate an automated trigger algorithm designed to detect potentially adverse events in children with Attention-Deficit/Hyperactivity Disorder (ADHD), who were monitored remotely between visits. We embedded a trigger algorithm derived from parent-reported ADHD rating scales within an electronic patient monitoring system. We categorized clinicians’ alert resolution outcomes and compared Vanderbilt ADHD rating scale scores between patients who did or did not have triggered alerts. A total of 146 out of 1738 parent reports (8%) triggered alerts for 98 patients. One hundred and eleven alerts (76%) required immediate clinician review. Nurses successfully contacted parents for 68 (61%) of actionable alerts; 46% (31/68) led to a change in care plan prior to the next scheduled appointment. Compared to patients without alerts, patients with alerts demonstrated worsened ADHD severity (β = 5.8, 95% CI: 3.5–8.1 [p < 0.001] within 90 days prior to an alert. The trigger algorithm facilitated timely changes in the care plan in between face-to-face visits.

[Electronic decision support to promote medication safety]

Because of its inherent complexity, it is a considerable challenge to tailor drug treatment to a prevalent disease and its subgroups, which are increasingly defined by genomic variability (personalized medicine) and require consideration of context information such as co-morbidity, co-medication, patient preferences, and the specific characteristics of the healthcare sector. Thus, optimum treatment decisions might not be taken intuitively any longer, because decisions must be made both rapidly and increasingly based on analyses of complex relations of numerous variables that exceed the processing performance of a human brain. Hence, computer support is indispensable to ensure error-free high-performance medicine. A key step in computer-supported medication safety is to implement a computerized physician order entry (CPOE) system that compiles a patient's medication in a structured and coded format enabling the link to clinical decision support (CDS) systems. Implementing a CPOE is hence a strategic step for a hospital, which is crucial to exhaustingly and consistently prevent medication errors. Thereby, the best performance of a CPOE is achieved if it is deeply integrated into an electronic patient record thus enabling access to relevant patient information, which again has to be structured to allow processing. To efficiently support drug treatment, CDS systems must fulfill high-quality standards with regard to underlying data, integration, and user-interaction to ensure that they support but do not impede the provision of care.

Design and Implementation of a Visual Analytics Electronic Antibiogram within an Electronic Health Record System at a Tertiary Pediatric Hospital

BACKGROUND

Hospitals use antibiograms to guide optimal empiric antibiotic therapy, reduce inappropriate antibiotic usage, and identify areas requiring intervention by antimicrobial stewardship programs. Creating a hospital antibiogram is a time-consuming manual process that is typically performed annually.

OBJECTIVE

We aimed to apply visual analytics software to electronic health record (EHR) data to build an automated, electronic antibiogram ("e-antibiogram") that adheres to national guidelines and contains filters for patient characteristics, thereby providing access to detailed, clinically relevant, and up-to-date antibiotic susceptibility data.

METHODS

We used visual analytics software to develop a secure, EHR-linked, condition- and patient-specific e-antibiogram that supplies susceptibility maps for organisms and antibiotics in a comprehensive report that is updated on a monthly basis. Antimicrobial susceptibility data were grouped into nine clinical scenarios according to the specimen source, hospital unit, and infection type. We implemented the e-antibiogram within the EHR system at Children's Hospital of Philadelphia, a tertiary pediatric hospital and analyzed e-antibiogram access sessions from March 2016 to March 2017.

RESULTS

The e-antibiogram was implemented in the EHR with over 6,000 inpatient, 4,500 outpatient, and 3,900 emergency department isolates. The e-antibiogram provides access to rolling 12-month pathogen and susceptibility data that is updated on a monthly basis. E-antibiogram access sessions increased from an average of 261 sessions per month during the first 3 months of the study to 345 sessions per month during the final 3 months.

CONCLUSION

An e-antibiogram that was built and is updated using EHR data and adheres to national guidelines is a feasible replacement for an annual, static, manually compiled antibiogram. Future research will examine the impact of the e-antibiogram on antibiotic prescribing patterns.

Cytochrome P450 interactions are common and consequential in Massachusetts hospital discharges

Despite the recognition that drug-drug interactions contribute substantially to preventable health-care costs, the prevalence of such interactions related to the cytochrome P450 system in clinical practice remains poorly characterized. This study drew retrospective hospital discharge cohorts from a large health claims data set and a large health system data set. For every hospital discharge, frequency of co-occurrence of substrates and inducers or inhibitors at cytochrome P450 2D6, 2C19, 3A4 and 1A2 were determined. A total of 124 520 individuals in the state of Massachusetts (health claims cohort) and 77 026 individuals in two large academic medical centers (electronic health record (EHR) cohort) were examined. In the claims cohort, 35 157 (28.2%) exhibited at least one CYP450 drug-drug interaction at hospital discharge, whereas in the EHR cohort, 36 750 (47.7%) had at least one interaction. The most commonly affected CYP450 systems were 2C19 and 2D6, with putative interactions observed in at least 10% of individuals at discharge in each cohort. Odds of hospital readmission within 90 days among those discharged with at least one interaction were 10-16% greater, with mean health-care cost $574/month greater over the subsequent year, after adjusting for age, sex, insurance type, total number of medications prescribed, Charlson comorbidity score and presence or absence of a psychiatric diagnosis. These two distinct clinical data types show that CYP450 drug-drug interactions are prevalent and associated with greater probability of early hospital readmission and greater health-care cost, despite the widespread availability and application of drug-drug interaction checking software.

The Effects of Medication Alerts on Prescriber Response in a Pediatric Hospital

OBJECTIVE

More than 70% of hospitals in the United States have electronic health records (EHRs). Clinical decision support (CDS) presents clinicians with electronic alerts during the course of patient care; however, alert fatigue can influence a provider's response to any EHR alert. The primary goal was to evaluate the effects of alert burden on user response to the alerts.

METHODS

We performed a retrospective study of medication alerts over a 24-month period (1/2013-12/2014) in a large pediatric academic medical center. The institutional review board approved this study. The primary outcome measure was alert salience, a measure of whether or not the prescriber took any corrective action on the order that generated an alert. We estimated the ideal number of alerts to maximize salience. Salience rates were examined for providers at each training level, by day of week, and time of day through logistic regressions.

RESULTS

While salience never exceeded 38%, 49 alerts/day were associated with maximal salience in our dataset. The time of day an order was placed was associated with alert salience (maximal salience 2am). The day of the week was also associated with alert salience (maximal salience on Wednesday). Provider role did not have an impact on salience.

CONCLUSION

Alert burden plays a role in influencing provider response to medication alerts. An increased number of alerts a provider saw during a one-day period did not directly lead to decreased response to alerts. Given the multiple factors influencing the response to alerts, efforts focused solely on burden are not likely to be effective.

Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system

BACKGROUND

Although alert fatigue is blamed for high override rates in contemporary clinical decision support systems, the concept of alert fatigue is poorly defined. We tested hypotheses arising from two possible alert fatigue mechanisms: (A) cognitive overload associated with amount of work, complexity of work, and effort distinguishing informative from uninformative alerts, and (B) desensitization from repeated exposure to the same alert over time.

METHODS

Retrospective cohort study using electronic health record data (both drug alerts and clinical practice reminders) from January 2010 through June 2013 from 112 ambulatory primary care clinicians. The cognitive overload hypotheses were that alert acceptance would be lower with higher workload (number of encounters, number of patients), higher work complexity (patient comorbidity, alerts per encounter), and more alerts low in informational value (repeated alerts for the same patient in the same year). The desensitization hypothesis was that, for newly deployed alerts, acceptance rates would decline after an initial peak.

RESULTS

On average, one-quarter of drug alerts received by a primary care clinician, and one-third of clinical reminders, were repeats for the same patient within the same year. Alert acceptance was associated with work complexity and repeated alerts, but not with the amount of work. Likelihood of reminder acceptance dropped by 30% for each additional reminder received per encounter, and by 10% for each five percentage point increase in proportion of repeated reminders. The newly deployed reminders did not show a pattern of declining response rates over time, which would have been consistent with desensitization. Interestingly, nurse practitioners were 4 times as likely to accept drug alerts as physicians.

CONCLUSIONS

Clinicians became less likely to accept alerts as they received more of them, particularly more repeated alerts. There was no evidence of an effect of workload per se, or of desensitization over time for a newly deployed alert. Reducing within-patient repeats may be a promising target for reducing alert overrides and alert fatigue.

Concept and implementation of a study dashboard module for a continuous monitoring of trial recruitment and documentation

BACKGROUND

The difficulty of managing patient recruitment and documentation for clinical trials prompts a demand for instruments for closely monitoring these critical but unpredictable processes. Increasingly adopted Electronic Data Capture (EDC) applications provide novel opportunities to reutilize stored information for an efficient management of traceable trial workflows. In related clinical and administrative settings, so-called digital dashboards that continuously visualize time-dependent parameters have recently met a growing acceptance. To investigate the technical feasibility of a study dashboard for monitoring the progress of patient recruitment and trial documentation, we set out to develop a propositional prototype in the form of a separate software module.

METHODS

After narrowing down functional requirements in semi-structured interviews with study coordinators, we analyzed available interfaces of a locally deployed EDC application, and designed the prototypical study dashboard based on previous findings. The module thereby leveraged a standardized export format in order to extract and import relevant trial data into a clinical data warehouse. Web-based reporting tools then facilitated the definition of diverse views, including diagrams of the progress of patient accrual and form completion at different granularity levels. To estimate the utility of the dashboard and its compatibility with current workflows, we interviewed study coordinators after a demonstration of sample outputs from ongoing trials.

RESULTS

The employed tools promoted a rapid development. Displays of the implemented dashboard are organized around an entry page that integrates key metrics for available studies, and which links to more detailed information such as study-specific enrollment per center. The interviewed experts commented that the included graphical summaries appeared suitable for detecting that something was generally amiss, although practical remedies would mostly depend on additional information such as access to the original patient-specific data. The dependency on a separate application was seen as a downside. Interestingly, the prospective users warned that in some situations knowledge of specific accrual statistics might undermine blinding in a subtle yet intricate fashion, so ignorance of certain patient features was seen as sometimes preferable for reproducibility.

DISCUSSION

Our proposed study dashboard graphically recaps key progress indicators of patient accrual and trial documentation. The modular implementation illustrates the technical feasibility of the approach. The use of a study dashboard might introduce certain technical requirements as well as subtle interpretative complexities, which may have to be weighed against potential efficiency gains.

Interface design recommendations for computerised clinical audit and feedback: Hybrid usability evidence from a research-led system

BACKGROUND

Audit and Feedback (A&F) is a widely used quality improvement technique that measures clinicians' clinical performance and reports it back to them. Computerised A&F (e-A&F) system interfaces may consist of four key components: (1) Summaries of clinical performance; (2) Patient lists; (3) Patient-level data; (4) Recommended actions. There is a lack of evidence regarding how to best design e-A&F interfaces; establishing such evidence is key to maximising usability, and in turn improving patient safety.

AIM

To evaluate the usability of a novel theoretically-informed and research-led e-A&F system for primary care (the Performance Improvement plaN GeneratoR: PINGR).

OBJECTIVES

(1) Describe PINGR's design, rationale and theoretical basis; (2) Identify usability issues with PINGR; (3) Understand how these issues may interfere with the cognitive goals of end-users; (4) Translate the issues into recommendations for the user-centred design of e-A&F systems.

METHODS

Eight experienced health system evaluators performed a usability inspection using an innovative hybrid approach consisting of five stages: (1) Development of representative user tasks, Goals, and Actions; (2) Combining Heuristic Evaluation and Cognitive Walkthrough methods into a single protocol to identify usability issues; (3) Consolidation of issues; (4) Severity rating of consolidated issues; (5) Analysis of issues according to usability heuristics, interface components, and Goal-Action structure.

RESULTS

A final list of 47 issues were categorised into 8 heuristic themes. The most error-prone heuristics were 'Consistency and standards' (13 usability issues; 28% of the total) and 'Match between system and real world' (n=10, 21%). The recommended actions component of the PINGR interface had the most usability issues (n=21, 45%), followed by patient-level data (n=5, 11%), patient lists (n=4, 9%), and summaries of clinical performance (n=4, 9%). The most error-prone Actions across all user Goals were: (1) Patient selection from a list; (2) Data identification from a figure (both population-level and patient-level); (3) Disagreement with a system recommendation.

CONCLUSIONS

By contextualising our findings within the wider literature on health information system usability, we provide recommendations for the design of e-A&F system interfaces relating to their four key components, in addition to how they may be integrated within a system.

Think time: A novel approach to analysis of clinicians' behavior after reduction of drug-drug interaction alerts

OBJECTIVES

Pharmacologic interaction alerting offers the potential for safer medication prescribing, but research reveals persistent concerns regarding alert fatigue. Research studies have tried various strategies to resolve this problem, with low overall success. We examined the effects of targeted alert reduction on clinician behavior in a resource constrained hospital.

METHODS

A physician and a pharmacy informaticist reduced alert levels of several drug-drug interactions (DDI) that clinicians almost always overrode with approval from and knowledge of the medical staff. This study evaluated the behavioral changes in prescribers and non-prescribers as measured by "think time", a new metric for evaluating the resolution time for an alert, before and after suppression of selected DDI alerts.

RESULTS

The user-seen DDI alert rate decreased from 9.98% of all orders to 9.20% (p=0.0001) with an overall volume reduction of 10.3%. There was no statistical difference in the reduction of cancelled (-10.00%) vs. proceed orders (-11.07%). Think time decreased overall by 0.61s (p<0.0001). Think time unexpectedly increased for cancelled orders 1.00s which while not statistically significant (p=0.28) is generally thought to be clinically noteworthy. For overrides, think time decreased 0.67s which was significant (p<0.0001). Think time lowered for both prescribers and non-prescribers. Targeted specialists had shorter think times initially, which shortened more than non-targeted specialists.

CONCLUSIONS

Targeted DDI alert reductions reduce alert burden overall, and increase net efficiency as measured by think time for all prescribers better than for non-prescribers. Think time may increase when cancelling or changing orders in response to DDI alerts vs. a decision to override an alert.

Payer Formulary Alerts as a Cause of Patient Harm and the Journey to Change Them

BACKGROUND AND OBJECTIVES

A safety event drew attention to unsafe and inappropriate payer formulary alerts. These alerts display formulary, coverage, and eligibility data from the pharmacy benefits manager in response to an electronic prescription. They are intended to redirect prescribers to medications that are covered by insurance; however, these alerts were found to be inaccurate and contribute to potentially harmful alerts. Our objective was to reduce inappropriate payer formulary alerts by 30% within 1 year and to change the ePrescribing certification requirements to prevent future instances of harm.

METHODS

Using process mapping we identified the changes that were required both locally and nationally through our electronic health record (EHR) vendor and ePrescribing transaction broker. We partnered with vendors to show the safety risk and to suggest modifications to the payer formulary alert content and ePrescribing certification criteria. On the basis of the new criteria, we modified and deactivated inappropriate alerts. Rates were followed weekly for 13 months and a control chart was used to track progress.

RESULTS

From January 2014 to January 2015, we reviewed 59 325 payer formulary alerts from ambulatory care and 11 630 from the emergency department and inpatient wards. Both local and national modifications resulted in significant and sustained decreases in inappropriate alerts.

CONCLUSIONS

Enduring and meaningful change required partnership with multiple stakeholders, including EHR vendors, ePrescribing vendors, and pharmacy benefits managers. Improving drug alerts, reducing alert fatigue, and promoting value-based prescribing in the EHR will likely require similar partnerships.

Ten key considerations for the successful optimization of large-scale health information technology

Implementation and adoption of complex health information technology (HIT) is gaining momentum internationally. This is underpinned by the drive to improve the safety, quality, and efficiency of care. Although most of the benefits associated with HIT will only be realized through optimization of these systems, relatively few health care organizations currently have the expertise or experience needed to undertake this. It is extremely important to have systems working before embarking on HIT optimization, which, much like implementation, is an ongoing, difficult, and often expensive process. We discuss some key organization-level activities that are important in optimizing large-scale HIT systems. These include considerations relating to leadership, strategy, vision, and continuous cycles of improvement. Although these alone are not sufficient to fully optimize complex HIT, they provide a starting point for conceptualizing this important area.