Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study

Transforming primary medical research knowledge into clinical decision

While the utility of computerized clinical decision support (CCDS) for multiple select clinical domains has been clearly demonstrated, much less is known about the full breadth of domains to which CCDS approaches could be productively applied. To explore the applicability of CCDS to general medical knowledge, we sampled a total of 500 primary research articles from 4 high-impact medical journals. Employing rule-based templates, we created high-level CCDS rules for 72% (361/500) of primary medical research articles. We subsequently identified data sources needed to implement those rules. Ourfindings suggest that CCDS approaches, perhaps in the form of non-interruptive infobuttons, could be much more broadly applied. In addition, our analytic methods appear to provide a means of prioritizing and quantitating the relative utility of available data sources for purposes of CCDS.

Rationale and design of the Novel Uses of adaptive Designs to Guide provider Engagement in Electronic Health Records (NUDGE-EHR) pragmatic adaptive randomized trial: a trial protocol

BACKGROUND

The prescribing of high-risk medications to older adults remains extremely common and results in potentially avoidable health consequences. Efforts to reduce prescribing have had limited success, in part because they have been sub-optimally timed, poorly designed, or not provided actionable information. Electronic health record (EHR)-based tools are commonly used but have had limited application in facilitating deprescribing in older adults. The objective is to determine whether designing EHR tools using behavioral science principles reduces inappropriate prescribing and clinical outcomes in older adults.

METHODS

The Novel Uses of Designs to Guide provider Engagement in Electronic Health Records (NUDGE-EHR) project uses a two-stage, 16-arm adaptive randomized pragmatic trial with a "pick-the-winner" design to identify the most effective of many potential EHR tools among primary care providers and their patients ≥ 65 years chronically using benzodiazepines, sedative hypnotic ("Z-drugs"), or anticholinergics in a large integrated delivery system. In stage 1, we randomized providers and their patients to usual care (n = 81 providers) or one of 15 EHR tools (n = 8 providers per arm) designed using behavioral principles including salience, choice architecture, or defaulting. After 6 months of follow-up, we will rank order the arms based upon their impact on the trial's primary outcome (for both stages): reduction in inappropriate prescribing (via discontinuation or tapering). In stage 2, we will randomize (a) stage 1 usual care providers in a 1:1 ratio to one of the up to 5 most promising stage 1 interventions or continue usual care and (b) stage 1 providers in the unselected arms in a 1:1 ratio to one of the 5 most promising interventions or usual care. Secondary and tertiary outcomes include quantities of medication prescribed and utilized and clinically significant adverse outcomes.

DISCUSSION

Stage 1 launched in October 2020. We plan to complete stage 2 follow-up in December 2021. These results will advance understanding about how behavioral science can optimize EHR decision support to improve prescribing and health outcomes. Adaptive trials have rarely been used in implementation science, so these findings also provide insight into how trials in this field could be more efficiently conducted.

TRIAL REGISTRATION

Clinicaltrials.gov ( NCT04284553 , registered: February 26, 2020).

Implementation of Electronic Decision Support for Diabetic Care in a Student-Run Clinic

Background and objectives Type 2 diabetes mellitus (T2DM) is a complex disease that can lead to complications. Electronic decision support in the electronic medical record (EMR) aids management. There is no study demonstrating the effectiveness of electronic decision support in assisting medical student providers in student-run free clinics. Methods There were 71 T2DM patients seen by medical students. Twenty-three encounters used a Diabetes Progress Note (DPN) that was created from consensus, opinion-based guidelines. Each note received a total composite score based on an eight-point scale for adherence to guidelines. Statistical comparisons between mean composite scores were performed using independent t-tests. Results The mean total composite score of DPN users was significantly greater than DPN non-users (5.35 vs. 4.23, p = 0.008), with a significant difference in the physical exam component (1.70 vs. 1.31, p = 0.002). Conclusions In this exploratory study, medical student providers at an attending-supervised, student-run free clinic that used electronic decision support during T2DM patient visits improved adherence to screening for diabetic complications and standard of care.

Implementation of Electronic Decision Support for Diabetic Care in a Student-Run Clinic

Background and objectives Type 2 diabetes mellitus (T2DM) is a complex disease that can lead to complications. Electronic decision support in the electronic medical record (EMR) aids management. There is no study demonstrating the effectiveness of electronic decision support in assisting medical student providers in student-run free clinics. Methods There were 71 T2DM patients seen by medical students. Twenty-three encounters used a Diabetes Progress Note (DPN) that was created from consensus, opinion-based guidelines. Each note received a total composite score based on an eight-point scale for adherence to guidelines. Statistical comparisons between mean composite scores were performed using independent t-tests. Results The mean total composite score of DPN users was significantly greater than DPN non-users (5.35 vs. 4.23, p = 0.008), with a significant difference in the physical exam component (1.70 vs. 1.31, p = 0.002). Conclusions In this exploratory study, medical student providers at an attending-supervised, student-run free clinic that used electronic decision support during T2DM patient visits improved adherence to screening for diabetic complications and standard of care.

Fentanyl Quality Assurance Project Prompted Change in Clinical Workflow and Test Configurations

BACKGROUND

Deaths attributable to fentanyl (FEN, a synthetic opioid) are high in Appalachia and highest in West Virginia. The goal of the study was to determine FEN prevalence among specimens submitted for definitive opioid testing and monitor responses to provider notifications of unexpected FEN findings during Q1 2020.

METHODS

All definitive opioid test data were reviewed daily for FEN signatures in Q1 2020. Unexpected FEN results were communicated to providers and monitored for 10 days to record actions taken. Prevalence data were categorized. Behavioral Medicine (BMED) leaders analyzed January data and implemented FEN screening in the clinic. BMED Q1 clinic visits and order volumes for drug screens were reviewed after Q1.

RESULTS

FEN positivity was 11% in Q1; >60% of findings were unexpected. Actions were taken for 54% of notifications in January but only 18% in March. Notifications required 70 hours of combined laboratory effort each month. BMED providers ordered 44% of definitive opioid tests and 69% of definitive FEN tests. Data prompted the addition of FEN to routine drug screen panels in the laboratory, and a 10% random FEN screening rate in the BMED opioid use disorder clinics (COAT).

CONCLUSIONS

Prevalence of FEN positivity was higher than initially expected, even for this region in Appalachia. Expanded presence of FEN screening should assist BMED providers with clinical efforts and help identify patients in need of intervention/therapy.

Digital Health in Physicians' and Pharmacists' Office: A Comparative Study of e-Prescription Systems' Architecture and Digital Security in Eight Countries

e-Prescription systems are key components and drivers of digital health. They can enhance the safety of the patients, and are gaining popularity in health care systems around the world. Yet, there is little knowledge on comparative international analysis of e-Prescription systems' architecture and digital security. We report, in this study, original findings from a comparative analysis of the e-Prescription systems in eight different countries, namely, Canada, United States, United Kingdom, Australia, Spain, Japan, Sweden, and Denmark. We surveyed the databases related to pharmacies, eHealth, e-Prescriptions, and related digital health websites for each country, and their system architectures. We also compared the digital security and privacy protocols in place within and across these digital systems. We evaluated the systems' authentication protocols used by pharmacies to verify patients' identities during the medication dispensing process. Furthermore, we examined the supporting systems/services used to manage patients' medication histories and enhance patients' medication safety. Taken together, we report, in this study, original comparative findings on the limitations and challenges of the surveyed systems as well as in adopting e-Prescription systems. While the present study was conducted before the onset of COVID-19, e-Prescription systems have become highly relevant during the current pandemic and hence, a deeper understanding of the country systems' architecture and digital security that can help design effective strategies against the pandemic. e-Prescription systems can help reduce physical contact and the risk of exposure to the virus, as well as the wait times in pharmacies, thus enhancing patient safety and improving planetary health.

Effects of Computerized Decision Support Systems on Practitioner Performance and Patient Outcomes: Systematic Review

Background

Computerized decision support systems (CDSSs) are software programs that support the decision making of practitioners and other staff. Other reviews have analyzed the relationship between CDSSs, practitioner performance, and patient outcomes. These reviews reported positive practitioner performance in over half the articles analyzed, but very little information was found for patient outcomes.

Objective

The purpose of this review was to analyze the relationship between CDSSs, practitioner performance, and patient medical outcomes. PubMed, CINAHL, Embase, Web of Science, and Cochrane databases were queried.

Methods

Articles were chosen based on year published (last 10 years), high quality, peer-reviewed sources, and discussion of the relationship between the use of CDSS as an intervention and links to practitioner performance or patient outcomes. Reviewers used an Excel spreadsheet (Microsoft Corporation) to collect information on the relationship between CDSSs and practitioner performance or patient outcomes. Reviewers also collected observations of participants, intervention, comparison with control group, outcomes, and study design (PICOS) along with those showing implicit bias. Articles were analyzed by multiple reviewers following the Kruse protocol for systematic reviews. Data were organized into multiple tables for analysis and reporting.

Results

Themes were identified for both practitioner performance (n=38) and medical outcomes (n=36). A total of 66% (25/38) of articles had occurrences of positive practitioner performance, 13% (5/38) found no difference in practitioner performance, and 21% (8/38) did not report or discuss practitioner performance. Zero articles reported negative practitioner performance. A total of 61% (22/36) of articles had occurrences of positive patient medical outcomes, 8% (3/36) found no statistically significant difference in medical outcomes between intervention and control groups, and 31% (11/36) did not report or discuss medical outcomes. Zero articles found negative patient medical outcomes attributed to using CDSSs.

Conclusions

Results of this review are commensurate with previous reviews with similar objectives, but unlike these reviews we found a high level of reporting of positive effects on patient medical outcomes.

Increasing uptake of hepatitis C virus infection case-finding, testing, and treatment in primary care: evaluation of the HepCATT (Hepatitis C Assessment Through to Treatment) trial

Background

Hepatitis C virus (HCV) infection is a key cause of liver disease but can be cured in more than 95% of patients. Around 70 000 people in England may have undiagnosed HCV infection and many more will not have been treated. Interventions to increase case-finding in primary care are likely to be cost-effective; however, evidence of effective interventions is lacking. The Hepatitis C Assessment Through to Treatment (HepCATT) trial assessed whether a complex intervention in primary care could increase case-finding, testing, and treatment of HCV.

Aim

To investigate the feasibility and acceptability of the HepCATT intervention.

Design and setting

A qualitative study with primary care practice staff from practices in the south west of England taking part in the HepCATT trial.

Method

Semi-structured interviews were carried out with GPs, nurses, and practice staff to ascertain their views of the HepCATT intervention at least 1 month after implementing the intervention in their practice. Normalisation process theory, which outlines the social processes involved in intervention implementation, informed thematic data analysis.

Results

Participants appreciated the HepCATT intervention for increasing knowledge and awareness of HCV. Although some initial technical difficulties were reported, participants saw the benefits of using the audit tool to systematically identify patients with HCV infection risk factors and found it straightforward to use. Participants valued the opportunity to discuss HCV testing with patients, especially those who may not have been previously aware of HCV risk. Future implementation should consider fully integrating software systems and additional resources to screen patient lists and conduct tests.

Conclusion

When supported by a complex intervention, primary care can play a crucial role in identifying and caring for patients with HCV infection, to help stem the HCV epidemic, and prevent HCV-related illness.

Clinical Implementation of Predictive Models Embedded within Electronic Health Record Systems: A Systematic Review

Predictive analytics using electronic health record (EHR) data have rapidly advanced over the last decade. While model performance metrics have improved considerably, best practices for implementing predictive models into clinical settings for point-of-care risk stratification are still evolving. Here, we conducted a systematic review of articles describing predictive models integrated into EHR systems and implemented in clinical practice. We conducted an exhaustive database search and extracted data encompassing multiple facets of implementation. We assessed study quality and level of evidence. We obtained an initial 3393 articles for screening, from which a final set of 44 articles was included for data extraction and analysis. The most common clinical domains of implemented predictive models were related to thrombotic disorders/anticoagulation (25%) and sepsis (16%). The majority of studies were conducted in inpatient academic settings. Implementation challenges included alert fatigue, lack of training, and increased work burden on the care team. Of 32 studies that reported effects on clinical outcomes, 22 (69%) demonstrated improvement after model implementation. Overall, EHR-based predictive models offer promising results for improving clinical outcomes, although several gaps in the literature remain, and most study designs were observational. Future studies using randomized controlled trials may help improve the generalizability of findings.

Information Management in the Emergency Department

Information management in the emergency department (ED) is a challenge for all providers. The volume of information required to care for each patient and to keep the ED functioning is immense. It must be managed through varying means of communication and in connection with ED information systems. Management of information in the ED is imperfect; different modes and methods of identification, interpretation, action, and communication can be beneficial or harmful to providers, patients, and departmental flow. This article reviews the state of information management in the ED and proposes recommendations to improve the management of information in the future.

Feedback at the Point of Care to Decrease Medication Alert Rates in an Electronic Health Record

Frequently overridden alerts in the electronic health record can highlight alerts that may need revision. This method is a way of fine-tuning clinical decision support. We evaluated the feasibility of a complementary, yet different method that directly involved pediatric emergency department (PED) providers in identifying additional medication alerts that were potentially incorrect or intrusive. We then evaluated the effect subsequent resulting modifications had on alert salience.

METHODS

We performed a prospective, interventional study over 34 months (March 6, 2014, to December 31, 2016) in the PED. We implemented a passive alert feedback mechanism by enhancing the native electronic health record functionality on alert reviews. End-users flagged potentially incorrect/bothersome alerts for review by the study's team. The alerts were updated when clinically appropriate and trends of the impact were evaluated.

RESULTS

More than 200 alerts were reported from both inside and outside the PED, suggesting an intuitive approach. On average, we processed 4 reviews per week from the PED, with attending physicians as major contributors. The general trend of the impact of these changes seems favorable.

DISCUSSION

The implementation of the review mechanism for user-selected alerts was intuitive and sustainable and seems to be able to detect alerts that are bothersome to the end-users. The method should be run in parallel with the traditional data-driven approach to support capturing of inaccurate alerts.

CONCLUSIONS

User-centered, context-specific alert feedback can be used for selecting suboptimal, interruptive medication alerts.

Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow

The overwhelming number of medication alerts generated by clinical decision support systems (CDSS) has led to inappropriate alert overrides, which may lead to unintended patient harm. This review highlights the factors affecting the alert appropriateness of CDSS and barriers to the fit of CDSS alert with clinical workflow. A literature review was conducted to identify features and functions pertinent to CDSS alert appropriateness using the five rights of CDSS. Moreover, a process improvement method, namely, Lean, was used as a tool to optimise clinical workflows, and the appropriate design for CDSS alert using a human automation interaction (HAI) model was recommended. Evaluating the appropriateness of CDSS alert and its impact on workflow provided insights into how alerts can be designed and triggered effectively to support clinical workflow. The application of Lean methods and tools to analyse alert efficiencies in supporting workflow in this study provides an in-depth understanding of alert-workflow fit problems and their root cause, which is required for improving CDSS design. The application of the HAI model is recommended in the design of CDSS alerts to support various levels and stages of alert automations, namely, information acquisition and analysis, decision action and action implementation.

Evaluation of an artificial intelligence clinical trial matching system in Australian lung cancer patients

Objective

The objective of this technical study was to evaluate the performance of an artificial intelligence (AI)-based system for clinical trials matching for a cohort of lung cancer patients in an Australian cancer hospital.

Methods

A lung cancer cohort was derived from clinical data from patients attending an Australian cancer hospital. Ten phases I–III clinical trials registered on clinicaltrials.gov and open to lung cancer patients at this institution were utilized for assessments. The trial matching system performance was compared to a gold standard established by clinician consensus for trial eligibility.

Results

The study included 102 lung cancer patients. The trial matching system evaluated 7252 patient attributes (per patient median 74, range 53–100) against 11 467 individual trial eligibility criteria (per trial median 597, range 243–4132). Median time for the system to run a query and return results was 15.5 s (range 7.2–37.8). In establishing the gold standard, clinician interrater agreement was high (Cohen’s kappa 0.70–1.00). On a per-patient basis, the performance of the trial matching system for eligibility was as follows: accuracy, 91.6%; recall (sensitivity), 83.3%; precision (positive predictive value), 76.5%; negative predictive value, 95.7%; and specificity, 93.8%.

Discussion and Conclusion

The AI-based clinical trial matching system allows efficient and reliable screening of cancer patients for clinical trials with 95.7% accuracy for exclusion and 91.6% accuracy for overall eligibility assessment; however, clinician input and oversight are still required. The automated system demonstrates promise as a clinical decision support tool to prescreen a large patient cohort to identify subjects suitable for further assessment.

Development, Implementation, and Evaluation of a Personalized Machine Learning Algorithm for Clinical Decision Support: Case Study With Shingles Vaccination

BACKGROUND

Although clinical decision support (CDS) alerts are effective reminders of best practices, their effectiveness is blunted by clinicians who fail to respond to an overabundance of inappropriate alerts. An electronic health record (EHR)-integrated machine learning (ML) algorithm is a potentially powerful tool to increase the signal-to-noise ratio of CDS alerts and positively impact the clinician's interaction with these alerts in general.

OBJECTIVE

This study aimed to describe the development and implementation of an ML-based signal-to-noise optimization system (SmartCDS) to increase the signal of alerts by decreasing the volume of low-value herpes zoster (shingles) vaccination alerts.

METHODS

We built and deployed SmartCDS, which builds personalized user activity profiles to suppress shingles vaccination alerts unlikely to yield a clinician's interaction. We extracted all records of shingles alerts from January 2017 to March 2019 from our EHR system, including 327,737 encounters, 780 providers, and 144,438 patients.

RESULTS

During the 6 weeks of pilot deployment, the SmartCDS system suppressed an average of 43.67% (15,425/35,315) potential shingles alerts (appointments) and maintained stable counts of weekly shingles vaccination orders (326.3 with system active vs 331.3 in the control group; P=.38) and weekly user-alert interactions (1118.3 with system active vs 1166.3 in the control group; P=.20).

CONCLUSIONS

All key statistics remained stable while the system was turned on. Although the results are promising, the characteristics of the system can be subject to future data shifts, which require automated logging and monitoring. We demonstrated that an automated, ML-based method and data architecture to suppress alerts are feasible without detriment to overall order rates. This work is the first alert suppression ML-based model deployed in practice and serves as foundational work in encounter-level customization of alert display to maximize effectiveness.

Reducing Alert Burden in Electronic Health Records: State of the Art Recommendations from Four Health Systems

BACKGROUND

Electronic health record (EHR) alert fatigue, while widely recognized as a concern nationally, lacks a corresponding comprehensive mitigation plan.

OBJECTIVES

The goal of this manuscript is to provide practical guidance to clinical informaticists and other health care leaders who are considering creating a program to manage EHR alerts.

METHODS

This manuscript synthesizes several approaches and recommendations for better alert management derived from four U.S. health care institutions that presented their experiences and recommendations at the American Medical Informatics Association 2019 Clinical Informatics Conference in Atlanta, Georgia, United States. The assembled health care institution leaders represent academic, pediatric, community, and specialized care domains. We describe governance and management, structural concepts and components, and human-computer interactions with alerts, and make recommendations regarding these domains based on our experience supplemented with literature review. This paper focuses on alerts that impact bedside clinicians.

RESULTS

The manuscript addresses the range of considerations relevant to alert management including a summary of the background literature about alerts, alert governance, alert metrics, starting an alert management program, approaches to evaluating alerts prior to deployment, and optimization of existing alerts. The manuscript includes examples of alert optimization successes at two of the represented institutions. In addition, we review limitations on the ability to evaluate alerts in the current state and identify opportunities for further scholarship.

CONCLUSION

Ultimately, alert management programs must strive to meet common goals of improving patient care, while at the same time decreasing the alert burden on clinicians. In so doing, organizations have an opportunity to promote the wellness of patients, clinicians, and EHRs themselves.

Primary care perspectives on implementation of clinical trial recruitment

Introduction:

Poor clinical trial (CT) recruitment is a significant barrier to translating basic science discoveries into medical practice. Improving support for primary care provider (PCP) referral of patients to CTs may be an important part of the solution. However, implementing CT referral support in primary care is not only technically challenging, but also presents challenges at the person and organization levels.

Methods:

The objectives of this study were (1) to characterize provider and clinical supervisor attitudes and perceptions regarding CT research, recruitment, and referrals in primary care and (2) to identify perceived workflow strategies and facilitators relevant to designing a technology-supported primary care CT referral program. Focus groups were conducted with PCPs, directors, and supervisors.

Results:

Analysis indicated widespread support for the intrinsic scientific value of CTs, while at the same time deep concerns regarding protecting patient well-being, perceived loss of control when patients participate in trials, concern about the impact of point-of-care referrals on clinic workflow, the need for standard processes, and the need for CT information that enables referring providers to quickly confirm that the burdens are justified by the benefits at both patient and provider levels. PCP suggestions pertinent to implementing a CT referral decision support system are reported.

Conclusion:

The results from this work contribute to developing an implementation approach to support increased referral of patients to CTs.

Electronic health record nested pragmatic randomized controlled trial of a reminder system for serum lithium level monitoring in patients with mood disorder: KONOTORI study protocol

Background

The weaknesses of classical explanatory randomized controlled trials (RCTs) include limited generalizability, high cost, and time burden. Pragmatic RCTs nested within electronic health records (EHRs) can be useful to overcome such limitations. Serum lithium monitoring has often been underutilized in real-world practice in Japan. This trial aims to evaluate the effectiveness of the EHR-nested reminder system for serum lithium level monitoring in the maintenance of therapeutic lithium concentration and in the improvement of the quality of care for patients on lithium maintenance therapy.

Methods

The Kyoto Toyooka nested controlled trial of reminders (KONOTORI trial) is an EHR-nested, parallel-group, superiority, stratified, permuted block-randomized controlled trial. Screening, random allocation, reminder output, and outcome collection will be conducted automatically by the EHR-nested trial program. Patients with a mood disorder taking lithium carbonate for maintenance therapy will be randomly allocated to the two-step reminder system for serum lithium monitoring or to usual care. The primary outcome is the achievement of therapeutic serum lithium concentration between 0.4 and 1.0 mEq/L at 18 months after informed consent.

Discussion

The KONOTORI trial uses EHRs to enable the efficient conduct of a pragmatic trial of the reminder system for lithium monitoring. This may contribute to improved quality of care for patients on lithium maintenance therapy.

Trial registration

University Hospital Medical Information Network (UMIN) Clinical Trials Registry, UMIN000033633. Registered on 3 July 2018.

Evaluating the Impact of Interruptive Alerts within a Health System: Use, Response Time, and Cumulative Time Burden

BACKGROUND

Health systems often employ interruptive alerts through the electronic health record to improve patient care. However, concerns of "alert fatigue" have been raised, highlighting the importance of understanding the time burden and impact of these alerts on providers.

OBJECTIVES

Our main objective was to determine the total time providers spent on interruptive alerts in both inpatient and outpatient settings. Our secondary objectives were to analyze dwell time for individual alerts and examine both provider and alert-related factors associated with dwell time variance.

METHODS

We retrospectively evaluated use and response to the 75 most common interruptive ("popup") alerts between June 1st, 2015 and November 1st, 2016 in a large academic health care system. Alert "dwell time" was calculated as the time between the alert appearing on a provider's screen until it was closed. The total number of alerts and dwell times per provider per month was calculated for inpatient and outpatient alerts and compared across alert type.

RESULTS

The median number of alerts seen by a provider was 12 per month (IQR 4-34). Overall, 67% of inpatient and 39% of outpatient alerts were closed in under 3 seconds. Alerts related to patient safety and those requiring more than a single click to proceed had significantly longer median dwell times of 5.2 and 6.7 seconds, respectively. The median total monthly time spent by providers viewing alerts was 49 seconds on inpatient alerts and 28 seconds on outpatient alerts.

CONCLUSION

Most alerts were closed in under 3 seconds and a provider's total time spent on alerts was less than 1 min/mo. Alert fatigue may lie in their interruptive and noncritical nature rather than time burden. Monitoring alert interaction time can function as a valuable metric to assess the impact of alerts on workflow and potentially identify routinely ignored alerts.

Smart pumps improve medication safety but increase alert burden in neonatal care

Background

Smart pumps have been widely adopted but there is limited evidence to understand and support their use in pediatric populations. Our objective was to assess whether smart pumps are effective at reducing medication errors in the neonatal population and determine whether they are a source of alert burden and alert fatigue in an intensive care environment.

Methods

Using smart pump records, over 370,000 infusion starts for continuously infused medications used in neonates and infants hospitalized in a level IV NICU from 2014 to 2016 were evaluated. Attempts to exceed preset soft and hard maximum limits, percent variance from those limits, and pump alert frequency, patterns and salience were evaluated.

Results

Smart pumps prevented 160 attempts to exceed the hard maximum limit for doses that were as high as 7–29 times the maximum dose and resulted in the reprogramming or cancellation of 2093 infusions after soft maximum alerts. While the overall alert burden from smart pumps for continuous infusions was not high, alerts clustered around specific patients and medications, and a small portion (17%) of infusions generated the majority of alerts. Soft maximum alerts were often overridden (79%), consistent with low alert salience.

Conclusions

Smart pumps have the ability to improve neonatal medication safety when compliance with dose error reducing software is high. Numerous attempts to administer high doses were intercepted by dosing alerts. Clustered alerts may generate a high alert burden and limit safety benefit by desensitizing providers to alerts. Future efforts should address ways to improve alert salience.

Minimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration

OBJECTIVES

Previous studies have looked at National Early Warning Score performance in predicting in-hospital deterioration and death, but data are lacking with respect to patient outcomes following implementation of National Early Warning Score. We sought to determine the effectiveness of National Early Warning Score implementation on predicting and preventing patient deterioration in a clinical setting.

DESIGN

Retrospective cohort study.

SETTING

Tertiary care academic facility and a community hospital.

PATIENTS

Patients 18 years old or older hospitalized from March 1, 2014, to February 28, 2015, during preimplementation of National Early Warning Score to August 1, 2015, to July 31, 2016, after National Early Warning Score was implemented.

INTERVENTIONS

Implementation of National Early Warning Score within the electronic health record and associated best practice alert.

MEASUREMENTS AND MAIN RESULTS

In this study of 85,322 patients (42,402 patients pre-National Early Warning Score and 42,920 patients post-National Early Warning Score implementation), the primary outcome of rate of ICU transfer or death did not change after National Early Warning Score implementation, with adjusted hazard ratio of 0.94 (0.84-1.05) and 0.90 (0.77-1.05) at our academic and community hospital, respectively. In total, 175,357 best practice advisories fired during the study period, with the best practice advisory performing better at the community hospital than the academic at predicting an event within 12 hours 7.4% versus 2.2% of the time, respectively. Retraining National Early Warning Score with newly generated hospital-specific coefficients improved model performance.

CONCLUSIONS

At both our academic and community hospital, National Early Warning Score had poor performance characteristics and was generally ignored by frontline nursing staff. As a result, National Early Warning Score implementation had no appreciable impact on defined clinical outcomes. Refitting of the model using site-specific data improved performance and supports validating predictive models on local data.

Effect of an Electronic Health Record Decision Support Alert to Decrease Excess Cervical Cancer Screening

OBJECTIVE

Cervical cancer screening is often conducted in excess of current screening guidelines. The objective of this study was to evaluate the effect of an electronic health record (EHR) clinical decision support alert to decrease guideline-nonadherent cervical cancer screening beyond the age limits of screening or posthysterectomy.

MATERIALS AND METHODS

The proportion of guideline-nonadherent Pap tests in women younger than 21 years or older than 65 years or posthysterectomy were compared 4 months before and 3 months after implementation of an EHR clinical decision support alert warning providers that a Pap test is not indicated. Providers could cancel the Pap test or override the alert and place the order. Provider characteristics and Pap test indications were summarized by preintervention/postintervention period using descriptive statistics. The proportions of nonindicated Pap tests were compared by intervention period and provider characteristics using generalized estimating equation models.

RESULTS

In women beyond the screening age limits or posthysterectomy, a total of 388 Pap tests were ordered before intervention, and 313 tests were ordered after intervention. Proportion of guideline-nonadherent tests was similar before (62%) and after intervention (63%); thus, implementation of the clinical decision support alert did not change the proportion of guideline-nonadherent Pap tests ordered (OR = 1.08, 95% CI = 0.77-1.52). It is notable that 52% of guideline-nonadherent tests were ordered by 11 providers. Even when controlling for providers who ordered more than 1 test during the study period, multivariate analysis showed that male providers were more likely to order guideline-nonadherent Pap tests (OR = 2.30, 95% CI = 1.36-3.89); no other differences by provider characteristics were observed.

CONCLUSIONS

An EHR clinical decision support alert does not decrease guideline-nonadherent cervical cancer screening. These data suggest efforts to optimize clinical decision support should be focused on other aspects of cervical cancer prevention.

Improving laboratory test utilisation at the multihospital Yale New Haven Health System

Background

Waste persists in healthcare and negatively impacts patients. Clinicians have direct control over test ordering and ongoing international efforts to improve test utilisation have identified multifaceted approaches as critical to the success of interventions. Prior to 2015, Yale New Haven Health lacked a coherent strategy for laboratory test utilisation management.

Methods

In 2015, a system-wide laboratory formulary committee was formed at Yale New Haven Health to manage multiple interventions designed to improve test utilisation. We report here on specific interventions conducted between 2015 and 2017 including reduction of (1) obsolete or misused testing, (2) duplicate orders, and (3) daily routine lab testing. These interventions were driven by a combination of modifications to computerised physician order entry, test utilisation dashboards and physician education. Measurements included test order volume, blood savings and cost savings.

Results

Testing for a number of obsolete/misused analytes was eliminated or significantly decreased depending on alert rule at order entry. Hard stops significantly decreased duplicate testing and educational sessions significantly decreased daily orders of routine labs and increased blood savings but the impact waned over time for select groups. In total, we realised approximately $100 000 of cost savings during the study period.

Conclusion

Through a multifaceted approach to utilisation management, we show significant reductions in low-value clinical testing that have led to modest but significant savings in both costs and patients’ blood.

A Real-Time Automated Patient Screening System for Clinical Trials Eligibility in an Emergency Department: Design and Evaluation

Background

One critical hurdle for clinical trial recruitment is the lack of an efficient method for identifying subjects who meet the eligibility criteria. Given the large volume of data documented in electronic health records (EHRs), it is labor-intensive for the staff to screen relevant information, particularly within the time frame needed. To facilitate subject identification, we developed a natural language processing (NLP) and machine learning–based system, Automated Clinical Trial Eligibility Screener (ACTES), which analyzes structured data and unstructured narratives automatically to determine patients’ suitability for clinical trial enrollment. In this study, we integrated the ACTES into clinical practice to support real-time patient screening.

Objective

This study aimed to evaluate ACTES’s impact on the institutional workflow, prospectively and comprehensively. We hypothesized that compared with the manual screening process, using EHR-based automated screening would improve efficiency of patient identification, streamline patient recruitment workflow, and increase enrollment in clinical trials.

Methods

The ACTES was fully integrated into the clinical research coordinators’ (CRC) workflow in the pediatric emergency department (ED) at Cincinnati Children’s Hospital Medical Center. The system continuously analyzed EHR information for current ED patients and recommended potential candidates for clinical trials. Relevant patient eligibility information was presented in real time on a dashboard available to CRCs to facilitate their recruitment. To assess the system’s effectiveness, we performed a multidimensional, prospective evaluation for a 12-month period, including a time-and-motion study, quantitative assessments of enrollment, and postevaluation usability surveys collected from the CRCs.

Results

Compared with manual screening, the use of ACTES reduced the patient screening time by 34% (P<.001). The saved time was redirected to other activities such as study-related administrative tasks (P=.03) and work-related conversations (P=.006) that streamlined teamwork among the CRCs. The quantitative assessments showed that automated screening improved the numbers of subjects screened, approached, and enrolled by 14.7%, 11.1%, and 11.1%, respectively, suggesting the potential of ACTES in streamlining recruitment workflow. Finally, the ACTES achieved a system usability scale of 80.0 in the postevaluation surveys, suggesting that it was a good computerized solution.

Conclusions

By leveraging NLP and machine learning technologies, the ACTES demonstrated good capacity for improving efficiency of patient identification. The quantitative assessments demonstrated the potential of ACTES in streamlining recruitment workflow and improving patient enrollment. The postevaluation surveys suggested that the system was a good computerized solution with satisfactory usability.

The Effect of Patient-Specific Drug-Drug Interaction Alerting on the Frequency of Alerts: A Pilot Study

Background: False-positive drug-drug interaction alerts are frequent and result in alert fatigue that can result in prescribers bypassing important alerts. Development of a method to present patient-appropriate alerts is needed to help restore alert relevance. Objective: The purpose of this study was to assess the potential for patient-specific drug-drug interaction (DDI) alerts to reduce alert burden. Methods: This project was conducted at a tertiary care medical center. Seven of the most frequently encountered DDI alerts were chosen for developing patient-specific, algorithm-based DDI alerts. For each of the DDI pairs, 2 algorithms featuring different values for modifying factors were made. DDI alerts from the 7 drug pairs were collected over 30 days. Outcome measures included the number of DDI alerts generated before and after patient-specific algorithm application to the same patients over the same time period. Results: A total of 14 algorithms were generated, and each was evaluated by comparing the number of alerts generated by our existing, customized clinical decision support (CDS) software and the patient-specific algorithms. The CDS DDI alerting software generated an average of 185.3 alerts per drug pair over the 30-day study period. Patient-specific algorithms reduced the number of alerts resulting from the algorithms by 11.3% to 93.5%. Conclusion and Relevance: Patient-specific DDI alerting is an innovative and effective approach to reduce the number of DDI alerts, may potentially increase the appropriateness of alerts, and may decrease the potential for alert fatigue.

Let Them Speak for Themselves: Improving Adolescent Self-Report Rate on Pre-Visit Screening

BACKGROUND

Adolescent pre-visit screening on patient-generated health data is a common and efficient practice to guide clinical decision making. However, proxy informants (eg, parents or caregivers) often complete these forms, which may lead to incorrect information or lack of confidentiality. Our objective was to improve the adolescent self-report rate on pre-visit screening.

METHODS

We conducted an interventional study using an interrupted time series design to compare adolescent self-report rates (percent of adolescents ages 12-18 years completing their own pre-visit screening) over 16 months in general pediatric ambulatory clinics. We collected data using a computerized clinical decision support system with waiting room electronic tablet screening. Preintervention rates were low, and we created and implemented 2 electronic workflow alerts, one each to the patient/caregiver and clinical staff, reminding them that the adolescent should answer the questions independently. We included the first encounter from each adolescent and evaluated changes in adolescent self-reporting between pre- and postintervention periods using interrupted time series analysis.

RESULTS

Patients or caregivers completed 2670 qualifying pre-visit screenings across 19 preintervention, 7 intervention, and 44 postintervention weeks. Self-reporting by younger adolescents nearly doubled, with a significant increase of 19.3 percentage points (confidence interval [CI], 9.1-29.5) from the baseline 20.5%. Among older adolescents, the stable baseline rate of 53.6% increased by 9.2 absolute percentage points (CI, -7.0 to 25.3). There were no significant pre- or postintervention secular trends.

CONCLUSIONS

Two automated alerts directing clinic personnel and families to have adolescents self-report significantly and sustainably improved younger adolescent self-reporting on electronic patient-generated health data instruments.

Understanding the use of digital technology to promote human papillomavirus vaccination - A RE-AIM framework approach

The human papillomavirus virus (HPV) vaccine is effective at preventing various cancers, but coverage falls short of targets that are needed for community protection. Here, we use the RE-AIM implementation framework (Reach, Effectiveness, Adoption, Implementation, Maintenance) to understand how text, email, and electronic health record (EHR) reminders and social media campaigns can be used as part of policy and practice interventions to increase HPV vaccination. These technology-based interventions could be used together and mainstreamed into clinical and system-based practice to have the greatest impact. Of the interventions explored, text-based, email-based, and EHR reminders have the most evidence behind them to support their effectiveness. While there are several studies of promotion of the HPV vaccine on social media, more studies are needed to demonstrate their effects and better methods are needed to be able to attribute results to these interventions.

Physician clinical decision support system prompts and administration of subsequent doses of HPV vaccine: A randomized clinical trial

BACKGROUND

HPV vaccine is effective in preventing several cancers and anogenital warts, yet rates of HPV vaccination series completion in the United States are low. A primary reason identified by parents for vaccinating children against HPV is a health care provider's recommendation. Although most clinicians embrace vaccine recommendations, they are not always carried out evenly and subsequent HPV vaccines are missed.

METHODS

Using an electronic health records-based decision support system (CHICA) clinicians were randomized to either usual practice or to receive an automated reminder to recommend the 2nd or 3rd dose of HPV vaccine. The reminder was delivered to clinicians of all intervention group eligible adolescents who had already initiated the vaccine series. Logistic regression models with generalized estimating equations were used for data analysis.

RESULTS

A total of 1285 clinical encounters were observed across 29 randomized pediatric providers over a 13-month time frame (50.7% control group, 49.3% intervention group). Overall, patients were 44.9% female, 59.4% Black, 22.1% Hispanic, and 48.8% were ages 11-12 yrs. Within the control group, 421 (64.7%) received a subsequent HPV vaccine, compared to 481 (75.9%) (OR: 1.72, (95% CI 1.35-2.19)). Adjusted analysis showed no difference between the groups (aOR 1.52 (95% CI 0.88-2.62)) or when examined by age (11-12yrs aOR 1.66, (95% CI 0.79-3.48)) and 13-17yrs (aOR 1.19, (95% CI 0.76-1.85)) or gender female (aOR 1.39 (95% CI 0.71-2.72)) and males (aOR 1.67 (95% CI 0.95-2.92)). When results were stratified by both age and gender, there was similarly no statistically significant effect between the two groups.

CONCLUSIONS

Automated physician reminders for subsequent 2nd and 3rd doses of HPV vaccination were used. Despite increased rates of vaccination in the intervention group, the differences did not reach the level of statistical significance. Future studies with multifaceted approaches may be needed to examine the efficacy of computer-based reminders.

CLINICAL TRIAL REGISTRATION

NCT02558803, "HPV Vaccination: Evaluation of Reminder Prompts for Doses 2 & 3".

A Multifaceted Quality Improvement Intervention to Improve Watchful Waiting in Acute Otitis Media Management

INTRODUCTION

Despite the American Academy of Pediatrics (AAP) guidelines for acute otitis media (AOM) describing a watchful waiting (WW) approach in qualifying patients, immediate antibiotics are consistently overutilized. The study team developed a multifaceted quality improvement intervention that educated providers and families about WW and included a behavioral component to modify physician prescribing patterns.

METHODS

We used data from a prior study of 250 patients 18 years old and younger with AOM in a tertiary care children's hospital emergency department (ED) to characterize baseline AOM management before interventions. In this study, interventions took place from September to December 2016. Following the interventions, 65 patients were randomly selected, which would allow for the detection of a 20% increase in adherence to AAP guidelines for management of AOM.

RESULTS

In the preintervention cohort of 250 patients, 247 had documented AOM. Two hundred thirty-one (93.5%) received immediate antibiotics, 7 (2.8%) underwent WW, and 9 (3.6%) were sent home without antibiotics. Overall management agreed with AAP guidelines at a rate of 44.1%. In the postintervention cohort of 65 patients, 63 met age and diagnostic criteria for AOM; 56 (88.9%) patients received immediate antibiotics; and 7 (11.1%) underwent WW. Postintervention, which the ED management of AOM agreed with AAP guidelines 60.3% of the time, was significantly increased from preintervention adherence (P = 0.02).

CONCLUSIONS

A multipronged quality improvement intervention for AOM management in a single pediatric ED significantly improved adherence to AAP guidelines by increasing WW and reducing immediate antibiotic prescriptions.

Utilization of computerized clinical decision support for potentially inappropriate medications

Background: Electronic medical record (EMR) alerts may inform point of care decisions, including the decision to prescribe potentially inappropriate medications (PIM) identified in the Beers criteria. EMR alerts may not be considered relevant or informative in the clinician context, leading to a phenomenon colloquially known as “alert fatigue.”

Objective: To assess the frequency of clinical interaction with EMR alerts and associated deprescribing behaviors in ambulatory settings.

Methods: This is a retrospective observational study in two ambulatory clinics (the Kaye Edmonton Clinic Senior’s Clinic and the Lynnwood Family Practice Clinic) in Edmonton over an observational period of 30 months. Statistical analysis was done using descriptive statistics, chi-square and regression analysis.

Results: The reminder performance for interactions with the alert was 17.2% across the two clinics. The Number Needed to Remind (NNR) or mean number of alerts shown on clinician screens prior to a single interaction of any kind with the alert was 5.8. When actions were defined as a deprescribing (ie discontinuation) event that was related to the alert and that particular interaction in the EMR, the reminder performance was 1.2%, for an NNR of 82.8.

Conclusion: The configuration of alerts in the EMR was not associated with a clinically detectable increase in the uptake of the Beers criteria for high hazard medications.

Use of Social Psychology to Improve Adherence to National Bronchiolitis Guidelines

OBJECTIVES

The American Academy of Pediatrics' bronchiolitis guidelines recommend against albuterol and corticosteroids for treating and chest radiographs (CRs) for diagnosing infants with bronchiolitis. However, high rates of nonadherence have been documented. Our objective was to improve guideline adherence in infants with bronchiolitis.

METHODS

This quality improvement study was conducted in 1 urban academic pediatric primary care clinic caring for predominately minority and publicly insured children. We tested provider guideline education, display of guidelines in patient care areas, and monthly e-mails to all providers documenting deviation rates, with individual e-mails to providers who deviated. P-charts and interrupted time series analysis were used to estimate the effect of the intervention.

RESULTS

There were 380 children <2 years of age with a diagnosis of bronchiolitis in the 16 nonsummer months preintervention and 417 in the 15 postintervention months. Rates of prescribed and administered albuterol declined from 45.7% in the baseline period to 13.7% in the intervention period and CR use dropped from a mean of 10.1% to 3.4%, both demonstrating special cause variation. Steroid use did not change significantly. In interrupted time series analyses, the intervention was associated with a significant decrease in albuterol use (P < .001) but not in CR or steroid use. Emergency department visits declined slightly but admissions for bronchiolitis were stable.

CONCLUSIONS

Traditional quality improvement efforts coupled with social psychology techniques resulted in improved guideline adherence in outpatient bronchiolitis management. Additional study will help identify which techniques are most effective for increasing guideline adherence in cases of low-value care.

When an Alert is Not an Alert: A Pilot Study to Characterize Behavior and Cognition Associated with Medication Alerts

Introduction. Preventable adverse drug events are a significant patient-safety concern, yet most medication alerts are disregarded. Pharmacists encounter the highest number of medication alerts and likely have developed behaviors to cope with alerting inefficiencies. The study objective was to better understand alert override behavior relating to a motivational construct framework. Methods. Mixed-methods study of 10 pharmacists (567 verifications) with eye-tracking observations and retrospective think aloud interviews. Results. Pharmacists spent on average 14 seconds longer verifying orders with alerts than orders without alerts (p<0.001). Verification occurred before alerts were triggered, and no order changes occurred after alerts. Pharmacists reported 62% of alerts as unhelpful and 21% as frustrating. Alert interactions took on average 3.9 seconds. Discussion. Pharmacists anticipate alerts by making appropriate checks and changes before alert prompts. Medication alerts seem to be useful. However, the observed pharmacists' behavior suggests changes in the alert context are needed to match cognition.

Challenges optimizing the after visit summary

BACKGROUND

The after visit summary (AVS) is a paper or electronic document given to patients after a medical appointment, which is intended to summarize patients' health and guide future care, including self-management tasks.

OBJECTIVE

To describe experiences of health systems implementing a redesigned outpatient AVS in commercially available electronic health record (EHR) systems to inform future optimization.

MATERIALS AND METHODS

We conducted semi-structured interviews with information technology and clinical leaders at 12 hospital and community-based healthcare institutions across the continental United States focusing on the process of AVS redesign and implementation. We also report our experience implementing a redesigned AVS in the Epic EHR at the Mount Sinai Hospital in New York City, NY.

RESULTS

Health systems experienced many challenges implementing the redesigned AVS. While many IT leaders noted that the redesigned AVS is easier to understand and the document is better organized, they claim the effort is time-consuming, Epic system upgrades render AVS modifications non-functional, and primary care and specialty practices have different needs in regards to content and formatting. Our team was able to modify the document by changing the order of print groups, modifying the font size, bolding section headers, and inserting page breaks. Similar to other health systems, our team found that it is difficult to achieve some desired features due to limitations in the EHR platform.

CONCLUSION

Health IT leaders view the AVS as a valuable source of information for patients. However, limitations to AVS modifications in EHR systems present challenges to optimizing the tool. EHR vendors should incorporate learning from healthcare systems innovation efforts and consider building more flexibility into their product development.

Accuracy of Provider-Selected Indications for Antibiotic Orders

Documentation of antibiotic indication provides helpful information for antimicrobial stewardship, but accuracy is not understood. Review of 396 antibiotic orders in a pediatric ICU and adult medicine step-down unit found 90% agreement between provider-selected indication and independent review. Prompts to enter antibiotic indication during order entry provide largely accurate information. Infect Control Hosp Epidemiol 2018;39:111-113.

The Impact of a Location-Sensing Electronic Health Record on Clinician Efficiency and Accuracy: A Pilot Simulation Study

BACKGROUND

Through the Health Information Technology for Economic and Clinical Health Act of 2009, the federal government invested $26 billion in electronic health records (EHRs) to improve physician performance and patient safety; however, these systems have not met expectations. One of the cited issues with EHRs is the human-computer interaction, as exhibited by the excessive number of interactions with the interface, which reduces clinician efficiency. In contrast, real-time location systems (RTLS)-technologies that can track the location of people and objects-have been shown to increase clinician efficiency. RTLS can improve patient flow in part through the optimization of patient verification activities. However, the data collected by RTLS have not been effectively applied to optimize interaction with EHR systems.

OBJECTIVES

We conducted a pilot study with the intention of improving the human-computer interaction of EHR systems by incorporating a RTLS. The aim of this study is to determine the impact of RTLS on process metrics (i.e., provider time, number of rooms searched to find a patient, and the number of interactions with the computer interface), and the outcome metric of patient identification accuracy METHODS:  A pilot study was conducted in a simulated emergency department using a locally developed camera-based RTLS-equipped EHR that detected the proximity of subjects to simulated patients and displayed patient information when subjects entered the exam rooms. Ten volunteers participated in 10 patient encounters with the RTLS activated (RTLS-A) and then deactivated (RTLS-D). Each volunteer was monitored and actions recorded by trained observers. We sought a 50% improvement in time to locate patients, number of rooms searched to locate patients, and the number of mouse clicks necessary to perform those tasks.

RESULTS

The time required to locate patients (RTLS-A = 11.9 ± 2.0 seconds vs. RTLS-D = 36.0 ± 5.7 seconds, p < 0.001), rooms searched to find patient (RTLS-A = 1.0 ± 1.06 vs. RTLS-D = 3.8 ± 0.5, p < 0.001), and number of clicks to access patient data (RTLS-A = 1.0 ± 0.06 vs. RTLS-D = 4.1 ± 0.13, p < 0.001) were significantly reduced with RTLS-A relative to RTLS-D. There was no significant difference between RTLS-A and RTLS-D for patient identification accuracy.

CONCLUSION

This pilot demonstrated in simulation that an EHR equipped with real-time location services improved performance in locating patients and reduced error compared with an EHR without RTLS. Furthermore, RTLS decreased the number of mouse clicks required to access information. This study suggests EHRs equipped with real-time location services that automates patient location and other repetitive tasks may improve physician efficiency, and ultimately, patient safety.

Automating a Manual Sepsis Screening Tool in a Pediatric Emergency Department

OBJECTIVE

This article describes the method of integrating a manual pediatric emergency department sepsis screening process into the electronic health record that leverages existing clinical documentation and keeps providers in their current, routine clinical workflows.

METHODS

Criteria in the manual pediatric emergency department sepsis screening tool were mapped to standard documentation routinely entered in the electronic health record. Data elements were extracted and scored from the medical history, medication record, vital signs, and physical assessments. Scores that met a predefined sepsis risk threshold triggered interruptive system alerts which notified emergency department staff to perform sepsis huddles and consider appropriate interventions. Statistical comparison of the new electronic tool to the manual process was completed by a two-tail paired t-test.

RESULTS

The performance of the pediatric electronic sepsis screening tool was evaluated by comparing flowsheet row documentation of the manual, sepsis alert process against the interruptive system alert instance of the electronic sepsis screening tool. In an 8-week testing period, the automated pediatric electronic sepsis screening tool identified 100% of patients flagged by the manual process (n = 29), on average, 68 minutes earlier.

CONCLUSION

Integrating a manual sepsis screening tool into the electronic health record automated identification of pediatric sepsis screening in a busy emergency department. The electronic sepsis screening tool is as accurate as a manual process and would alert bedside clinicians significantly earlier in the emergency department course. Deployment of this electronic tool has the capability to improve timely sepsis detection and management of patients at risk for sepsis without requiring additional documentation by providers.

Nudging physicians and patients with autopend clinical decision support to improve diabetes management

OBJECTIVES

To determine the impact on routine glycated hemoglobin (A1C) laboratory test completion of incorporating an autopend laboratory order functionality into clinical decision support, which (1) routed provider alerts to a separate electronic folder, (2) automatically populated preauthorization forms, and (3) linked the timing and content of electronic patient health maintenance topic (HMT) reminders to the provider authorization.

STUDY DESIGN

Observational pre-post study from November 2011 (1 year before autopend) through June 2014 (1.5 years after).

METHODS

The study included HMT reminders concerning an A1C test for patients with type 1 or type 2 diabetes (N = 15,630 HMT reminders; 8792 patients) in a large multispecialty ambulatory healthcare system. A Cox proportional hazard model, adjusted for patient and provider demographics, estimated the likelihood of laboratory test completion based on 3 HMT reminder characteristics: preautopend versus postautopend period, read versus unread, and the patient's time to reading.

RESULTS

In the postautopend period, the median time for patients to read reminders decreased (1 vs 3 days; P <.001) and the median time to complete laboratory tests decreased (40 vs 48 days; P <.001). Comparing preautopend HMT reminders with a similar time to reading, the likelihood of A1C laboratory test completion increased after autopend by between 21.1% (hazard ratio [HR], 1.211; P = .050), when time to reading was 57 days, and 33.9% (HR, 1.339; P = .003), when time to reading was 0 days. This result included 68% of the reminders. There was no statistical difference in A1C laboratory test completion for unread reminders in the preautopend versus postautopend period.

CONCLUSIONS

Automated patient-centered decision support can improve guideline-concordant monitoring of A1C among patients with diabetes, particularly among patients who read reminders in a timely fashion.

Assessing patient risk of central line-associated bacteremia via machine learning

BACKGROUND

Central line-associated bloodstream infections (CLABSIs) contribute to increased morbidity, length of hospital stay, and cost. Despite progress in understanding the risk factors, there remains a need to accurately predict the risk of CLABSIs and, in real time, prevent them from occurring.

METHODS

A predictive model was developed using retrospective data from a large academic healthcare system. Models were developed with machine learning via construction of random forests using validated input variables.

RESULTS

Fifteen variables accounted for the most significant effect on CLABSI prediction based on a retrospective study of 70,218 unique patient encounters between January 1, 2013, and May 31, 2016. The area under the receiver operating characteristic curve for the best-performing model was 0.82 in production.

DISCUSSION

This model has multiple applications for resource allocation for CLABSI prevention, including serving as a tool to target patients at highest risk for potentially cost-effective but otherwise time-limited interventions.

CONCLUSIONS

Machine learning can be used to develop accurate models to predict the risk of CLABSI in real time prior to the development of infection.

Advocating for Greater Usability in Clinical Technologies: The Role of the Practicing Nurse

Health care, especially ICUs, rely on multiple types of technology to promote the best patient outcomes. Unfortunately, too often these technologies are poorly designed, causing errors, additional workload, and unnecessary frustration. The purpose of this article is to (1) empower nurses with the needed usability and usability testing vocabulary to identify and articulate clinical technology usability problems and (2) provide ideas on ways nurses can advocate to have an impact on positive change related to technology usability within a health care organization.

Reducing indwelling urinary catheter use through staged introduction of electronic clinical decision support in a multicenter hospital system

Objective

To integrate electronic clinical decision support tools into clinical practice and to evaluate the impact on indwelling urinary catheter (IUC) use and catheter-associated urinary tract infections (CAUTIs).

Design, Setting, and Participants

This 4-phase observational study included all inpatients at a multicampus, academic medical center between 2011 and 2015.

Interventions

Phase 1 comprised best practices training and standardization of electronic documentation. Phase 2 comprised real-time electronic tracking of IUC duration. In phase 3, a triggered alert reminded clinicians of IUC duration. In phase 4, a new IUC order (1) introduced automated order expiration and (2) required consideration of alternatives and selection of an appropriate indication.

Results

Overall, 2,121 CAUTIs, 179,070 new catheters, 643,055 catheter days, and 2,186 reinsertions occurred in 3·85 million hospitalized patient days during the study period. The CAUTI rate per 10,000 patient days decreased incrementally in each phase from 9·06 in phase 1 to 1·65 in phase 4 (relative risk [RR], 0·182; 95% confidence interval [CI], 0·153–0·216; P<·001). New catheters per 1,000 patient days declined from 53·4 in phase 1 to 39·5 in phase 4 (RR, 0·740; 95% CI, 0·730; P<·001), and catheter days per 1,000 patient days decreased from 194·5 in phase 1 to 140·7 in phase 4 (RR, 0·723; 95% CI, 0·719–0·728; P<·001). The reinsertion rate declined from 3·66% in phase 1 to 3·25% in phase 4 (RR, 0·894; 95% CI, 0·834–0·959; P=·0017).

Conclusions

The phased introduction of decision support tools was associated with progressive declines in new catheters, total catheter days, and CAUTIs. Clinical decision support tools offer a viable and scalable intervention to target hospital-wide IUC use and hold promise for other quality improvement initiatives.

Identification of Patients Expected to Benefit from Electronic Alerts for Acute Kidney Injury

BACKGROUND AND OBJECTIVES

Electronic alerts for heterogenous conditions such as AKI may not provide benefit for all eligible patients and can lead to alert fatigue, suggesting that personalized alert targeting may be useful. Uplift-based alert targeting may be superior to purely prognostic-targeting of interventions because uplift models assess marginal treatment effect rather than likelihood of outcome.

DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS

This is a secondary analysis of a clinical trial of 2278 adult patients with AKI randomized to an automated, electronic alert system versus usual care. We used three uplift algorithms and one purely prognostic algorithm, trained in 70% of the data, and evaluated the effect of targeting alerts to patients with higher scores in the held-out 30% of the data. The performance of the targeting strategy was assessed as the interaction between the model prediction of likelihood to benefit from alerts and randomization status. The outcome of interest was maximum relative change in creatinine from the time of randomization to 3 days after randomization.

RESULTS

The three uplift score algorithms all gave rise to a significant interaction term, suggesting that a strategy of targeting individuals with higher uplift scores would lead to a beneficial effect of AKI alerting, in contrast to the null effect seen in the overall study. The prognostic model did not successfully stratify patients with regards to benefit of the intervention. Among individuals in the high uplift group, alerting was associated with a median reduction in change in creatinine of -5.3% (P=0.03). In the low uplift group, alerting was associated with a median increase in change in creatinine of +5.3% (P=0.005). Older individuals, women, and those with a lower randomization creatinine were more likely to receive high uplift scores, suggesting that alerts may benefit those with more slowly developing AKI.

CONCLUSIONS

Uplift modeling, which accounts for treatment effect, can successfully target electronic alerts for AKI to those most likely to benefit, whereas purely prognostic targeting cannot.

The Use of Clinical Decision Support in Reducing Diagnosis of and Treatment of Asymptomatic Bacteriuria

Clinical decision support (CDS) embedded within the electronic health record (EHR) is a potential antibiotic stewardship strategy for hospitalized patients. Reduction in urine testing and treating asymptomatic bacteriuria (ASB) is an important strategy to promote antibiotic stewardship. We created an intervention focused on reducing urine testing for asymptomatic patients at a large tertiary care center. The objective of this study was to design an intervention to reduce unnecessary urinalysis and urine culture (UC) orders as well as the treatment of ASB. We performed a quasiexperimental study among adult inpatients at a single academic institution. We implemented a bundled intervention, including information broadcast in newsletters, hospitalwide screensavers, and passive CDS messages in the EHR. We investigated the impact of this strategy on urinalysis, UC orders, and on the treatment of ASB by using an interrupted time series analysis. Our intervention led to reduced UC order as well as reduced antibiotic orders in response to urinalysis orders and UC results. This easily implementable bundle may play an important role as an antibiotic stewardship strategy.

Mixed methods study of medication-related decision support alerts experienced during electronic prescribing for inpatients at an English hospital

Objectives

Electronic prescribing and medication administration systems are being introduced in many hospitals worldwide, with varying degrees of clinical decision support including pop-up alerts. Previous research suggests that prescribers override a high proportion of alerts, but little research has been carried out in the UK. Our objective was to explore rates of alert overriding in different prescribing situations and prescribers’ perceptions around the use of decision support alerts in a UK hospital.

Methods

We conducted a mixed methods study on three cardiology wards, directly observing medical and non-medical prescribers’ alert override rates during both ward round and non-ward round prescribing; observations were followed by semi-structured interviews with prescribers, which were then transcribed and analysed thematically.

Results

Overall, 69% of 199 observed alerts were overridden. Alerts experienced during ward rounds were significantly more likely to be overridden than those outside of ward rounds (80% of 56 vs 51% of 63; p=0.001, Χ² test). While respondents acknowledged that alerts could be useful, several also described negative unintended consequences. Many were of the view that usefulness of alerts was limited if the alert was reminding them to do something they would do anyway, or suggesting something they did not feel was relevant. Findings suggest that targeting, timing and additional features of alerts are critical factors in determining whether they are acted on or overridden.

Conclusion

The majority of alerts were overridden. Alerts may be less likely to be overridden if they are built into the prescribing workflow.

Strategies to improve recruitment to randomised trials

BACKGROUND

Recruiting participants to trials can be extremely difficult. Identifying strategies that improve trial recruitment would benefit both trialists and health research.

OBJECTIVES

To quantify the effects of strategies for improving recruitment of participants to randomised trials. A secondary objective is to assess the evidence for the effect of the research setting (e.g. primary care versus secondary care) on recruitment.

SEARCH METHODS

We searched the Cochrane Methodology Review Group Specialised Register (CMR) in the Cochrane Library (July 2012, searched 11 February 2015); MEDLINE and MEDLINE In Process (OVID) (1946 to 10 February 2015); Embase (OVID) (1996 to 2015 Week 06); Science Citation Index & Social Science Citation Index (ISI) (2009 to 11 February 2015) and ERIC (EBSCO) (2009 to 11 February 2015).

SELECTION CRITERIA

Randomised and quasi-randomised trials of methods to increase recruitment to randomised trials. This includes non-healthcare studies and studies recruiting to hypothetical trials. We excluded studies aiming to increase response rates to questionnaires or trial retention and those evaluating incentives and disincentives for clinicians to recruit participants.

DATA COLLECTION AND ANALYSIS

We extracted data on: the method evaluated; country in which the study was carried out; nature of the population; nature of the study setting; nature of the study to be recruited into; randomisation or quasi-randomisation method; and numbers and proportions in each intervention group. We used a risk difference to estimate the absolute improvement and the 95% confidence interval (CI) to describe the effect in individual trials. We assessed heterogeneity between trial results. We used GRADE to judge the certainty we had in the evidence coming from each comparison.

MAIN RESULTS

We identified 68 eligible trials (24 new to this update) with more than 74,000 participants. There were 63 studies involving interventions aimed directly at trial participants, while five evaluated interventions aimed at people recruiting participants. All studies were in health care.We found 72 comparisons, but just three are supported by high-certainty evidence according to GRADE.1. Open trials rather than blinded, placebo trials. The absolute improvement was 10% (95% CI 7% to 13%).2. Telephone reminders to people who do not respond to a postal invitation. The absolute improvement was 6% (95% CI 3% to 9%). This result applies to trials that have low underlying recruitment. We are less certain for trials that start out with moderately good recruitment (i.e. over 10%).3. Using a particular, bespoke, user-testing approach to develop participant information leaflets. This method involved spending a lot of time working with the target population for recruitment to decide on the content, format and appearance of the participant information leaflet. This made little or no difference to recruitment: absolute improvement was 1% (95% CI -1% to 3%).We had moderate-certainty evidence for eight other comparisons; our confidence was reduced for most of these because the results came from a single study. Three of the methods were changes to trial management, three were changes to how potential participants received information, one was aimed at recruiters, and the last was a test of financial incentives. All of these comparisons would benefit from other researchers replicating the evaluation. There were no evaluations in paediatric trials.We had much less confidence in the other 61 comparisons because the studies had design flaws, were single studies, had very uncertain results or were hypothetical (mock) trials rather than real ones.

AUTHORS' CONCLUSIONS

The literature on interventions to improve recruitment to trials has plenty of variety but little depth. Only 3 of 72 comparisons are supported by high-certainty evidence according to GRADE: having an open trial and using telephone reminders to non-responders to postal interventions both increase recruitment; a specialised way of developing participant information leaflets had little or no effect. The methodology research community should improve the evidence base by replicating evaluations of existing strategies, rather than developing and testing new ones.

Hepatitis C virus testing for case identification in persons born during 1945-1965: Results from three randomized controlled trials

The Centers for Disease Control and Prevention and US Preventive Services Task Force recommend one-time hepatitis C virus (HCV) testing for persons born during 1945-1965 (birth cohort). However, few studies estimate the effect of birth cohort (BC) testing implementation on HCV diagnoses in primary care settings. We aimed to determine the probability of identifying HCV infections in primary care using targeted BC testing compared with usual care at three academic medical centers. From December 2012 to March 2014, each center compared one of three distinct interventions with usual care using an independently designed randomized controlled trial. Across centers, BC patients with no clinical documentation of previous HCV testing or diagnosis were randomly assigned to receive a one-time offering of HCV antibody (anti-HCV) testing via one of three independent implementation strategies (repeated-mailing outreach, electronic medical record-integrated provider best practice alert [BPA], and direct patient solicitation) or assigned to receive usual care. We estimated model-adjusted risk ratios (aRR) of anti-HCV-positive (anti-HCV+) identification using BC testing versus usual care. In the repeated mailing trial, 8992 patients (intervention, n = 2993; control, n = 5999) were included in the analysis. The intervention was eight times as likely to identify anti-HCV+ patients compared with controls (aRR, 8.0; 95% confidence interval [CI], 2.8-23.0; adjusted probabilities: intervention, 0.27%; control, 0.03%). In the BPA trial, data from 14,475 patients (BC, n = 8928; control, n = 5,547) were analyzed. The intervention was 2.6 times as likely to identify anti-HCV+ patients versus controls (aRR, 2.6; 95% CI, 1.1-6.4; adjusted probabilities: intervention, 0.29%; control, 0.11%). In the patient-solicitation trial, 8873 patients (BC, n = 4307; control, n = 4566) were analyzed. The intervention was five times as likely to identify anti-HCV+ patients compared with controls (aRR, 5.3; 95% CI, 2.3-12.3; adjusted probabilities: intervention, 0.68%; control, 0.11%). Conclusion: BC testing was effective in identifying previously undiagnosed HCV infections in primary care settings. (Hepatology 2018;67:524-533).

Alert override as a habitual behavior - a new perspective on a persistent problem

Quantifying alert override has been the focus of much research in health informatics, with override rate traditionally viewed as a surrogate inverse indicator for alert effectiveness. However, relying on alert override to assess computerized alerts assumes that alerts are being read and determined to be irrelevant by users. Our research suggests that this is unlikely to be the case when users are experiencing alert overload. We propose that over time, alert override becomes habitual. The override response is activated by environmental cues and repeated automatically, with limited conscious intention. In this paper we outline this new perspective on understanding alert override. We present evidence consistent with the notion of alert override as a habitual behavior and discuss implications of this novel perspective for future research on alert override, a common and persistent problem accompanying decision support system implementation.

A survey of practices for the use of electronic health records to support research recruitment

Electronic health records (EHRs) provide great promise for identifying cohorts and enhancing research recruitment. Such approaches are sorely needed, but there are few descriptions in the literature of prevailing practices to guide their use. A multidisciplinary workgroup was formed to examine current practices in the use of EHRs in recruitment and to propose future directions. The group surveyed consortium members regarding current practices. Over 98% of the Clinical and Translational Science Award Consortium responded to the survey. Brokered and self-service data warehouse access are in early or full operation at 94% and 92% of institutions, respectively, whereas, EHR alerts to providers and to research teams are at 45% and 48%, respectively, and use of patient portals for research is at 20%. However, these percentages increase significantly to 88% and above if planning and exploratory work were considered cumulatively. For most approaches, implementation reflected perceived demand. Regulatory and workflow processes were similarly varied, and many respondents described substantive restrictions arising from logistical constraints and limitations on collaboration and data sharing. Survey results reflect wide variation in implementation and approach, and point to strong need for comparative research and development of best practices to protect patients and facilitate interinstitutional collaboration and multisite research.