When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record

A Practical Approach to Optimize Computerized Provider Order Entry Systems and Reduce Clinician Burden: Pre-Post Evaluation of Vendor-Derived "Order Friction" Data

Computerized provider order entry (CPOE) systems have been cited as a significant contributor to clinician burden. Vendor-derived measures and data sets have been developed to help with optimization of CPOE systems. We describe how we analyzed vendor-derived Order Friction (OF) EHR log data at our health system and propose a practical approach for optimizing CPOE systems by reducing OF. We also conducted a pre-post intervention study using OF data to evaluate the impact of defaulting the frequency of urine, stool and nasal swab tests and found that all modified orders had significantly fewer changes required per order (p<0.01). Our proposed approach is a six-step process: 1) understand the ordering process, 2) understand OF data elements contextually, 3) explore ordering user-level factors, 4) evaluate order volume and friction from different order sources, 5) optimize order-level design, 6) identify high volume alerts to evaluate for appropriateness.

Assessing Usability and Ambulatory Clinical Staff Satisfaction with Two Electronic Health Records

BACKGROUND

A growing body of literature has linked usability limitations within electronic health records (EHRs) to adverse outcomes which may in turn affect EHR system transitions. NewYork-Presbyterian Hospital, Columbia University College of Physicians and Surgeons (CU), and Weill Cornell Medical College (WC) are a tripartite organization with large academic medical centers that initiated a phased transition of their EHRs to one system, EpicCare.

OBJECTIVES

This article characterizes usability perceptions stratified by provider roles by surveying WC ambulatory clinical staff already utilizing EpicCare and CU ambulatory clinical staff utilizing iterations of Allscripts before the implementation of EpicCare campus-wide.

METHODS

A customized 19-question electronic survey utilizing usability constructs based on the Health Information Technology Usability Evaluation Scale was anonymously administered prior to EHR transition. Responses were recorded with self-reported demographics.

RESULTS

A total of 1,666 CU and 1,065 WC staff with ambulatory self-identified work setting were chosen. Select demographic statistics between campus staff were generally similar with small differences in patterns of clinical and EHR experience. Results demonstrated significant differences in EHR usability perceptions among ambulatory staff based on role and EHR system. WC staff utilizing EpicCare accounted for more favorable usability metrics than CU across all constructs. Ordering providers (OPs) denoted less usability than non-OPs. The Perceived Usefulness and User Control constructs accounted for the largest differences in usability perceptions. The Cognitive Support and Situational Awareness construct was similarly low for both campuses. Prior EHR experience demonstrated limited associations.

CONCLUSION

Usability perceptions can be affected by role and EHR system. OPs consistently denoted less usability overall and were more affected by EHR system than non-OPs. While there was greater perceived usability for EpicCare to perform tasks related to care coordination, documentation, and error prevention, there were persistent shortcomings regarding tab navigation and cognitive burden reduction, which have implications on provider efficiency and wellness.

The Effect of Implementation of Guideline Order Bundles Into a General Admission Order Set on Clinical Practice Guideline Adoption: Quasi-Experimental Study

BACKGROUND

Clinical practice guidelines (CPGs) and associated order sets can help standardize patient care and lead to higher-value patient care. However, difficult access and poor usability of these order sets can result in lower use rates and reduce the CPGs' impact on clinical outcomes. At our institution, we identified multiple CPGs for general pediatrics admissions where the appropriate order set was used in <50% of eligible encounters, leading to decreased adoption of CPG recommendations.

OBJECTIVE

We aimed to determine how integrating disease-specific order groups into a common general admission order set influences adoption of CPG-specific order bundles for patients meeting CPG inclusion criteria admitted to the general pediatrics service.

METHODS

We integrated order bundles for asthma, heavy menstrual bleeding, musculoskeletal infection, migraine, and pneumonia into a common general pediatrics order set. We compared pre- and postimplementation order bundle use rates for eligible encounters at both an intervention and nonintervention site for integrated CPGs. We also assessed order bundle adoption for nonintegrated CPGs, including bronchiolitis, acute gastroenteritis, and croup. In a post hoc analysis of encounters without order bundle use, we compared the pre- and postintervention frequency of diagnostic uncertainty at the time of admission.

RESULTS

CPG order bundle use rates for incorporated CPGs increased by +9.8% (from 629/856, 73.5% to 405/486, 83.3%) at the intervention site and by +5.1% (896/1351, 66.3% to 509/713, 71.4%) at the nonintervention site. Order bundle adoption for nonintegrated CPGs decreased from 84% (536/638) to 68.5% (148/216), driven primarily by decreases in bronchiolitis order bundle adoption in the setting of the COVID-19 pandemic. Diagnostic uncertainty was more common in admissions without CPG order bundle use after implementation (28/227, 12.3% vs 19/81, 23.4%).

CONCLUSIONS

The integration of CPG-specific order bundles into a general admission order set improved overall CPG adoption. However, integrating only some CPGs may reduce adoption of order bundles for excluded CPGs. Diagnostic uncertainty at the time of admission is likely an underrecognized barrier to guideline adherence that is not addressed by an integrated admission order set.

Objective risk assessment vs standard care for acute coronary syndromes-The Australian GRACE Risk tool Implementation Study (AGRIS): a process evaluation

Background

Structured risk-stratification to guide clinician assessment and engagement with evidence-based therapies may reduce care variance and improve patient outcomes for Acute Coronary Syndrome (ACS). The Australian Grace Risk score Intervention Study (AGRIS) explored the impact of the GRACE Risk Tool for stratification of ischaemic and bleeding risk in ACS. While hospitals in the active arm had a higher overall rate of invasive ACS management, there was neutral impact on important secondary prevention prescriptions/referrals, hospital performance measures, myocardial infarction and 12-month mortality leading to early trial cessation. Given the Grace Risk Tool is under investigation internationally, this process evaluation study provides important insights into the possible contribution of implementation fidelity on the AGRIS study findings.

Methods

Using maximum variation sampling, five hospitals were selected from the 12 centres enrolled in the active arm of AGRIS. From these facilities, 16 local implementation stakeholders (Cardiology advanced practice nurses, junior and senior doctors, study coordinators) consented to a semi-structured interview guided by the Theoretical Domains Framework. Directed Content Analysis of qualitative data was structured using the Capability/Opportunity/Motivation-Behaviour (COM-B) model.

Results

Physical capability was enhanced by tool usability. While local stakeholders supported educating frontline clinicians, non-cardiology clinicians struggled with specialist terminology. Physical opportunity was enhanced by the paper-based format but was hampered when busy clinicians viewed risk-stratification as one more thing to do, or when form visibility was neglected. Social opportunity was supported by a culture of research/evidence yet challenged by clinical workflow and rotating medical officers. Automatic motivation was strengthened by positive reinforcement. Reflective motivation revealed the GRACE Risk Tool as supporting but potentially overriding clinical judgment. Divergent professional roles and identity were a major barrier to integration of risk-stratification into routine Emergency Department practice. The cumulative result revealed poor form completion behaviors and a failure to embed risk-stratification into routine patient assessment, communication, documentation, and clinical practice behaviors.

Conclusions

Numerous factors negatively influenced AGRIS implementation fidelity. Given the prominence of risk assessment recommendations in United States, European and Australian guidelines, strategies that strengthen collaboration with Emergency Departments and integrate automated processes for risk-stratification may improve future translation internationally.

Alert burden in pediatric hospitals: a cross-sectional analysis of six academic pediatric health systems using novel metrics

BACKGROUND

Excessive electronic health record (EHR) alerts reduce the salience of actionable alerts. Little is known about the frequency of interruptive alerts across health systems and how the choice of metric affects which users appear to have the highest alert burden.

OBJECTIVE

(1) Analyze alert burden by alert type, care setting, provider type, and individual provider across 6 pediatric health systems. (2) Compare alert burden using different metrics.

MATERIALS AND METHODS

We analyzed interruptive alert firings logged in EHR databases at 6 pediatric health systems from 2016-2019 using 4 metrics: (1) alerts per patient encounter, (2) alerts per inpatient-day, (3) alerts per 100 orders, and (4) alerts per unique clinician days (calendar days with at least 1 EHR log in the system). We assessed intra- and interinstitutional variation and how alert burden rankings differed based on the chosen metric.

RESULTS

Alert burden varied widely across institutions, ranging from 0.06 to 0.76 firings per encounter, 0.22 to 1.06 firings per inpatient-day, 0.98 to 17.42 per 100 orders, and 0.08 to 3.34 firings per clinician day logged in the EHR. Custom alerts accounted for the greatest burden at all 6 sites. The rank order of institutions by alert burden was similar regardless of which alert burden metric was chosen. Within institutions, the alert burden metric choice substantially affected which provider types and care settings appeared to experience the highest alert burden.

CONCLUSION

Estimates of the clinical areas with highest alert burden varied substantially by institution and based on the metric used.

The influence of integrating clinical practice guideline order bundles into a general admission order set on guideline adoption

Objectives of this study were to (1) describe barriers to using clinical practice guideline (CPG) admission order sets in a pediatric hospital and (2) determine if integrating CPG order bundles into a general admission order set increases adoption of CPG-recommended orders compared to standalone CPG order sets. We identified CPG-eligible encounters and surveyed admitting physicians to understand reasons for not using the associated CPG order set. We then integrated CPG order bundles into a general admission order set and evaluated effectiveness through summative usability testing in a simulated environment. The most common reasons for the nonuse of CPG order sets were lack of awareness or forgetting about the CPG order set. In usability testing, CPG order bundle use increased from 27.8% to 66.6% while antibiotic ordering errors decreased from 62.9% to 18.5% with the new design. Integrating CPG-related order bundles into a general admission order set improves CPG order set use in simulation by addressing the most common barriers to CPG adoption.

OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases

OBJECTIVE

To assess usability and usefulness of a machine learning-based order recommender system applied to simulated clinical cases.

MATERIALS AND METHODS

43 physicians entered orders for 5 simulated clinical cases using a clinical order entry interface with or without access to a previously developed automated order recommender system. Cases were randomly allocated to the recommender system in a 3:2 ratio. A panel of clinicians scored whether the orders placed were clinically appropriate. Our primary outcome included the difference in clinical appropriateness scores. Secondary outcomes included total number of orders, case time, and survey responses.

RESULTS

Clinical appropriateness scores per order were comparable for cases randomized to the order recommender system (mean difference -0.11 order per score, 95% CI: [-0.41, 0.20]). Physicians using the recommender placed more orders (median 16 vs 15 orders, incidence rate ratio 1.09, 95%CI: [1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Physicians used recommender suggestions in 98% of available cases. Approximately 95% of participants agreed the system would be useful for their workflows.

DISCUSSION

User testing with a simulated electronic medical record interface can assess the value of machine learning and clinical decision support tools for clinician usability and acceptance before live deployments.

CONCLUSIONS

Clinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface in a simulated setting. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.

Effects of Electronic Health Record Implementation and Barriers to Adoption and Use: A Scoping Review and Qualitative Analysis of the Content

Despite the great advances in the field of electronic health records (EHRs) over the past 25 years, implementation and adoption challenges persist, and the benefits realized remain below expectations. This scoping review aimed to present current knowledge about the effects of EHR implementation and the barriers to EHR adoption and use. A literature search was conducted in PubMed, Web of Science, IEEE Xplore Digital Library and ACM Digital Library for studies published between January 2005 and May 2020. In total, 7641 studies were identified of which 142 met the criteria and attained the consensus of all researchers on inclusion. Most studies (n = 91) were published between 2017 and 2019 and 81 studies had the United States as the country of origin. Both positive and negative effects of EHR implementation were identified, relating to clinical work, data and information, patient care and economic impact. Resource constraints, poor/insufficient training and technical/educational support for users, as well as poor literacy and skills in technology were the identified barriers to adoption and use that occurred frequently. Although this review did not conduct a quality analysis of the included papers, the lack of uniformity in the use of EHR definitions and detailed contextual information concerning the study settings could be observed.

Information Management in the Emergency Department

Information management in the emergency department (ED) is a challenge for all providers. The volume of information required to care for each patient and to keep the ED functioning is immense. It must be managed through varying means of communication and in connection with ED information systems. Management of information in the ED is imperfect; different modes and methods of identification, interpretation, action, and communication can be beneficial or harmful to providers, patients, and departmental flow. This article reviews the state of information management in the ED and proposes recommendations to improve the management of information in the future.

ClinicNet: machine learning for personalized clinical order set recommendations

OBJECTIVE

This study assesses whether neural networks trained on electronic health record (EHR) data can anticipate what individual clinical orders and existing institutional order set templates clinicians will use more accurately than existing decision support tools.

MATERIALS AND METHODS

We process 57 624 patients worth of clinical event EHR data from 2008 to 2014. We train a feed-forward neural network (ClinicNet) and logistic regression applied to the traditional problem structure of predicting individual clinical items as well as our proposed workflow of predicting existing institutional order set template usage.

RESULTS

ClinicNet predicts individual clinical orders (precision = 0.32, recall = 0.47) better than existing institutional order sets (precision = 0.15, recall = 0.46). The ClinicNet model predicts clinician usage of existing institutional order sets (avg. precision = 0.31) with higher average precision than a baseline of order set usage frequencies (avg. precision = 0.20) or a logistic regression model (avg. precision = 0.12).

DISCUSSION

Machine learning methods can predict clinical decision-making patterns with greater accuracy and less manual effort than existing static order set templates. This can streamline existing clinical workflows, but may not fit if historical clinical ordering practices are incorrect. For this reason, manually authored content such as order set templates remain valuable for the purposeful design of care pathways. ClinicNet's capability of predicting such personalized order set templates illustrates the potential of combining both top-down and bottom-up approaches to delivering clinical decision support content.

CONCLUSION

ClinicNet illustrates the capability for machine learning methods applied to the EHR to anticipate both individual clinical orders and existing order set templates, which has the potential to improve upon current standards of practice in clinical order entry.

Physician Usage and Acceptance of a Machine Learning Recommender System for Simulated Clinical Order Entry

Clinical decision support tools that automatically disseminate patterns of clinical orders have the potential to improve patient care by reducing errors of omission and streamlining physician workflows. However, it is unknown if physicians will accept such tools or how their behavior will be affected. In this randomized controlled study, we exposed 34 licensed physicians to a clinical order entry interface and five simulated emergency cases, with randomized availability of a previously developed clinical order recommender system. With the recommender available, physicians spent similar time per case (6.7 minutes), but placed more total orders (17.1 vs. 15.8). The recommender demonstrated superior recall (59% vs 41%) and precision (25% vs 17%) compared to manual search results, and was positively received by physicians recognizing workflow benefits. Further studies must assess the potential clinical impact towards a future where electronic health records automatically anticipate clinical needs.

Adherence to guidelines across different specialties to prevent infections in patients undergoing immunosuppressive therapies

BACKGROUND

Substantial numbers of patients are now receiving either immunosuppressive therapies or chemotherapy. There are significant risks in such patients of developing opportunistic infections or re-activation of latent infections, with higher associated morbidity and mortality. The aim of this quality improvement project was to determine how effective 5 different specialties were in assessing and mitigating risks of developing opportunistic infections or re-activation of latent infections in patients undergoing immunosuppressive therapies.

METHODS

This was a single centre audit where records of patients attending clinics providing immunosuppressive therapies were reviewed for the following: evidence of screening for blood-borne virus [BBV] infections, varicella and measles immunity, latent/active TB or hypogammaglobulinaemia, and whether appropriate vaccines had been advised or various infection risks discussed. These assessments were audited against both national and international guidelines, or a cross-specialty consensus guideline where specific recommendations were lacking. Two sub-populations were also analysed separately: patients receiving more potent immunosuppression and black and minority ethnic [BME] patients,.

RESULTS

For the 204 patients fulfilling the inclusion criteria, BBV, varicella/measles and latent TB screening was inconsistent, as was advice for vaccinations, with few areas complying with specialty or consensus guidelines. Less than 10% of patients in one specialty were tested for HIV. In BME patients screening for HIV [60%], measles [0%] and varicella [40%] immunity and latent [30%] or active [20%] TB was low. Only 38% of patients receiving potent immunosuppression received Pneumocystis prophylaxis, with 3 of 4 specialties providing less than 15% of patients in this category with prophylaxis.

CONCLUSIONS

Compliance with guidelines to mitigate risks of infection from immunosuppressive therapies was either inconsistent or poor for most specialties. New approaches to highlight such risks and assist appropriate pre-immunosuppression screening are needed.

Organizational diagnostics: a systematic approach to identifying technology and workflow issues in clinical settings

Objectives

Healthcare organizations need to rapidly adapt to new technology, policy changes, evolving payment strategies, and other environmental changes. We report on the development and application of a structured methodology to support technology and process improvement in healthcare organizations, Systematic Iterative Organizational Diagnostics (SIOD). SIOD was designed to evaluate clinical work practices, diagnose technology and workflow issues, and recommend potential solutions.

Materials and Methods

SIOD consists of five stages: (1) Background Scan, (2) Engagement Building, (3) Data Acquisition, (4) Data Analysis, and (5) Reporting and Debriefing. Our team applied the SIOD approach in two ambulatory clinics and an integrated ambulatory care center and used SIOD components during an evaluation of a large-scale health information technology transition.

Results

During the initial SIOD application in two ambulatory clinics, five major analysis themes were identified, grounded in the data: putting patients first, reducing the chaos, matching space to function, technology making work harder, and staffing is more than numbers. Additional themes were identified based on SIOD application to a multidisciplinary clinical center. The team also developed contextually grounded recommendations to address issues identified through applying SIOD.

Discussion

The SIOD methodology fills a problem identification gap in existing process improvement systems through an emphasis on issue discovery, holistic clinic functionality, and inclusion of diverse perspectives. SIOD can diagnose issues where approaches as Lean, Six Sigma, and other organizational interventions can be applied.

Conclusion

The complex structure of work and technology in healthcare requires specialized diagnostic strategies to identify and resolve issues, and SIOD fills this need.

The Impact of Changes to an Electronic Admission Order Set on Prescribing and Clinical Outcomes in the Intensive Care Unit

BACKGROUND

Implementation of disease-specific order sets has improved compliance with standards of care for a variety of diseases. Evidence of the impact admission order sets can have on care is limited.

OBJECTIVE

The main purpose of this article is to evaluate the impact of changes made to an electronic critical care admission order set on provider prescribing patterns and clinical outcomes.

METHODS

A retrospective, observational before-and-after exploratory study was performed on adult patients admitted to the medical intensive care unit using the Inpatient Critical Care Admission Order Set. The primary outcome measure was the percentage change in the number of orders for scheduled acetaminophen, a histamine-2 receptor antagonist (H2RA), and lactated ringers at admission before implementation of the revised order set compared with after implementation. Secondary outcomes assessed clinical impact of changes made to the order set.

RESULTS

The addition of a different dosing strategy for a medication already available on the order set (scheduled acetaminophen vs. as needed acetaminophen) had no impact on physician prescribing (0 vs. 0%, p = 1.000). The addition of a new medication class (an H2RA) to the order set significantly increased the number of patients prescribed an H2RA for stress ulcer prophylaxis (0 vs. 20%, p < 0.001). Rearranging the list of maintenance intravenous fluids to make lactated ringers the first fluid option in place of normal saline significantly decreased the number of orders for lactated ringers (17 vs. 4%, p = 0.005). The order set changes had no significant impact on clinical outcomes such as incidence of transaminitis, gastrointestinal bleed, and acute kidney injury.

CONCLUSION

Making changes to an admission order set can impact provider prescribing patterns. The type of change made to the order set, in addition to the specific medication changed, may have an effect on how influential the changes are on prescribing patterns.

Formative Usability Testing Reduces Severe Blood Product Ordering Errors

BACKGROUND

Medical errors in blood product orders and administration are common, especially for pediatric patients. A failure modes and effects analysis in our health care system indicated high risk from the electronic blood ordering process.

OBJECTIVES

There are two objectives of this study as follows:(1) To describe differences in the design of the original blood product orders and order sets in the system (original design), new orders and order sets designed by expert committee (DEC), and a third-version developed through user-centered design (UCD).(2) To compare the number and type of ordering errors, task completion rates, time on task, and user preferences between the original design and that developed via UCD.

METHODS

A multidisciplinary expert committee proposed adjustments to existing blood product order sets resulting in the DEC order set. When that order set was tested with front-line users, persistent failure modes were detected, so orders and order sets were redesigned again via formative usability testing. Front-line users in their native clinical workspaces were observed ordering blood in realistic simulated scenarios using a think-aloud protocol. Iterative adjustments were made between participants. In summative testing, participants were randomized to use the original design or UCD for five simulated scenarios. We evaluated differences in ordering errors, time on task, and users' design preference with two-sample t-tests.

RESULTS

Formative usability testing with 27 providers from seven specialties led to 18 changes made to the DEC to produce the UCD. In summative testing, error-free task completion for the original design was 36%, which increased to 66% in UCD (30%, 95% confidence interval [CI]: 3.9-57%; p = 0.03). Time on task did not vary significantly.

CONCLUSION

UCD led to substantially different blood product orders and order sets than DEC. Users made fewer errors when ordering blood products for pediatric patients in simulated scenarios when using the UCD orders and order sets compared with the original design.