Internal Medicine Resident Engagement with a Laboratory Utilization Dashboard: Mixed Methods Study

Uses of Electronic Health Record Data to Measure the Clinical Learning Environment of Graduate Medical Education Trainees: A Systematic Review

PURPOSE

This study systematically reviews the uses of electronic health record (EHR) data to measure graduate medical education (GME) trainee competencies.

METHOD

In January 2022, the authors conducted a systematic review of original research in MEDLINE from database start to December 31, 2021. The authors searched for articles that used the EHR as their data source and in which the individual GME trainee was the unit of observation and/or unit of analysis. The database query was intentionally broad because an initial survey of pertinent articles identified no unifying Medical Subject Heading terms. Articles were coded and clustered by theme and Accreditation Council for Graduate Medical Education (ACGME) core competency.

RESULTS

The database search yielded 3,540 articles, of which 86 met the study inclusion criteria. Articles clustered into 16 themes, the largest of which were trainee condition experience (17 articles), work patterns (16 articles), and continuity of care (12 articles). Five of the ACGME core competencies were represented (patient care and procedural skills, practice-based learning and improvement, systems-based practice, medical knowledge, and professionalism). In addition, 25 articles assessed the clinical learning environment.

CONCLUSIONS

This review identified 86 articles that used EHR data to measure individual GME trainee competencies, spanning 16 themes and 6 competencies and revealing marked between-trainee variation. The authors propose a digital learning cycle framework that arranges sequentially the uses of EHR data within the cycle of clinical experiential learning central to GME. Three technical components necessary to unlock the potential of EHR data to improve GME are described: measures, attribution, and visualization. Partnerships between GME programs and informatics departments will be pivotal in realizing this opportunity.

Engaging Multidisciplinary Clinical Users in the Design of an Artificial Intelligence-Powered Graphical User Interface for Intensive Care Unit Instability Decision Support

BACKGROUND

Critical instability forecast and treatment can be optimized by artificial intelligence (AI)-enabled clinical decision support. It is important that the user-facing display of AI output facilitates clinical thinking and workflow for all disciplines involved in bedside care.

OBJECTIVES

Our objective is to engage multidisciplinary users (physicians, nurse practitioners, physician assistants) in the development of a graphical user interface (GUI) to present an AI-derived risk score.

METHODS

Intensive care unit (ICU) clinicians participated in focus groups seeking input on instability risk forecast presented in a prototype GUI. Two stratified rounds (three focus groups [only nurses, only providers, then combined]) were moderated by a focus group methodologist. After round 1, GUI design changes were made and presented in round 2. Focus groups were recorded, transcribed, and deidentified transcripts independently coded by three researchers. Codes were coalesced into emerging themes.

RESULTS

Twenty-three ICU clinicians participated (11 nurses, 12 medical providers [3 mid-level and 9 physicians]). Six themes emerged: (1) analytics transparency, (2) graphical interpretability, (3) impact on practice, (4) value of trend synthesis of dynamic patient data, (5) decisional weight (weighing AI output during decision-making), and (6) display location (usability, concerns for patient/family GUI view). Nurses emphasized having GUI objective information to support communication and optimal GUI location. While providers emphasized need for recommendation interpretability and concern for impairing trainee critical thinking. All disciplines valued synthesized views of vital signs, interventions, and risk trends but were skeptical of placing decisional weight on AI output until proven trustworthy.

CONCLUSION

Gaining input from all clinical users is important to consider when designing AI-derived GUIs. Results highlight that health care intelligent decisional support systems technologies need to be transparent on how they work, easy to read and interpret, cause little disruption to current workflow, as well as decisional support components need to be used as an adjunct to human decision-making.

Toward a Learning Health Care System: A Systematic Review and Evidence-Based Conceptual Framework for Implementation of Clinical Analytics in a Digital Hospital

Objective  A learning health care system (LHS) uses routinely collected data to continuously monitor and improve health care outcomes. Little is reported on the challenges and methods used to implement the analytics underpinning an LHS. Our aim was to systematically review the literature for reports of real-time clinical analytics implementation in digital hospitals and to use these findings to synthesize a conceptual framework for LHS implementation.

Methods  Embase, PubMed, and Web of Science databases were searched for clinical analytics derived from electronic health records in adult inpatient and emergency department settings between 2015 and 2021. Evidence was coded from the final study selection that related to (1) dashboard implementation challenges, (2) methods to overcome implementation challenges, and (3) dashboard assessment and impact. The evidences obtained, together with evidence extracted from relevant prior reviews, were mapped to an existing digital health transformation model to derive a conceptual framework for LHS analytics implementation.

Results  A total of 238 candidate articles were reviewed and 14 met inclusion criteria. From the selected studies, we extracted 37 implementation challenges and 64 methods employed to overcome such challenges. We identified common approaches for evaluating the implementation of clinical dashboards. Six studies assessed clinical process outcomes and only four studies evaluated patient health outcomes. A conceptual framework for implementing the analytics of an LHS was developed.

Conclusion  Health care organizations face diverse challenges when trying to implement real-time data analytics. These challenges have shifted over the past decade. While prior reviews identified fundamental information problems, such as data size and complexity, our review uncovered more postpilot challenges, such as supporting diverse users, workflows, and user-interface screens. Our review identified practical methods to overcome these challenges which have been incorporated into a conceptual framework. It is hoped this framework will support health care organizations deploying near-real-time clinical dashboards and progress toward an LHS.

Pandemic Response in the Clinical Laboratory: The Utility of Interactive Dashboards

The ability to access and analyze data is critical to manage a laboratory and respond and adapt to changes, particularly during a pandemic. Data analytic tools can not only improve laboratory operations, but also increase the visibility of the laboratory in the healthcare system and demonstrate the positive impact of the laboratory on patient care. In this article, we describe the creation and utility of laboratory dashboards. Several dashboards were designed to assist with pandemic response. For each dashboard, a stored procedure was created that performed a SQL query of our laboratory information system mirror database. We utilized the business analytics platform, Tableau, for data visualization. Users could modify the data by selecting a specific date range, time window, work shift, institution(s), specific test(s), and/or testing platform(s). Access was controlled by OKTA integration to the host server over the web, behind the hospital firewall. During the April 2020 surge, we saw an increase in blood gas testing and corresponding decrease in non-critical testing such as Vitamin D. At our institution, SARS-CoV-2 molecular testing was performed using four primary platforms, four in-house and one send-out. Weekly and hourly testing volumes as well as turnaround times fluctuated based on reagent availability, new testing requests, staffing, and operational changes. Productivity dashboards indicated that coagulation testing volumes were highest on the third shift and that all three analyzers may not be necessary. Further, specimen volumes and productivity of accessioning staff varied throughout the day. Phlebotomy venipuncture volumes and patient wait times also varied throughout the pandemic. A decrease in ambulatory draws was seen during the surge but after reopening draw volumes, particularly at offsite locations, surpassed prepandemic volumes. We demonstrate that data analytics and interactive dashboards are powerful tools, are helpful in response to a pandemic and lead to improved TAT, supply utilization, staffing and workflows. Furthermore, dashboards provide objective data to review with hospital leadership and promote collaboration.

OUP accepted manuscript

Objectives

(1) Systematically review the literature on computerized audit and feedback (e-A&F) systems in healthcare. (2) Compare features of current systems against e-A&F best practices. (3) Generate hypotheses on how e-A&F systems may impact patient care and outcomes.

Methods

We searched MEDLINE (Ovid), EMBASE (Ovid), and CINAHL (Ebsco) databases to December 31, 2020. Two reviewers independently performed selection, extraction, and quality appraisal (Mixed Methods Appraisal Tool). System features were compared with 18 best practices derived from Clinical Performance Feedback Intervention Theory. We then used realist concepts to generate hypotheses on mechanisms of e-A&F impact. Results are reported in accordance with the PRISMA statement.

Results

Our search yielded 4301 unique articles. We included 88 studies evaluating 65 e-A&F systems, spanning a diverse range of clinical areas, including medical, surgical, general practice, etc. Systems adopted a median of 8 best practices (interquartile range 6–10), with 32 systems providing near real-time feedback data and 20 systems incorporating action planning. High-confidence hypotheses suggested that favorable e-A&F systems prompted specific actions, particularly enabled by timely and role-specific feedback (including patient lists and individual performance data) and embedded action plans, in order to improve system usage, care quality, and patient outcomes.

Conclusions

e-A&F systems continue to be developed for many clinical applications. Yet, several systems still lack basic features recommended by best practice, such as timely feedback and action planning. Systems should focus on actionability, by providing real-time data for feedback that is specific to user roles, with embedded action plans.

Protocol Registration

PROSPERO CRD42016048695.

Use, Perceived Usability, and Barriers to Implementation of a Patient Safety Dashboard Integrated within a Vendor EHR

BACKGROUND

Preventable adverse events continue to be a threat to hospitalized patients. Clinical decision support in the form of dashboards may improve compliance with evidence-based safety practices. However, limited research describes providers' experiences with dashboards integrated into vendor electronic health record (EHR) systems.

OBJECTIVE

This study was aimed to describe providers' use and perceived usability of the Patient Safety Dashboard and discuss barriers and facilitators to implementation.

METHODS

The Patient Safety Dashboard was implemented in a cluster-randomized stepped wedge trial on 12 units in neurology, oncology, and general medicine services over an 18-month period. Use of the Dashboard was tracked during the implementation period and analyzed in-depth for two 1-week periods to gather a detailed representation of use. Providers' perceptions of tool usability were measured using the Health Information Technology Usability Evaluation Scale (rated 1-5). Research assistants conducted field observations throughout the duration of the study to describe use and provide insight into tool adoption.

RESULTS

The Dashboard was used 70% of days the tool was available, with use varying by role, service, and time of day. On general medicine units, nurses logged in throughout the day, with many logins occurring during morning rounds, when not rounding with the care team. Prescribers logged in typically before and after morning rounds. On neurology units, physician assistants accounted for most logins, accessing the Dashboard during daily brief interdisciplinary rounding sessions. Use on oncology units was rare. Satisfaction with the tool was highest for perceived ease of use, with attendings giving the highest rating (4.23). The overall lowest rating was for quality of work life, with nurses rating the tool lowest (2.88).

CONCLUSION

This mixed methods analysis provides insight into the use and usability of a dashboard tool integrated within a vendor EHR and can guide future improvements and more successful implementation of these types of tools.

Self-reported test ordering practices among Canadian internal medicine physicians and trainees: a multicenter cross-sectional survey

Background

Over-testing is a recognized problem, but clinicians usually lack information about their personal test ordering volumes. In the absence of data, clinicians rely on self-perception to inform their test ordering practices. In this study we explore clinician self-perception of diagnostic test ordering intensity.

Methods

We conducted a cross-sectional survey of inpatient General Internal Medicine (GIM) attending physicians and trainees at three Canadian teaching hospitals. We collected information about: self-reported test ordering intensity, perception of colleagues test ordering intensity, and importance of clinical utility, patient comfort, and cost when ordering tests. We compared responses of clinicians who self-identified as high vs low utilizers of diagnostic tests, and attending physicians vs trainees.

Results

Only 15% of inpatient GIM clinicians self-identified as high utilizers of diagnostic tests, while 73% felt that GIM clinicians in aggregate (“others”) order too many tests. Survey respondents identified clinical utility as important when choosing to order tests (selected by 94%), followed by patient comfort (48%) and cost (23%). Self-identified low/average utilizers of diagnostic tests were more likely to report considering cost compared to high utilizers (27% vs 5%, P = 0.04). Attending physicians were more likely to consider patient comfort (70% vs 41%, p = 0.01) and cost (42% vs 17%, p = 0.003) than trainees.

Conclusions

In the absence of data, providers seem to recognize that over investigation is a problem, but few self-identify as being high test utilizers. Moreover, a significant percentage of respondents did not consider cost or patient discomfort when ordering tests. Our findings highlight challenges in reducing over-testing in the current era.

Effect of Social Comparison Feedback on Laboratory Test Ordering for Hospitalized Patients: A Randomized Controlled Trial

BACKGROUND

Social comparison feedback is an increasingly popular strategy that uses performance report cards to modify physician behavior. Our objective was to test the effect of such feedback on the ordering of routine laboratory tests for hospitalized patients, a practice considered overused.

METHODS

This was a single-blinded randomized controlled trial. Between January and June 2016, physicians on six general medicine teams at the Hospital of the University of Pennsylvania were cluster randomized with equal allocation to two arms: (1) those e-mailed a summary of their routine laboratory test ordering vs. the service average for the prior week, linked to a continuously updated personalized dashboard containing patient-level details, and snapshot of the dashboard and (2) those who did not receive the intervention. The primary outcome was the count of routine laboratory test orders placed by a physician per patient-day. We modeled the count of orders by each physician per patient-day after the intervention as a function of trial arm and the physician's order count before the intervention. The count outcome was modeled using negative binomial models with adjustment for clustering within teams.

RESULTS

One hundred and fourteen interns and residents participated. We did not observe a statistically significant difference in adjusted reduction in routine laboratory ordering between the intervention and control physicians (physicians in the intervention group ordered 0.14 fewer tests per patient-day than physicians in the control group, 95% CI - 0.56 to 0.27, p = 0.50). Physicians whose absolute ordering rate deviated from the peer rate by more than 1.0 laboratory test per patient-day reduced their laboratory ordering by 0.80 orders per patient-day (95% CI - 1.58 to - 0.02, p = 0.04).

CONCLUSIONS

Personalized social comparison feedback on routine laboratory ordering did not change targeted behavior among physicians, although there was a significant decrease in orders among participants who deviated more from the peer rate.

TRIAL REGISTRATION

Clinicaltrials.gov registration: #NCT02330289.

Relationship Between Institutional Investment in High-Value Care (HVC) Performance Improvement and Internal Medicine Residents' Perceptions of HVC Training

PURPOSE

To measure the association between institutional investment in high-value care (HVC) performance improvement and resident HVC experiences.

METHOD

The authors analyzed data from two 2014 surveys assessing institutions' investments in HVC performance improvement as reported by program directors (PDs) and residents' perceptions of the frequency of HVC teaching, participation in HVC-focused quality improvement (QI), and views on HVC topics. The authors measured the association between institutional investment and resident-reported experiences using logistic regression, controlling for program and resident characteristics.

RESULTS

The sample included 214 programs and 9,854 residents (59.3% of 361 programs, 55.2% of 17,851 residents surveyed). Most PDs (158/209; 75.6%) reported some support. Residents were more likely to report HVC discussions with faculty at least a few times weekly if they trained in programs that offered HVC-focused faculty development (odds ratio [OR] = 1.19; 95% confidence interval [CI] 1.04-1.37; P = .01), that supported such faculty development (OR = 1.21; 95% CI 1.04-1.41; P = .02), or that provided physician cost-of-care performance data (OR = 1.19; 95% CI 1.03-1.39; P = .02). Residents were more likely to report participation in HVC QI if they trained in programs with a formal HVC curriculum (OR = 1.83; 95% CI 1.48-2.27; P < .001) or with HVC-focused faculty development (OR = 1.46; 95% CI 1.15-1.85; P = .002).

CONCLUSIONS

Institutional investment in HVC-related faculty development and physician feedback on costs of care may increase the frequency of HVC teaching and resident participation in HVC-related QI.

The Use of Individual Provider Performance Reports by US Hospitals

Medicare reimbursement for hospitals is increasingly tied to performance. The use of individual provider performance reports offers the potential to improve clinical outcomes through social comparison, and isolated cases of clinical dashboard uses at specific institutions have been previously reported. However, little is known about overall trends in how hospitals use the electronic health record to track and provide feedback on provider performance. We used data from 2013 to 2015 from the American Hospital Association (AHA) Annual Survey Information Technology Supplement, which asked hospitals if they have used electronic data to create performance profiles. We linked these data to AHA Annual Survey responses for all general adult and pediatric hospitals. Multivariable logistic regression was used to model the odds of use as a function of hospital characteristics. In 2015, 65.8% of the 2334 respondents used performance profiles, whereas 59.3% of the 2077 respondents used them in 2013. Report use was associated with non-profit status (odds ratio [OR], 2.77; 95% confidence interval [CI], 1.94-3.95) compared to for-profit, large hospital size (OR, 2.37; 95% CI, 1.56-3.60) compared to small size, highest quartile of bed-adjusted expenditures compared to bottom quartile (OR, 2.09; 95% CI, 1.55-2.82; P < .01), and participation in a health maintenance organization (OR, 1.50; 95% CI, 1.17-1.90; P < .01) or bundled payment program (OR, 1.61; 95% CI, 1.18- 2.19; P < .01). While a majority of hospitals now use such profiles, more than a third do not. The hospitals that do not use performance profiles may be less well positioned to adapt to value-based payment reforms.