Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods

Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences

OBJECTIVES

To develop and validate a novel measure, action entropy, for assessing the cognitive effort associated with electronic health record (EHR)-based work activities.

MATERIALS AND METHODS

EHR-based audit logs of attending physicians and advanced practice providers (APPs) from four surgical intensive care units in 2019 were included. Neural language models (LMs) were trained and validated separately for attendings' and APPs' action sequences. Action entropy was calculated as the cross-entropy associated with the predicted probability of the next action, based on prior actions. To validate the measure, a matched pairs study was conducted to assess the difference in action entropy during known high cognitive effort scenarios, namely, attention switching between patients and to or from the EHR inbox.

RESULTS

Sixty-five clinicians performing 5 904 429 EHR-based audit log actions on 8956 unique patients were included. All attention switching scenarios were associated with a higher action entropy compared to non-switching scenarios (P < .001), except for the from-inbox switching scenario among APPs. The highest difference among attendings was for the from-inbox attention switching: Action entropy was 1.288 (95% CI, 1.256-1.320) standard deviations (SDs) higher for switching compared to non-switching scenarios. For APPs, the highest difference was for the to-inbox switching, where action entropy was 2.354 (95% CI, 2.311-2.397) SDs higher for switching compared to non-switching scenarios.

DISCUSSION

We developed a LM-based metric, action entropy, for assessing cognitive burden associated with EHR-based actions. The metric showed discriminant validity and statistical significance when evaluated against known situations of high cognitive effort (ie, attention switching). With additional validation, this metric can potentially be used as a screening tool for assessing behavioral action phenotypes that are associated with higher cognitive burden.

CONCLUSION

An LM-based action entropy metric-relying on sequences of EHR actions-offers opportunities for assessing cognitive effort in EHR-based workflows.

A scoping review of rule-based clinical decision support malfunctions

OBJECTIVE

Conduct a scoping review of research studies that describe rule-based clinical decision support (CDS) malfunctions.

MATERIALS AND METHODS

In April 2022, we searched three bibliographic databases (MEDLINE, CINAHL, and Embase) for literature referencing CDS malfunctions. We coded the identified malfunctions according to an existing CDS malfunction taxonomy and added new categories for factors not already captured. We also extracted and summarized information related to the CDS system, such as architecture, data source, and data format.

RESULTS

Twenty-eight articles met inclusion criteria, capturing 130 malfunctions. Architectures used included stand-alone systems (eg, web-based calculator), integrated systems (eg, best practices alerts), and service-oriented architectures (eg, distributed systems like SMART or CDS Hooks). No standards-based CDS malfunctions were identified. The "Cause" category of the original taxonomy includes three new types (organizational policy, hardware error, and data source) and two existing causes were expanded to include additional layers. Only 29 malfunctions (22%) described the potential impact of the malfunction on patient care.

DISCUSSION

While a substantial amount of research on CDS exists, our review indicates there is a limited focus on CDS malfunctions, with even less attention on malfunctions associated with modern delivery architectures such as SMART and CDS Hooks.

CONCLUSION

CDS malfunctions can and do occur across several different care delivery architectures. To account for advances in health information technology, existing taxonomies of CDS malfunctions must be continually updated. This will be especially important for service-oriented architectures, which connect several disparate systems, and are increasing in use.

Physician EHR Time and Visit Volume Following Adoption of Team-Based Documentation Support

Importance

Physicians spend the plurality of active electronic health record (EHR) time on documentation. Excessive documentation limits time spent with patients and is associated with burnout. Organizations need effective strategies to reduce physician documentation burden; however, evidence on team-based documentation (eg, medical scribes) has been limited to small, single-institution studies lacking rigorous estimates of how documentation support changes EHR time and visit volume.

Objectives

To analyze how EHR documentation time and visit volume change following the adoption of team-based documentation approaches.

Design, Setting, and Participants

This national longitudinal cohort study analyzed physician-week EHR metadata from September 2020 through April 2021. A 2-way fixed-effects difference-in-differences regression approach was used to analyze changes in the main outcomes after team-based documentation support adoption. Event study regression models were used to examine variation in changes over time and stratified models to analyze the moderating role of support intensity. The sample included US ambulatory physicians using the EHR. Data were analyzed between October 2022 and September 2023.

Exposure

Team-based documentation support, defined as new onset and consistent use of coauthored documentation with another clinical team member.

Main Outcomes and Measures

The main outcomes included weekly visit volume, EHR documentation time, total EHR time, and EHR time outside clinic hours.

Results

Of 18 265 physicians, 1024 physicians adopted team-based documentation support, with 17 241 comparison physicians who did not adopt such support. The sample included 57.2% primary care physicians, 31.6% medical specialists, and 11.2% surgical specialists; 40.0% practiced in academic settings and 18.4% in outpatient safety-net settings. For adopter physicians, visit volume increased by 6.0% (2.5 visits/wk [95% CI, 1.9-3.0]; P < .001), and documentation time decreased by 9.1% (23.3 min/wk [95% CI, -30.3 to -16.2]; P < .001). Following a 20-week postadoption learning period, visits per week increased by 10.8% and documentation time decreased by 16.2%. Only high-intensity adopters (>40% of note text authored by others) realized reductions in documentation time, both for the full postadoption period (-53.9 min/wk [95% CI, -65.3 to -42.4]; 21.0% decrease; P < .001) and following the learning period (-72.2 min/wk; 28.1% decrease). Low adopters saw no meaningful change in EHR time but realized a similar increase in visit volume.

Conclusions and Relevance

In this national longitudinal cohort study, physicians who adopted team-based documentation experienced increased visit volume and reduced documentation and EHR time, especially after a learning period.

A Scalable and Extensible Logical Data Model of Electronic Health Record Audit Logs for Temporal Data Mining (RNteract): Model Conceptualization and Formulation

BACKGROUND

Increased workload, including workload related to electronic health record (EHR) documentation, is reported as a main contributor to nurse burnout and adversely affects patient safety and nurse satisfaction. Traditional methods for workload analysis are either administrative measures (such as the nurse-patient ratio) that do not represent actual nursing care or are subjective and limited to snapshots of care (eg, time-motion studies). Observing care and testing workflow changes in real time can be obstructive to clinical care. An examination of EHR interactions using EHR audit logs could provide a scalable, unobtrusive way to quantify the nursing workload, at least to the extent that nursing work is represented in EHR documentation. EHR audit logs are extremely complex; however, simple analytical methods cannot discover complex temporal patterns, requiring use of state-of-the-art temporal data-mining approaches. To effectively use these approaches, it is necessary to structure the raw audit logs into a consistent and scalable logical data model that can be consumed by machine learning (ML) algorithms.

OBJECTIVE

We aimed to conceptualize a logical data model for nurse-EHR interactions that would support the future development of temporal ML models based on EHR audit log data.

METHODS

We conducted a preliminary review of EHR audit logs to understand the types of nursing-specific data captured. Using concepts derived from the literature and our previous experience studying temporal patterns in biomedical data, we formulated a logical data model that can describe nurse-EHR interactions, the nurse-intrinsic and situational characteristics that may influence those interactions, and outcomes of relevance to the nursing workload in a scalable and extensible manner.

RESULTS

We describe the data structure and concepts from EHR audit log data associated with nursing workload as a logical data model named RNteract. We conceptually demonstrate how using this logical data model could support temporal unsupervised ML and state-of-the-art artificial intelligence (AI) methods for predictive modeling.

CONCLUSIONS

The RNteract logical data model appears capable of supporting a variety of AI-based systems and should be generalizable to any type of EHR system or health care setting. Quantitatively identifying and analyzing temporal patterns of nurse-EHR interactions is foundational for developing interventions that support the nursing documentation workload and address nurse burnout.

Machine learning to predict notes for chart review in the oncology setting: a proof of concept strategy for improving clinician note-writing

OBJECTIVE

Leverage electronic health record (EHR) audit logs to develop a machine learning (ML) model that predicts which notes a clinician wants to review when seeing oncology patients.

MATERIALS AND METHODS

We trained logistic regression models using note metadata and a Term Frequency Inverse Document Frequency (TF-IDF) text representation. We evaluated performance with precision, recall, F1, AUC, and a clinical qualitative assessment.

RESULTS

The metadata only model achieved an AUC 0.930 and the metadata and TF-IDF model an AUC 0.937. Qualitative assessment revealed a need for better text representation and to further customize predictions for the user.

DISCUSSION

Our model effectively surfaces the top 10 notes a clinician wants to review when seeing an oncology patient. Further studies can characterize different types of clinician users and better tailor the task for different care settings.

CONCLUSION

EHR audit logs can provide important relevance data for training ML models that assist with note-writing in the oncology setting.

The implementation of a multidisciplinary, electronic health record embedded care pathway to improve structured data recording and decrease electronic health record burden

INTRODUCTION

Theoretically, the added value of electronic health records (EHRs) is extensive. Reusable data capture in EHRs could lead to major improvements in quality measurement, scientific research, and decision support. To achieve these goals, structured and standardized recording of healthcare data is a prerequisite. However, time spent on EHRs by physicians is already high. This study evaluated the effect of implementing an EHR embedded care pathway with structured data recording on the EHR burden of physicians.

MATERIALS AND METHODS

Before and six months after implementation, consultations were recorded and analyzed with video-analytic software. Main outcome measures were time spent on specific tasks within the EHR, total consultation duration, and usability indicators such as required mouse clicks and keystrokes. Additionally, a validated questionnaire was completed twice to evaluate changes in physician perception of EHR system factors and documentation process factors.

RESULTS

Total EHR time in initial oncology consultations was significantly reduced by 3.7 min, a 27 % decrease. In contrast, although a decrease of 13 % in consultation duration was observed, no significant effect on EHR time was found in follow-up consultations. Additionally, perceptions of physicians regarding the EHR and documentation improved significantly.

DISCUSSION

Our results have shown that it is possible to achieve structured data capture while simultaneously reducing the EHR burden, which is a decisive factor in end-user acceptance of documentation systems. Proper alignment of structured documentation with workflows is critical for success.

CONCLUSION

Implementing an EHR embedded care pathway with structured documentation led to decreased EHR burden.

Guidance for reporting analyses of metadata on electronic health record use

INTRODUCTION

Research on how people interact with electronic health records (EHRs) increasingly involves the analysis of metadata on EHR use. These metadata can be recorded unobtrusively and capture EHR use at a scale unattainable through direct observation or self-reports. However, there is substantial variation in how metadata on EHR use are recorded, analyzed and described, limiting understanding, replication, and synthesis across studies.

RECOMMENDATIONS

In this perspective, we provide guidance to those working with EHR use metadata by describing 4 common types, how they are recorded, and how they can be aggregated into higher-level measures of EHR use. We also describe guidelines for reporting analyses of EHR use metadata-or measures of EHR use derived from them-to foster clarity, standardization, and reproducibility in this emerging and critical area of research.

Enhanced Patient Portal Engagement Associated with Improved Weight Loss Outcomes in Post-Bariatric Surgery Patients

Background

Bariatric surgery is an effective intervention for obesity, but it requires comprehensive postoperative self-management to achieve optimal outcomes. While patient portals are generally seen as beneficial in engaging patients in health management, the link between their use and post-bariatric surgery weight loss remains unclear.

Objective

This study investigated the association between patient portal engagement and postoperative body mass index (BMI) reduction among bariatric surgery patients.

Methods

This retrospective longitudinal study included patients who underwent Roux-en-Y gastric bypass (RYGB) or sleeve gastrectomy (SG) at Vanderbilt University Medical Center (VUMC) between January 2018 and March 2021. Using generalized estimating equations, we estimated the association between active days of postoperative patient portal use and the reduction of BMI percentage (%BMI) at 3, 6, and 12 months post-surgery. Covariates included duration since surgery, the patient's age at the time of surgery, gender, race and ethnicity, type of bariatric surgery, severity of comorbid conditions, and socioeconomic disadvantage.

Results

The study included 1,415 patients, mostly female (80.9%), with diverse racial and ethnic backgrounds. 805 (56.9%) patients underwent RYGB and 610 (43.1%) underwent SG. By one-year post-surgery, the mean (SD) %BMI reduction was 31.1% (8.3%), and the mean (SD) number of patient portal active days was 61.0 (41.2). A significantly positive association was observed between patient portal engagement and %BMI reduction, with variations revealed over time. Each 10-day increment of active portal use was associated with a 0.57% ([95% CI: 0.42- 0.72], P < .001) and 0.35% ([95% CI: 0.22- 0.49], P < .001) %BMI reduction at 3 and 6 months postoperatively. The association was not statistically significant at 12 months postoperatively (β=-0.07, [95% CI: -0.24- 0.09], P = .54). Various portal functions, including messaging, visits, my record, medical tools, billing, resources, and others, were positively associated with %BMI reduction at 3- and 6-months follow-ups.

Conclusions

Greater patient portal engagement, which may represent stronger adherence to postoperative instructions, better self-management of health, and enhanced communication with care teams, was associated with improved postoperative weight loss. Future investigations are needed to identify important portal features that contribute to the long-term success of weight loss management.

Creating Conversion Factors from EHR Event Log Data: A Comparison of Investigator-Derived and Vendor-Derived Metrics for Primary Care Physicians

Physicians spend a large amount of time with the electronic health record (EHR), which the majority believe contributes to their burnout. However, there are limitedstandardized measures of physician EHR time. Vendor-derived metrics are standardized but may underestimate real-world EHR experience. Investigator-derived metrics may be more reliable but not standardized, particularly with regard to timeout thresholds defining inactivity. This study aimed to enable standardized investigator-derived metrics using conversion factors between raw event log-derived metrics and Signal (Epic System's standardized metric) for primary care physicians. This was an observational, retrospective longitudinal study of EHR raw event logs and Signal data from a quaternary academic medical center and its community affiliates in California, over a 6-month period. The study evaluated 242 physicians over 1370 physician-months, comparing 53.7 million event logs to 6850 Signal metrics, in five different time based metrics. Results show that inactivity thresholds for event log metric derivation that most closely approximate Signal metrics ranged from 90 seconds (Visit Navigator) to 360 seconds ("Pajama time") depending on the metric. Based on this data, conversion factors for investigator-derived metrics across a wide range of inactivity thresholds, via comparison with Signal metrics, are provided which may allow researchers to consistently quantify EHR experience.

Clinical outcome prediction using observational supervision with electronic health records and audit logs

OBJECTIVE

Audit logs in electronic health record (EHR) systems capture interactions of providers with clinical data. We determine if machine learning (ML) models trained using audit logs in conjunction with clinical data ("observational supervision") outperform ML models trained using clinical data alone in clinical outcome prediction tasks, and whether they are more robust to temporal distribution shifts in the data.

MATERIALS AND METHODS

Using clinical and audit log data from Stanford Healthcare, we trained and evaluated various ML models including logistic regression, support vector machine (SVM) classifiers, neural networks, random forests, and gradient boosted machines (GBMs) on clinical EHR data, with and without audit logs for two clinical outcome prediction tasks: major adverse kidney events within 120 days of ICU admission (MAKE-120) in acute kidney injury (AKI) patients and 30-day readmission in acute stroke patients. We further tested the best performing models using patient data acquired during different time-intervals to evaluate the impact of temporal distribution shifts on model performance.

RESULTS

Performance generally improved for all models when trained with clinical EHR data and audit log data compared with those trained with only clinical EHR data, with GBMs tending to have the overall best performance. GBMs trained with clinical EHR data and audit logs outperformed GBMs trained without audit logs in both clinical outcome prediction tasks: AUROC 0.88 (95% CI: 0.85-0.91) vs. 0.79 (95% CI: 0.77-0.81), respectively, for MAKE-120 prediction in AKI patients, and AUROC 0.74 (95% CI: 0.71-0.77) vs. 0.63 (95% CI: 0.62-0.64), respectively, for 30-day readmission prediction in acute stroke patients. The performance of GBM models trained using audit log and clinical data degraded less in later time-intervals than models trained using only clinical data.

CONCLUSION

Observational supervision with audit logs improved the performance of ML models trained to predict important clinical outcomes in patients with AKI and acute stroke, and improved robustness to temporal distribution shifts.

Differences in Health Professionals' Engagement With Electronic Health Records Based on Inpatient Race and Ethnicity

Importance

US health professionals devote a large amount of effort to engaging with patients' electronic health records (EHRs) to deliver care. It is unknown whether patients with different racial and ethnic backgrounds receive equal EHR engagement.

Objective

To investigate whether there are differences in the level of health professionals' EHR engagement for hospitalized patients according to race or ethnicity during inpatient care.

Design, Setting, and Participants

This cross-sectional study analyzed EHR access log data from 2 major medical institutions, Vanderbilt University Medical Center (VUMC) and Northwestern Medicine (NW Medicine), over a 3-year period from January 1, 2018, to December 31, 2020. The study included all adult patients (aged ≥18 years) who were discharged alive after hospitalization for at least 24 hours. The data were analyzed between August 15, 2022, and March 15, 2023.

Exposures

The actions of health professionals in each patient's EHR were based on EHR access log data. Covariates included patients' demographic information, socioeconomic characteristics, and comorbidities.

Main Outcomes and Measures

The primary outcome was the quantity of EHR engagement, as defined by the average number of EHR actions performed by health professionals within a patient's EHR per hour during the patient's hospital stay. Proportional odds logistic regression was applied based on outcome quartiles.

Results

A total of 243 416 adult patients were included from VUMC (mean [SD] age, 51.7 [19.2] years; 54.9% female and 45.1% male; 14.8% Black, 4.9% Hispanic, 77.7% White, and 2.6% other races and ethnicities) and NW Medicine (mean [SD] age, 52.8 [20.6] years; 65.2% female and 34.8% male; 11.7% Black, 12.1% Hispanic, 69.2% White, and 7.0% other races and ethnicities). When combining Black, Hispanic, or other race and ethnicity patients into 1 group, these patients were significantly less likely to receive a higher amount of EHR engagement compared with White patients (adjusted odds ratios, 0.86 [95% CI, 0.83-0.88; P < .001] for VUMC and 0.90 [95% CI, 0.88-0.92; P < .001] for NW Medicine). However, a reduction in this difference was observed from 2018 to 2020.

Conclusions and Relevance

In this cross-sectional study of inpatient EHR engagement, the findings highlight differences in how health professionals distribute their efforts to patients' EHRs, as well as a method to measure these differences. Further investigations are needed to determine whether and how EHR engagement differences are correlated with health care outcomes.

The Role of Data Science in Closing the Implementation Gap

Data science has the potential to greatly enhance efforts to translate evidence into practice in critical care. The intensive care unit is a data-rich environment enabling insight into both patient-level care patterns and clinician-level treatment patterns. By applying artificial intelligence to these novel data sources, implementation strategies can be tailored to individual patients, individual clinicians, and individual situations, revealing when evidence-based practices are missed and facilitating context-sensitive clinical decision support. To achieve these goals, technology developers should work closely with clinicians to create unbiased applications that are integrated into the clinical workflow.

A guide to mitigating audit log-related risk in medical professional liability cases

Following the American Recovery and Reinvestment Act in 2009, use of electronic health records (EHRs) has become ubiquitous. Accordingly, one should expect most medical professional liability cases to involve review of patient records produced from EHRs. When questions arise regarding who was involved in care of a patient, what they knew and when, or the meaning, completeness, integrity, validity, timeliness, confidentiality, accuracy, or legitimacy of data, or ways that the EHR's user interface or automated clinical decision support tools may have contributed to the alleged events, one often turns to the EHR and its audit log. This manuscript discusses lines of defense incorporated into the design, development, implementation, and use of EHRs to ensure their integrity and the types of EHR transaction logs (e.g., audit log) that exist. Using these logs can help one answer questions that often arise in medical malpractice cases. Finally, there are "best practices" surrounding EHR audit logs that health care organizations should implement. When used appropriately, EHRs and their audit logs provide another source of information to help hospital risk managers, legal counsel, and EHR expert witnesses to investigate adverse incidents and, if needed, prosecute or defend clinicians and/or health care organizations involved in the patient's care.

Effects of a QI intervention on pediatric asthma treatment using patient outcomes and workflow in an emergency department

OBJECTIVE

Evaluate a nurse-initiated quality improvement (QI) intervention aimed at enhancing asthma treatment in a pediatric emergency department (ED), utilizing outcomes and workflow.

METHODS

We evaluated the impact of QI interventions for pediatric patients presenting to the ED with asthma with pre-post analysis. A pediatric asthma score (PAS) of >8 indicated moderate to severe asthma. This secondary analysis of the electronic health record (EHR), evaluated on 1) patient outcomes (time to clinical treatment, ED length of stay [EDLOS], admissions and discharges home), 2) clinical workflow.

RESULTS

We compared 886 visits occurring between 01/01/2015 and 09/27/2015 (pre-implementation period) with 752 visits between 01/01/2016 and 09/27/2016 (post-implementation). Time to first documentation of PAS was decreased post-intervention (p<.001) by >30 min (75 ± 57 to 39 ± 54 min). There were significant decreases in time to treatment with both steroid and bronchodilator administration (both p<.001). EDLOS did not significantly change. Based on acuity level, those discharged home from the ED with high acuity (PAS score ≥8), had a significant decrease in time to initial PAS, steroid and bronchodilator use and EDLOS. Of those with high acuity who were admitted to the hospital, there was a difference pre- to post-implementation, in time to first PAS (p<.05), but not to treatment. Workflow visualization provided additional insights and detailed (task level) comparisons of the timing of ED activities.

CONCLUSIONS

Nurse-initiated ED interventions, can significantly improve the timeliness of pediatric asthma evaluation and treatment. Examining workflow along with the outcomes, can better inform QI evaluations and clinical management.

Using Time Series Clustering to Segment and Infer Emergency Department Nursing Shifts from Electronic Health Record Log Files

Few computational approaches exist for abstracting electronic health record (EHR) log files into clinically meaningful phenomena like clinician shifts. Because shifts are a fundamental unit of work recognized in clinical settings, shifts may serve as a primary unit of analysis in the study of documentation burden. We conducted a proof- of-concept study to investigate the feasibility of a novel approach using time series clustering to segment and infer clinician shifts from EHR log files. From 33,535,585 events captured between April-June 2021, we computationally identified 43,911 potential shifts among 2,285 (74.2%) emergency department nurses. On average, computationally-identified shifts were 10.6±3.1 hours long. Based on data distributions, we classified these shifts based on type: day, evening, night; and length: 12-hour, 8-hour, other. We validated our method through manual chart review of computationally-identified 12-hour shifts achieving 92.0% accuracy. Preliminary results suggest unsupervised clustering methods may be a reasonable approach for rapidly identifying clinician shifts.

Effect of restricting electronic health records on clinician efficiency: substudy of a randomized clinical trial

A prior randomized controlled trial (RCT) showed no significant difference in wrong-patient errors between clinicians assigned to a restricted electronic health record (EHR) configuration (limiting to 1 record open at a time) versus an unrestricted EHR configuration (allowing up to 4 records open concurrently). However, it is unknown whether an unrestricted EHR configuration is more efficient. This substudy of the RCT compared clinician efficiency between EHR configurations using objective measures. All clinicians who logged onto the EHR during the substudy period were included. The primary outcome measure of efficiency was total active minutes per day. Counts were extracted from audit log data, and mixed-effects negative binomial regression was performed to determine differences between randomized groups. Incidence rate ratios (IRRs) were calculated with 95% confidence intervals (CIs). Among a total of 2556 clinicians, there was no significant difference between unrestricted and restricted groups in total active minutes per day (115.1 vs 113.3 min, respectively; IRR, 0.99; 95% CI, 0.93-1.06), overall or by clinician type and practice area.

Disparities in eye clinic patient encounters among patients requiring language interpreter services

BACKGROUND

Communication barriers are a major cause of health disparities for patients with limited English proficiency (LEP). Medical interpreters play an important role in bridging this gap, however the impact of interpreters on outpatient eye center visits has not been studied. We aimed to evaluate the differences in length of eyecare visits between LEP patients self-identifying as requiring a medical interpreter and English speakers at a tertiary, safety-net hospital in the United States.

METHODS

A retrospective review of patient encounter metrics collected by our electronic medical record was conducted for all visits between January 1, 2016 and March 13, 2020. Patient demographics, primary language spoken, self-identified need for interpreter and encounter characteristics including new patient status, patient time waiting for providers and time in room were collected. We compared visit times by patient's self-identification of need for an interpreter, with our main outcomes being time spent with ophthalmic technician, time spent with eyecare provider, and time waiting for eyecare provider. Interpreter services at our hospital are typically remote (via phone or video).

RESULTS

A total of 87,157 patient encounters were analyzed, of which 26,443 (30.3%) involved LEP patients identifying as requiring an interpreter. After adjusting for patient age at visit, new patient status, physician status (attending or resident), and repeated patient visits, there was no difference in the length of time spent with technician or physician, or time spent waiting for physician, between English speakers and patients identifying as needing an interpreter. Patients who self-identified as requiring an interpreter were more likely to have an after-visit summary printed for them, and were also more likely to keep their appointment once it was made when compared to English speakers.

CONCLUSIONS

Encounters with LEP patients who identify as requiring an interpreter were expected to be longer than those who did not indicate need for an interpreter, however we found that there was no difference in the length of time spent with technician or physician. This suggests providers may adjust their communication strategy during encounters with LEP patients identifying as needing an interpreter. Eyecare providers must be aware of this to prevent negative impacts on patient care. Equally important, healthcare systems should consider ways to prevent unreimbursed extra time from being a financial disincentive for seeing patients who request interpreter services.

Monitoring the Implementation of Tobacco Cessation Support Tools: Using Novel Electronic Health Record Activity Metrics

BACKGROUND

Clinical decision support (CDS) tools in electronic health records (EHRs) are often used as core strategies to support quality improvement programs in the clinical setting. Monitoring the impact (intended and unintended) of these tools is crucial for program evaluation and adaptation. Existing approaches for monitoring typically rely on health care providers' self-reports or direct observation of clinical workflows, which require substantial data collection efforts and are prone to reporting bias.

OBJECTIVE

This study aims to develop a novel monitoring method leveraging EHR activity data and demonstrate its use in monitoring the CDS tools implemented by a tobacco cessation program sponsored by the National Cancer Institute's Cancer Center Cessation Initiative (C3I).

METHODS

We developed EHR-based metrics to monitor the implementation of two CDS tools: (1) a screening alert reminding clinic staff to complete the smoking assessment and (2) a support alert prompting health care providers to discuss support and treatment options, including referral to a cessation clinic. Using EHR activity data, we measured the completion (encounter-level alert completion rate) and burden (the number of times an alert was fired before completion and time spent handling the alert) of the CDS tools. We report metrics tracked for 12 months post implementation, comparing 7 cancer clinics (2 clinics implemented the screening alert and 5 implemented both alerts) within a C3I center, and identify areas to improve alert design and adoption.

RESULTS

The screening alert fired in 5121 encounters during the 12 months post implementation. The encounter-level alert completion rate (clinic staff acknowledged completion of screening in EHR: 0.55; clinic staff completed EHR documentation of screening results: 0.32) remained stable over time but varied considerably across clinics. The support alert fired in 1074 encounters during the 12 months. Providers acted upon (ie, not postponed) the support alert in 87.3% (n=938) of encounters, identified a patient ready to quit in 12% (n=129) of encounters, and ordered a referral to the cessation clinic in 2% (n=22) of encounters. With respect to alert burden, on average, both alerts fired over 2 times (screening alert: 2.7; support alert: 2.1) before completion; time spent postponing the screening alert was similar to completing (52 vs 53 seconds) the alert, and time spent postponing the support alert was more than completing (67 vs 50 seconds) the alert per encounter. These findings inform four areas where the alert design and use can be improved: (1) improving alert adoption and completion through local adaptation, (2) improving support alert efficacy by additional strategies including training in provider-patient communication, (3) improving the accuracy of tracking for alert completion, and (4) balancing alert efficacy with the burden.

CONCLUSIONS

EHR activity metrics were able to monitor the success and burden of tobacco cessation alerts, allowing for a more nuanced understanding of potential trade-offs associated with alert implementation. These metrics can be used to guide implementation adaptation and are scalable across diverse settings.

A Novel Use of Bar Code Medication Administration Data to Assess Nurse Staffing and Workload

OBJECTIVE

The aim of the study is to introduce an innovative use of bar code medication administration (BCMA) data, medication pass analysis, that allows for the examination of nurse staffing and workload using data generated during regular nursing workflow.

METHODS

Using 1 year (October 1, 2014-September 30, 2015) of BCMA data for 11 acute care units in one Veterans Affairs Medical Center, we determined the peak time for scheduled medications and included medications scheduled for and administered within 2 hours of that time in analyses. We established for each staff member their daily peak-time medication pass characteristics (number of patients, number of peak-time scheduled medications, duration, start time), generated unit-level descriptive statistics, examined staffing trends, and estimated linear mixed-effects models of duration and start time.

RESULTS

As the most frequent (39.7%) scheduled medication time, 9:00 was the peak-time medication pass; 98.3% of patients (87.3% of patient-days) had a 9:00 medication. Use of nursing roles and number of patients per staff varied across units and over time. Number of patients, number of medications, and unit-level factors explained significant variability in registered nurse (RN) medication pass duration (conditional R2  = 0.237; marginal R2  = 0.199; intraclass correlation = 0.05). On average, an RN and a licensed practical nurse (LPN) with four patients, each with six medications, would be expected to take 70 and 74 minutes, respectively, to complete the medication pass. On a unit with median 10 patients per LPN, the median duration (127 minutes) represents untimely medication administration on more than half of staff days. With each additional patient assigned to a nurse, average start time was earlier by 4.2 minutes for RNs and 1.4 minutes for LPNs.

CONCLUSION

Medication pass analysis of BCMA data can provide health systems a means for assessing variations in staffing, workload, and nursing practice using data generated during routine patient care activities.

Using event logs to observe interactions with electronic health records: an updated scoping review shows increasing use of vendor-derived measures

OBJECTIVE

The aim of this article is to compare the aims, measures, methods, limitations, and scope of studies that employ vendor-derived and investigator-derived measures of electronic health record (EHR) use, and to assess measure consistency across studies.

MATERIALS AND METHODS

We searched PubMed for articles published between July 2019 and December 2021 that employed measures of EHR use derived from EHR event logs. We coded the aims, measures, methods, limitations, and scope of each article and compared articles employing vendor-derived and investigator-derived measures.

RESULTS

One hundred and two articles met inclusion criteria; 40 employed vendor-derived measures, 61 employed investigator-derived measures, and 1 employed both. Studies employing vendor-derived measures were more likely than those employing investigator-derived measures to observe EHR use only in ambulatory settings (83% vs 48%, P = .002) and only by physicians or advanced practice providers (100% vs 54% of studies, P < .001). Studies employing vendor-derived measures were also more likely to measure durations of EHR use (P < .001 for 6 different activities), but definitions of measures such as time outside scheduled hours varied widely. Eight articles reported measure validation. The reported limitations of vendor-derived measures included measure transparency and availability for certain clinical settings and roles.

DISCUSSION

Vendor-derived measures are increasingly used to study EHR use, but only by certain clinical roles. Although poorly validated and variously defined, both vendor- and investigator-derived measures of EHR time are widely reported.

CONCLUSION

The number of studies using event logs to observe EHR use continues to grow, but with inconsistent measure definitions and significant differences between studies that employ vendor-derived and investigator-derived measures.

Team is brain: leveraging EHR audit log data for new insights into acute care processes

OBJECTIVE

To determine whether novel measures of contextual factors from multi-site electronic health record (EHR) audit log data can explain variation in clinical process outcomes.

MATERIALS AND METHODS

We selected one widely-used process outcome: emergency department (ED)-based team time to deliver tissue plasminogen activator (tPA) to patients with acute ischemic stroke (AIS). We evaluated Epic audit log data (that tracks EHR user-interactions) for 3052 AIS patients aged 18+ who received tPA after presenting to an ED at three Northern California health systems (Stanford Health Care, UCSF Health, and Kaiser Permanente Northern California). Our primary outcome was door-to-needle time (DNT) and we assessed bivariate and multivariate relationships with six audit log-derived measures of treatment team busyness and prior team experience.

RESULTS

Prior team experience was consistently associated with shorter DNT; teams with greater prior experience specifically on AIS cases had shorter DNT (minutes) across all sites: (Site 1: -94.73, 95% CI: -129.53 to 59.92; Site 2: -80.93, 95% CI: -130.43 to 31.43; Site 3: -42.95, 95% CI: -62.73 to 23.17). Teams with greater prior experience across all types of cases also had shorter DNT at two sites: (Site 1: -6.96, 95% CI: -14.56 to 0.65; Site 2: -19.16, 95% CI: -36.15 to 2.16; Site 3: -11.07, 95% CI: -17.39 to 4.74). Team busyness was not consistently associated with DNT across study sites.

CONCLUSIONS

EHR audit log data offers a novel, scalable approach to measure key contextual factors relevant to clinical process outcomes across multiple sites. Audit log-based measures of team experience were associated with better process outcomes for AIS care, suggesting opportunities to study underlying mechanisms and improve care through deliberate training, team-building, and scheduling to maximize team experience.

Using electronic health record audit log data for research: insights from early efforts

Electronic health record audit logs capture a time-sequenced record of clinician activities while using the system. Audit log data therefore facilitate unobtrusive measurement at scale of clinical work activities and workflow as well as derivative, behavioral proxies (eg, teamwork). Given its considerable research potential, studies leveraging these data have burgeoned. As the field has matured, the challenges of using the data to answer significant research questions have come into focus. In this Perspective, we draw on our research experiences and insights from the broader audit log literature to advance audit log research. Specifically, we make 2 complementary recommendations that would facilitate substantial progress toward audit log-based measures that are: (1) transparent and validated, (2) standardized to allow for multisite studies, (3) sensitive to meaningful variability, (4) broader in scope to capture key aspects of clinical work including teamwork and coordination, and (5) linked to patient and clinical outcomes.

Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach

Raw audit logs provide a comprehensive record of clinicians' activities on an electronic health record (EHR) and have considerable potential for studying clinician behaviors. However, research using raw audit logs is limited because they lack context for clinical tasks, leading to difficulties in interpretation. We describe a novel unsupervised approach using the comparison and visualization of EHR action embeddings to learn context and structure from raw audit log activities. Using a dataset of 15 767 634 raw audit log actions performed by 88 intern physicians over 6 months of EHR use across inpatient and outpatient settings, we demonstrated that embeddings can be used to learn the situated context for EHR-based work activities, identify discrete clinical workflows, and discern activities typically performed across diverse contexts. Our approach represents an important methodological advance in raw audit log research, facilitating the future development of metrics and predictive models to measure clinician behaviors at the macroscale.

Mining context-aware resource profiles in the presence of multitasking

Healthcare organisations are becoming increasingly aware of the need to improve their care processes and to manage their scarce resources efficiently to secure high-quality care standards. As these processes are knowledge-intensive and heavily depend on human resources, a comprehensive understanding of the complex relationship between processes and resources is indispensable for efficient resource management. Organisational mining, a subfield of Process Mining, reveals insights into how (human) resources organise their work based on analysing process execution data recorded in Health Information Systems (HIS). This can be used to, e.g., discover resource profiles which are groups of resources performing similar activity instances, providing an extensive overview of resource behaviour within healthcare organisations. Healthcare managers can employ these insights to allocate their resources efficiently, e.g., by improving the scheduling and staffing of nurses. Existing resource profiling algorithms are limited in their ability to apprehend the complex relationship between processes and resources because they do not take into account the context in which activities were executed, particularly in the context of multitasking. Therefore, this paper introduces ResProMin-MT to discover context-aware resource profiles in the presence of multitasking. In contrast to the state-of-the-art, ResProMin-MT is capable of taking into account more complex contextual activity dimensions, such as activity durations and the degree of multitasking by resources. We demonstrate the feasibility of our method within a real-life healthcare context, validated by medical domain experts.

Accuracy of Physician Electronic Health Record Usage Analytics using Clinical Test Cases

Usage log data are an important data source for characterizing the potential burden related to use of the electronic health record (EHR) system. However, the utility of this data source has been hindered by concerns related to the real-world validity and accuracy of the data. While time-motion studies have historically been used to address this concern, the restrictions caused by the pandemic have made it difficult to carry out these studies in-person. In this regard, we introduce a practical approach for conducting validation studies for usage log data in a controlled environment. By developing test runs based on clinical workflows and conducting them within a test EHR environment, it allows for both comparison of the recorded timings and retrospective investigation of any discrepancies. In this case report, we describe the utility of this approach for validating our physician EHR usage logs at a large academic teaching mental health hospital in Canada. A total of 10 test runs were conducted across 3 days to validate 8 EHR usage log metrics, finding differences between recorded measurements and the usage analytics platform ranging from 9 to 60%.

Primary Care Physician Gender and Electronic Health Record Workload

BACKGROUND

Prior research indicates that female physicians spend more time working in the electronic health record (EHR) than do male physicians.

OBJECTIVE

To examine gender differences in EHR usage among primary care physicians and identify potential causes for those differences.

DESIGN

Retrospective study of EHR usage by primary care physicians (PCPs) in an academic hospital system.

PARTICIPANTS

One hundred twenty-five primary care physicians INTERVENTIONS: N/A MAIN MEASURES: EHR usage including time spent working and volume of staff messages and patient messages.

KEY RESULTS

After adjusting for panel size and appointment volume, female PCPs spend 20% more time (1.9 h/month) in the EHR inbasket and 22% more time (3.7 h/month) on notes than do their male colleagues (p values 0.02 and 0.04, respectively). Female PCPs receive 24% more staff messages (9.6 messages/month), and 26% more patient messages (51.5 messages/month) (p values 0.03 and 0.004, respectively). The differences in EHR time are not explained by the percentage of female patients in a PCP's panel.

CONCLUSIONS

Female physicians spend more time working in their EHR inbaskets because both staff and patients make more requests of female PCPs. These differential EHR burdens may contribute to higher burnout rates in female PCPs.

Time Spent by Intensive Care Unit Nurses on the Electronic Health Record

BACKGROUND

The amount of time spent on the electronic health record is often cited as a contributing factor to burnout and work-related stress in nurses. Increased electronic health record use also reduces the time nurses have for direct contact with patients and families. There has been minimal investigation into the amount of time intensive care unit nurses spend on the electronic health record.

OBJECTIVE

To quantify the amount of time spent by intensive care unit nurses on the electronic health record.

METHODS

In this observational study, active electronic health record use time was analyzed for 317 intensive care unit nurses in a single institution from January 2019 through July 2020. Monthly data on electronic health record use by nurses in the medical, neurosurgical, and surgical-trauma intensive care units were evaluated.

RESULTS

Full-time intensive care unit nurses spent 28.9 hours per month on the electronic health record, about 17.5% of their clinical shift, for a total of 346.3 hours per year. Part-time nurses and those working as needed spent 20.5 hours per month (17.6%) and 7.4 hours per month (14.2%) on the electronic health record, respectively. Neurosurgical and medical intensive care unit nurses spent 25.0 hours and 19.9 hours per month, respectively. Nurses averaged 23 clicks per minute during use. Most time was spent on the task of documentation at 12.3 hours per month, which was followed by medical record review at 2.6 hours per month.

CONCLUSION

Intensive care unit nurses spend at least 17% of their shift on the electronic health record, primarily on documentation. Future interventions are necessary to reduce time spent on the electronic health record and to improve nurse and patient satisfaction.

Methodology paper for the General Medicine Inpatient Initiative Medical Education Database (GEMINI MedED): a retrospective cohort study of internal medicine resident case-mix, clinical care and patient outcomes

INTRODUCTION

Unwarranted variation in patient care among physicians is associated with negative patient outcomes and increased healthcare costs. Care variation likely also exists for resident physicians. Despite the global movement towards outcomes-based and competency-based medical education, current assessment strategies in residency do not routinely incorporate clinical outcomes. The widespread use of electronic health records (EHRs) may enable the implementation of in-training assessments that incorporate clinical care and patient outcomes.

METHODS AND ANALYSIS

The General Medicine Inpatient Initiative Medical Education Database (GEMINI MedED) is a retrospective cohort study of senior residents (postgraduate year 2/3) enrolled in the University of Toronto Internal Medicine (IM) programme between 1 April 2010 and 31 December 2020. This study focuses on senior IM residents and patients they admit overnight to four academic hospitals. Senior IM residents are responsible for overseeing all overnight admissions; thus, care processes and outcomes for these clinical encounters can be at least partially attributed to the care they provide. Call schedules from each hospital, which list the date, location and senior resident on-call, will be used to link senior residents to EHR data of patients admitted during their on-call shifts. Patient data will be derived from the GEMINI database, which contains administrative (eg, demographic and disposition) and clinical data (eg, laboratory and radiological investigation results) for patients admitted to IM at the four academic hospitals. Overall, this study will examine three domains of resident practice: (1) case-mix variation across residents, hospitals and academic year, (2) resident-sensitive quality measures (EHR-derived metrics that are partially attributable to resident care) and (3) variations in patient outcomes across residents and factors that contribute to such variation.

ETHICS AND DISSEMINATION

GEMINI MedED was approved by the University of Toronto Ethics Board (RIS#39339). Results from this study will be presented in academic conferences and peer-reviewed journals.

Evaluating the Impact of a Point-of-Care Cardiometabolic Clinical Decision Support Tool on Clinical Efficiency Using Electronic Health Record Audit Log Data: Algorithm Development and Validation

Background

Electronic health record (EHR) systems are becoming increasingly complicated, leading to concerns about rising physician burnout, particularly for primary care physicians (PCPs). Managing the most common cardiometabolic chronic conditions by PCPs during a limited clinical time with a patient is challenging.

Objective

This study aimed to evaluate a Cardiometabolic Sutter Health Advanced Reengineered Encounter (CM-SHARE), a web-based application to visualize key EHR data, on the EHR use efficiency.

Methods

We developed algorithms to identify key clinic workflow measures (eg, total encounter time, total physician time in the examination room, and physician EHR time in the examination room) using audit data, and we validated and calibrated the measures with time-motion data. We used a pre-post parallel design to identify propensity score–matched CM-SHARE users (cases), nonusers (controls), and nested-matched patients. Cardiometabolic encounters from matched case and control patients were used for the workflow evaluation. Outcome measures were compared between the cases and controls. We applied this approach separately to both the CM-SHARE pilot and spread phases.

Results

Time-motion observation was conducted on 101 primary care encounters for 9 PCPs in 3 clinics. There was little difference (<0.8 minutes) between the audit data–derived workflow measures and the time-motion observation. Two key unobservable times from audit data, physician entry into and exiting the examination room, were imputed based on time-motion studies. CM-SHARE was launched with 6 pilot PCPs in April 2016. During the prestudy period (April 1, 2015, to April 1, 2016), 870 control patients with 2845 encounters were matched with 870 case patients and encounters, and 727 case patients with 852 encounters were matched with 727 control patients and 3754 encounters in the poststudy period (June 1, 2016, to June 30, 2017). Total encounter time was slightly shorter (mean −2.7, SD 1.4 minutes, 95% CI −4.7 to −0.9; mean –1.6, SD 1.1 minutes, 95% CI −3.2 to −0.1) for cases than controls for both periods. CM-SHARE saves physicians approximately 2 minutes EHR time in the examination room (mean −2.0, SD 1.3, 95% CI −3.4 to −0.9) compared with prestudy period and poststudy period controls (mean −1.9, SD 0.9, 95% CI −3.8 to −0.5). In the spread phase, 48 CM-SHARE spread PCPs were matched with 84 control PCPs and 1272 cases with 3412 control patients, having 1119 and 4240 encounters, respectively. A significant reduction in total encounter time for the CM-SHARE group was observed for short appointments (≤20 minutes; 5.3-minute reduction on average) only. Total physician EHR time was significantly reduced for both longer and shorter appointments (17%-33% reductions).

Conclusions

Combining EHR audit log files and clinical information, our approach offers an innovative and scalable method and new measures that can be used to evaluate clinical EHR efficiency of digital tools used in clinical settings.

Identifying and adapting interventions to reduce documentation burden and improve nurses' efficiency in using electronic health record systems (The IDEA Study): protocol for a mixed methods study

Background

Although EHR systems have become a critical part of clinical care, nurses are experiencing a growing burden due to documentation requirements, taking time away from other important clinical activities. There is a need to address the inefficiencies and challenges that nurses face when documenting in and using EHRs. The objective of this study is to engage nurses in generating ideas on how organizations can support and optimize nurses’ experiences with their EHR systems, thereby improving efficiency and reducing EHR-related burden. This work will ensure the identified solutions are grounded in nurses’ perspectives and experiences and will address their specific EHR-related needs.

Methods

This mixed methods study will consist of three phases. Phase 1 will evaluate the accuracy of the EHR system’s analytics platform in capturing how nurses utilize the system in real-time for tasks such as documentation, chart review, and medication reconciliation. Phase 2 consists of a retrospective analysis of the nursing-specific analytics platform and focus groups with nurses to understand and contextualize their usage patterns. These focus groups will also be used to identify areas for improvement in the utilization of the EHR. Phase 3 will include focus groups with nurses to generate and adapt potential interventions to address the areas for improvement and assess the perceived relevance, feasibility, and impact of the potential interventions.

Discussion

This work will generate insights on addressing nurses’ EHR-related burden and burnout. By understanding and contextualizing inefficiencies and current practices, opportunities to improve EHR systems for nursing professional practice will be identified. The study findings will inform the co-design and implementation of interventions that will support adoption and impact. Future work will include the evaluation of the developed interventions, and research on scaling and disseminating the interventions for use in different organizations, EHR systems, and jurisdictions in Canada.

Quantifying the Electronic Health Record Burden in Head and Neck Cancer Care

Background  Although the main task of health care providers is to provide patient care, studies show that increasing amounts of time are spent on documentation.

Objective  To quantify the time and effort spent on the electronic health record (EHR) in head and neck cancer care.

Methods  Cross-sectional time–motion study. Primary outcomes were the percentages of time spent on the EHR and the three main tasks (chart review, input, placing orders), number of mouse events, and keystrokes per consultation. Secondary outcome measures were perceptions of health care providers regarding EHR documentation and satisfaction.

Results  In total, 44.0% of initial oncological consultation (IOC) duration and 30.7% of follow-up consultation (FUC) duration are spent on EHR tasks. During 80.0% of an IOC and 67.9% of a FUC, the patient and provider were actively communicating. Providers required 593 mouse events and 1,664 keystrokes per IOC and 140 mouse events and 597 keystrokes per FUC, indicating almost 13 mouse clicks and close to 40 keystrokes for every minute of consultation time. Less than a quarter of providers indicated that there is enough time for documentation.

Conclusion  This study quantifies the widespread concern of high documentation burden for health care providers in oncology, which has been related to burnout and a decrease of patient–clinician interaction. Despite excessive time and effort spent on the EHR, health care providers still felt this was insufficient for proper documentation. However, the need for accurate and complete documentation is high, as reuse of information becomes increasingly important. The challenge is to decrease the documentation burden while increasing the quality of EHR data.

Measuring impact of a quality improvement initiative on glaucoma clinic flow using an automated real-time locating system

Background

Lean methodology helps maximize value by reducing waste, first by defining what value and waste are in a system. In ophthalmology clinics, value is determined by the number of patients flowing through the clinic for a given time. We aimed to increase value using a lean-methodology guided policy change, then assessed its impact on clinic flow using an automated radiofrequency identification (RFID) based real-time locating system (RTLS).

Methods

A total of 6813 clinical visits occurred at a single academic institution’s outpatient glaucoma clinic between January 5, 2018 to July 3, 2018. Over that period, 1589 patients comprising 1972 (29%) of visits were enrolled, with 1031 clinical visits occurring before and 941 visits after a policy change. The original policy was to refract all patients that improved with pinhole testing. The policy change was not to refract patients with a visual acuity ≥20/30 unless a specific request was made by the patient. Pre-post analysis of an automated time-motion study was conducted for the data collected 3 months before and 3 months after the policy change occurred on March 30, 2018. Changes to process and wait times were summarized using descriptive statistics and fitted to linear mixed regression models adjusting for appointment type, clinic volume, and daily clinic trends.

Results

One thousand nine hundred twenty-three visits with 1588 patients were included in the analysis. Mean [SD] age was 65.9 [14.7] years and 892 [56.2%] were women. After the policy change, technician process time decreased by 2.9 min (p < 0.0001) while daily clinical patient volume increased from 51.9 ± 16.8 patients to 58.4 ± 17.4 patients (p < 0.038). No significant difference was found in total wait time (p = 0.18) or total visit time (p = 0.83).

Conclusions

Real-time locating systems are effective at capturing clinical flow data and assessing clinical practice change initiatives. The refraction policy change was associated with reduced technician process time and overall the clinic was able to care for 7 more patients per day without significantly increasing patient wait time.

Early Changes in Billing and Notes After Evaluation and Management Guideline Change

BACKGROUND

The American Medical Association updated guidance in 2021 for frequently used billing codes for outpatient evaluation and management (E/M) visits. The intent was to account for provider time outside of face-to-face encounters and to reduce onerous documentation requirements.

OBJECTIVE

To analyze E/M visit use, documentation length, and time spent in the electronic health record (EHR) before and after the guideline change.

DESIGN

Observational, retrospective, pre-post study.

SETTING

U.S.-based ambulatory practices using the Epic Systems EHR.

PARTICIPANTS

303 547 advanced practice providers and physicians across 389 organizations.

MEASUREMENTS

Data from September 2020 through April 2021 containing weekly provider-level E/M code and EHR use metadata were extracted from the Epic Signal database. We descriptively analyzed overall and specialty-specific changes in E/M visit use, note length, and time spent in the EHR before and after the new guidelines using provider-level paired t tests.

RESULTS

Following the new guidelines, level 3 visits decreased by 2.41 percentage points (95% CI, -2.48 to -2.34 percentage points) to 38.5% of all E/M visits, a 5.9% relative decrease from fall 2020. Level 4 visits increased by 0.89 percentage points (CI, 0.82 to 0.96 percentage points) to 40.9% of E/M visits, a 2.2% relative increase. Level 5 visits (the highest acuity level) increased by 1.85 percentage points (CI, 1.81 to 1.89 percentage points) to 10.1% of E/M visits, a 22.6% relative increase. These changes varied by specialty. We found no meaningful changes in measures of note length or time spent in the EHR.

LIMITATION

The Epic ambulatory client base may underrepresent smaller and independent practices.

CONCLUSION

Immediate changes in E/M coding contrast with null findings for changes in both note length and EHR time. Provider organizations are positioned to respond more rapidly to billing process changes than to changes in care delivery and associated EHR use behaviors. Fully realizing the intended benefits of this guideline change will require more time, facilitation, and scaling of best practices that more directly address EHR documentation practices and associated burden.

PRIMARY FUNDING SOURCE

None.

Opportunities to use electronic health record audit logs to improve cancer care

The rapid adoption of electronic health records (EHRs) has created extensive repositories of digitized data that can be used to inform improvements in care delivery, processes, and patient outcomes. While the clinical data captured in EHRs are widely used for such efforts, EHRs also capture audit log data that reflect how users interact with the EHR to deliver care. Automatically collected audit log data provide a unique opportunity for new insights into EHR user behavior and decision‐making processes. Here, we provide an overview of audit log data and examples that could be used to improve oncology care and outcomes in four domains: diagnostic reasoning and consumption, care team collaboration and communication, patient outcomes and experience, and provider burnout/fatigue. This data source could identify gaps in performance and care, physician uptake of EHR features that enhance decision‐making, and integration of data trends for oncology. Ensuring researchers and oncologists are familiar with the data's potential and developing the data engineering capacity to utilize this rich data source, will expand the breadth of research to improve cancer care.

Predicting physician burnout using clinical activity logs: Model performance and lessons learned

BACKGROUND

Burnout is a significant public health concern affecting more than half of the healthcare workforce; however, passive screening tools to detect burnout are lacking. We investigated the ability of machine learning (ML) techniques to identify burnout using passively collected electronic health record (EHR)-based audit log data.

METHOD

Physician trainees participated in a longitudinal study where they completed monthly burnout surveys and provided access to their EHR-based audit logs. Using the monthly burnout scores as the target outcome, we trained ML models using combinations of features derived from audit log data-aggregate measures of clinical workload, time series-based temporal measures of EHR use, and the baseline burnout score. Five ML models were constructed to predict burnout as a continuous score: penalized linear regression, support vector machine, neural network, random forest, and gradient boosting machine.

RESULTS

88 trainee physicians participated and completed 416 surveys; greater than10 million audit log actions were collected (Mean [Standard Deviation] = 25,691 [14,331] actions per month, per physician). The workload feature set predicted burnout score with a mean absolute error (MAE) of 0.602 (95% Confidence Interval (CI), 0.412-0.826), and was able to predict burnout status with an average AUROC of 0.595 (95% CI 0.355-0.808) and average accuracy 0.567 (95% CI 0.393-0.742). The temporal feature set had a similar performance, with MAE 0.596 (95% CI 0.391-0.826), and AUROC 0.581 (95% CI 0.343-0.790). The addition of the baseline burnout score to the workload features improved the model performance to a mean AUROC of 0.829 (95% CI 0.607-0.996) and mean accuracy of 0.781 (95% CI 0.587-0.936); however, this performance was not meaningfully different than using the baseline burnout score alone.

CONCLUSIONS

Current findings illustrate the complexities of predicting burnout exclusively based on clinical work activities as captured in the EHR, highlighting its multi-factorial and individualized nature. Future prediction studies of burnout should account for individual factors (e.g., resilience, physiological measurements such as sleep) and associated system-level factors (e.g., leadership).

Outpatient portal use among pregnant individuals: Cross-sectional, temporal, and cluster analysis of use

Background

Outpatient portal technology can improve patient engagement. For pregnant individuals, the level of engagement could have important implications for maternal and infant outcomes.

Objective

This study: (1) cross-sectionally and temporally characterized the outpatient portal use among pregnant individuals seen at our academic medical center; and (2) identified clusters of the outpatient portal user groups based on the cross-sectional and temporal patterns of use.

Methods

We used outpatient portal server-side log files to execute a hierarchical clustering algorithm to group 7663 pregnant individuals based on proportions of outpatient portal function use. Post-hoc analyses were performed to further assess outpatient portal use on key encounter characteristics.

Results

The most frequently used functions were MyRecord (access personal health information), Visits (manage appointments), Messaging (send/receive messages), and Billing (view bills, insurance information). Median outpatient portal function use plateaued by the third trimester. Four distinct clusters were identified among all pregnant individuals: “Schedulers,” “Resulters,” “Intense Digital Engagers,” and “Average Users.” Post-hoc analyses revealed that the use of the Visits function increased and the use of the MyRecord function decreased over time among clusters.

Conclusions

Our identification of distinct cluster groups of outpatient portal users among pregnant individuals underscores the importance of avoiding the use of generalizations when describing how such patients might engage with patient-facing technologies such as an outpatient portal. These results can be used to improve user experience and training with outpatient portal functions and may educate maternal health providers on patient engagement with the outpatient portal.

A Micro-analysis Approach in Understanding Electronic Medical Record Usage in Rural Communities: Comparison of Frequency of Use on Performance Before and During the COVID-19 Pandemic

In strengthening eHealth in the Philippines to support the universal health care (UHC) law, the scaling up and full adoption of electronic medical record (EMR) systems was strategically scheduled and supposedly completed in 2020. The Covid-19 pandemic, however, delayed these strengthening efforts. We wanted to assess the status of EMR adoption in primary clinics of rural health units (RHUs) and understand the frequency of use, particularly during the pandemic. Through analyses of EMR usage logs from selected RHUs in 2020, we estimated frequency of EMR usage based on duration of use and tested if this was influenced by the performing RHU and pandemic event. We also determined the most frequent EMR activities through process maps and tested if there were differences in the conduct of these activities before and during the pandemic. Results showed that EMR use during work hours was significantly dependent on the performing RHU (p<0.001). High-performing RHUs used EMRs more than 3 hours/day while low-performing RHUs used the systems for less. The pandemic either significantly decreased or increased EMR use during work hours by around 5 hours/day in some RHUs (p<0.01). Process maps revealed that there were additional activities performed by RHUs during the pandemic. Except for Update Patient Profile and Add Patient EMR features, significant differences (p<0.01) were observed in accessing frequently used features before and during the pandemic. The results suggest some uneven level of utilization of EMRs at the primary care level which can impact readiness to support full implementation of the UHC law. The study shows the potential of using a more granular approach in studying adoption to help improve the quality of EMR use and contribute to improving health service delivery and financing.

Characterization of Electronic Health Record Use Outside Scheduled Clinic Hours among Primary Care Pediatricians: A Retrospective Descriptive Task Analysis of Electronic Health Record Access Log Data (Preprint)

Background

Many of the benefits of electronic health records (EHRs) have not been achieved at expected levels because of a variety of unintended negative consequences such as documentation burden. Previous studies have characterized EHR use during and outside work hours, with many reporting that physicians spend considerable time on documentation-related tasks. These studies characterized EHR use during and outside work hours using clock time versus actual physician clinic schedules to define the outside work time.

Objective

This study aimed to characterize EHR work outside scheduled clinic hours among primary care pediatricians using a retrospective descriptive task analysis of EHR access log data and actual physician clinic schedules to define work time.

Methods

We conducted a retrospective, exploratory, descriptive task analysis of EHR access log data from primary care pediatricians in September 2019 at a large Midwestern pediatric health center to quantify and identify actions completed outside scheduled clinic hours. Mixed-effects statistical modeling was used to investigate the effects of age, sex, clinical full-time equivalent status, and EHR work during scheduled clinic hours on the use of EHRs outside scheduled clinic hours.

Results

Primary care pediatricians (n=56) in this study generated 1,523,872 access log data points (across 1069 physician workdays) and spent an average of 4.4 (SD 2.0) hours and 0.8 (SD 0.8) hours per physician per workday engaged in EHRs during and outside scheduled clinic hours, respectively. Approximately three-quarters of the time working in EHR during or outside scheduled clinic hours was spent reviewing data and reports. Mixed-effects regression revealed no associations of age, sex, or clinical full-time equivalent status with EHR use during or outside scheduled clinic hours.

Conclusions

For every hour primary care pediatricians spent engaged with the EHR during scheduled clinic hours, they spent approximately 10 minutes interacting with the EHR outside scheduled clinic hours. Most of their time (during and outside scheduled clinic hours) was spent reviewing data, records, and other information in EHR.

Predicting next-day discharge via electronic health record access logs

OBJECTIVE

Hospital capacity management depends on accurate real-time estimates of hospital-wide discharges. Estimation by a clinician requires an excessively large amount of effort and, even when attempted, accuracy in forecasting next-day patient-level discharge is poor. This study aims to support next-day discharge predictions with machine learning by incorporating electronic health record (EHR) audit log data, a resource that captures EHR users' granular interactions with patients' records by communicating various semantics and has been neglected in outcome predictions.

MATERIALS AND METHODS

This study focused on the EHR data for all adults admitted to Vanderbilt University Medical Center in 2019. We learned multiple advanced models to assess the value that EHR audit log data adds to the daily prediction of discharge likelihood within 24 h and to compare different representation strategies. We applied Shapley additive explanations to identify the most influential types of user-EHR interactions for discharge prediction.

RESULTS

The data include 26 283 inpatient stays, 133 398 patient-day observations, and 819 types of user-EHR interactions. The model using the count of each type of interaction in the recent 24 h and other commonly used features, including demographics and admission diagnoses, achieved the highest area under the receiver operating characteristics (AUROC) curve of 0.921 (95% CI: 0.919-0.923). By contrast, the model lacking user-EHR interactions achieved a worse AUROC of 0.862 (0.860-0.865). In addition, 10 of the 20 (50%) most influential factors were user-EHR interaction features.

CONCLUSION

EHR audit log data contain rich information such that it can improve hospital-wide discharge predictions.

Association Between Proficiency and Efficiency in Electronic Health Records Among Pediatricians at a Major Academic Health System

Objective: The purpose of this study was to evaluate the variations in electronic health record (EHR) activity among General and Specialty pediatricians by investigating the time spent and documentation length, normalized for workload.

Materials and Methods: We conducted a cross-sectional study of pediatric physicians using Epic EHR at a major Southeastern academic healthcare system. We collected user-level EHR activity data of 104 pediatric physicians over 91 days from April 1 to June 30, 2020.

Results: Of the 104 pediatrics physicians, 56 (54%) were General pediatricians and 48 (46%) were Specialists pediatricians. General pediatricians spent an average of 17.6 min [interquartile range (IQR): 12.9–37] using the EHR per appointment, while Specialists spent 35.7 min (IQR: 28–48.4) per appointment.

Significant negative associations were found between proficiency scores and the amount of time spent in the system for Generalists (p < 0.001). On the contrary, significant positive associations were found between proficiency scores and the amount of time spent in the system for Specialists (p < 0.01).

Conclusions: We report an association between EHR proficiency and efficiency levels among pediatricians within the same healthcare system, receiving the same EHR training, and using the same EHR system. The profound differences in EHR activity suggest that higher priority should be given to redesigning EHR training methods to accommodate the learning needs of physicians.

Characterizing Multitasking and Workflow Fragmentation in Electronic Health Records among Emergency Department Clinicians: Using Time-Motion Data to Understand Documentation Burden

BACKGROUND

The impact of electronic health records (EHRs) in the emergency department (ED) remains mixed. Dynamic and unpredictable, the ED is highly vulnerable to workflow interruptions.

OBJECTIVES

The aim of the study is to understand multitasking and task fragmentation in the clinical workflow among ED clinicians using clinical information systems (CIS) through time-motion study (TMS) data, and inform their applications to more robust and generalizable measures of CIS-related documentation burden.

METHODS

Using TMS data collected among 15 clinicians in the ED, we investigated the role of documentation burden, multitasking (i.e., performing physical and communication tasks concurrently), and workflow fragmentation in the ED. We focused on CIS-related tasks, including EHRs.

RESULTS

We captured 5,061 tasks and 877 communications in 741 locations within the ED. Of the 58.7 total hours observed, 44.7% were spent on CIS-related tasks; nearly all CIS-related tasks focused on data-viewing and data-entering. Over one-fifth of CIS-related task time was spent on multitasking. The mean average duration among multitasked CIS-related tasks was shorter than non-multitasked CIS-related tasks (20.7 s vs. 30.1 s). Clinicians experienced 1.4 ± 0.9 task switches/min, which increased by one-third when multitasking. Although multitasking was associated with a significant increase in the average duration among data-entering tasks, there was no significant effect on data-viewing tasks. When engaged in CIS-related task switches, clinicians were more likely to return to the same CIS-related task at higher proportions while multitasking versus not multitasking.

CONCLUSION

Multitasking and workflow fragmentation may play a significant role in EHR documentation among ED clinicians, particularly among data-entering tasks. Understanding where and when multitasking and workflow fragmentation occurs is a crucial step to assessing potentially burdensome clinician tasks and mitigating risks to patient safety. These findings may guide future research on developing more scalable and generalizable measures of CIS-related documentation burden that do not necessitate direct observation techniques (e.g., EHR log files).

Measuring Collaboration Through Concurrent Electronic Health Record Usage: Network Analysis Study

BACKGROUND

Collaboration is vital within health care institutions, and it allows for the effective use of collective health care worker (HCW) expertise. Human-computer interactions involving electronic health records (EHRs) have become pervasive and act as an avenue for quantifying these collaborations using statistical and network analysis methods.

OBJECTIVE

We aimed to measure HCW collaboration and its characteristics by analyzing concurrent EHR usage.

METHODS

By extracting concurrent EHR usage events from audit log data, we defined concurrent sessions. For each HCW, we established a metric called concurrent intensity, which was the proportion of EHR activities in concurrent sessions over all EHR activities. Statistical models were used to test the differences in the concurrent intensity between HCWs. For each patient visit, starting from admission to discharge, we measured concurrent EHR usage across all HCWs, which we called temporal patterns. Again, we applied statistical models to test the differences in temporal patterns of the admission, discharge, and intermediate days of hospital stay between weekdays and weekends. Network analysis was leveraged to measure collaborative relationships among HCWs. We surveyed experts to determine if they could distinguish collaborative relationships between high and low likelihood categories derived from concurrent EHR usage. Clustering was used to aggregate concurrent activities to describe concurrent sessions. We gathered 4 months of EHR audit log data from a large academic medical center's neonatal intensive care unit (NICU) to validate the effectiveness of our framework.

RESULTS

There was a significant difference (P<.001) in the concurrent intensity (proportion of concurrent activities: ranging from mean 0.07, 95% CI 0.06-0.08, to mean 0.36, 95% CI 0.18-0.54; proportion of time spent on concurrent activities: ranging from mean 0.32, 95% CI 0.20-0.44, to mean 0.76, 95% CI 0.51-1.00) between the top 13 HCW specialties who had the largest amount of time spent in EHRs. Temporal patterns between weekday and weekend periods were significantly different on admission (number of concurrent intervals per hour: 11.60 vs 0.54; P<.001) and discharge days (4.72 vs 1.54; P<.001), but not during intermediate days of hospital stay. Neonatal nurses, fellows, frontline providers, neonatologists, consultants, respiratory therapists, and ancillary and support staff had collaborative relationships. NICU professionals could distinguish high likelihood collaborative relationships from low ones at significant rates (3.54, 95% CI 3.31-4.37 vs 2.64, 95% CI 2.46-3.29; P<.001). We identified 50 clusters of concurrent activities. Over 87% of concurrent sessions could be described by a single cluster, with the remaining 13% of sessions comprising multiple clusters.

CONCLUSIONS

Leveraging concurrent EHR usage workflow through audit logs to analyze HCW collaboration may improve our understanding of collaborative patient care. HCW collaboration using EHRs could potentially influence the quality of patient care, discharge timeliness, and clinician workload, stress, or burnout.

Measuring Electronic Health Record Use in the Pediatric ICU Using Audit-Logs and Screen Recordings

BACKGROUND

 Time spent in the electronic health record (EHR) has been identified as an important unit of measure for health care provider clinical activity. The lack of validation of audit-log based inpatient EHR time may have resulted in underuse of this data in studies focusing on inpatient patient outcomes, provider efficiency, provider satisfaction, etc. This has also led to a dearth of clinically relevant EHR usage metrics consistent with inpatient provider clinical activity.

OBJECTIVE

 The aim of our study was to validate audit-log based EHR times using observed EHR-times extracted from screen recordings of EHR usage in the inpatient setting.

METHODS

 This study was conducted in a 36-bed pediatric intensive care unit (PICU) at Lucile Packard Children's Hospital Stanford between June 11 and July 14, 2020. Attending physicians, fellow physicians, hospitalists, and advanced practice providers with ≥0.5 full-time equivalent (FTE) for the prior four consecutive weeks and at least one EHR session recording were included in the study. Citrix session recording player was used to retrospectively review EHR session recordings that were captured as the provider interacted with the EHR.

RESULTS

 EHR use patterns varied by provider type. Audit-log based total EHR time correlated strongly with both observed total EHR time (r = 0.98, p < 0.001) and observed active EHR time (r = 0.95, p < 0.001). Each minute of audit-log based total EHR time corresponded to 0.95 (0.87-1.02) minutes of observed total EHR time and 0.75 (0.67-0.83) minutes of observed active EHR time. Results were similar when stratified by provider role.

CONCLUSION

 Our study found inpatient audit-log based EHR time to correlate strongly with observed EHR time among pediatric critical care providers. These findings support the use of audit-log based EHR-time as a surrogate measure for inpatient provider EHR use, providing an opportunity for researchers and other stakeholders to leverage EHR audit-log data in measuring clinical activity and tracking outcomes of workflow improvement efforts longitudinally and across provider groups.

Characterizing physician EHR use with vendor derived data: a feasibility study and cross-sectional analysis

OBJECTIVE

To derive 7 proposed core electronic health record (EHR) use metrics across 2 healthcare systems with different EHR vendor product installations and examine factors associated with EHR time.

MATERIALS AND METHODS

A cross-sectional analysis of ambulatory physicians EHR use across the Yale-New Haven and MedStar Health systems was performed for August 2019 using 7 proposed core EHR use metrics normalized to 8 hours of patient scheduled time.

RESULTS

Five out of 7 proposed metrics could be measured in a population of nonteaching, exclusively ambulatory physicians. Among 573 physicians (Yale-New Haven N = 290, MedStar N = 283) in the analysis, median EHR-Time8 was 5.23 hours. Gender, additional clinical hours scheduled, and certain medical specialties were associated with EHR-Time8 after adjusting for age and health system on multivariable analysis. For every 8 hours of scheduled patient time, the model predicted these differences in EHR time (P < .001, unless otherwise indicated): female physicians +0.58 hours; each additional clinical hour scheduled per month -0.01 hours; practicing cardiology -1.30 hours; medical subspecialties -0.89 hours (except gastroenterology, P = .002); neurology/psychiatry -2.60 hours; obstetrics/gynecology -1.88 hours; pediatrics -1.05 hours (P = .001); sports/physical medicine and rehabilitation -3.25 hours; and surgical specialties -3.65 hours.

CONCLUSIONS

For every 8 hours of scheduled patient time, ambulatory physicians spend more than 5 hours on the EHR. Physician gender, specialty, and number of clinical hours practicing are associated with differences in EHR time. While audit logs remain a powerful tool for understanding physician EHR use, additional transparency, granularity, and standardization of vendor-derived EHR use data definitions are still necessary to standardize EHR use measurement.

Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning

OBJECTIVE

The characteristics of clinician activities while interacting with electronic health record (EHR) systems can influence the time spent in EHRs and workload. This study aims to characterize EHR activities as tasks and define novel, data-driven metrics.

MATERIALS AND METHODS

We leveraged unsupervised learning approaches to learn tasks from sequences of events in EHR audit logs. We developed metrics characterizing the prevalence of unique events and event repetition and applied them to categorize tasks into 4 complexity profiles. Between these profiles, Mann-Whitney U tests were applied to measure the differences in performance time, event type, and clinician prevalence, or the number of unique clinicians who were observed performing these tasks. In addition, we apply process mining frameworks paired with clinical annotations to support the validity of a sample of our identified tasks. We apply our approaches to learn tasks performed by nurses in the Vanderbilt University Medical Center neonatal intensive care unit.

RESULTS

We examined EHR audit logs generated by 33 neonatal intensive care unit nurses resulting in 57 234 sessions and 81 tasks. Our results indicated significant differences in performance time for each observed task complexity profile. There were no significant differences in clinician prevalence or in the frequency of viewing and modifying event types between tasks of different complexities. We presented a sample of expert-reviewed, annotated task workflows supporting the interpretation of their clinical meaningfulness.

CONCLUSIONS

The use of the audit log provides an opportunity to assist hospitals in further investigating clinician activities to optimize EHR workflows.

Impact of Changes in EHR Use during COVID-19 on Physician Trainee Mental Health

OBJECTIVES

This article investigates the association between changes in electronic health record (EHR) use during the coronavirus disease 2019 (COVID-19) pandemic on the rate of burnout, stress, posttraumatic stress disorder (PTSD), depression, and anxiety among physician trainees (residents and fellows).

METHODS

A total of 222 (of 1,375, 16.2%) physician trainees from an academic medical center responded to a Web-based survey. We compared the physician trainees who reported that their EHR use increased versus those whose EHR use stayed the same or decreased on outcomes related to depression, anxiety, stress, PTSD, and burnout using univariable and multivariable models. We examined whether self-reported exposure to COVID-19 patients moderated these relationships.

RESULTS

Physician trainees who reported increased use of EHR had higher burnout (adjusted mean, 1.48 [95% confidence interval [CI] 1.24, 1.71] vs. 1.05 [95% CI 0.93, 1.17]; p = 0.001) and were more likely to exhibit symptoms of PTSD (adjusted mean = 15.09 [95% CI 9.12, 21.05] vs. 9.36 [95% CI 7.38, 11.28]; p = 0.035). Physician trainees reporting increased EHR use outside of work were more likely to experience depression (adjusted mean, 8.37 [95% CI 5.68, 11.05] vs. 5.50 [95% CI 4.28, 6.72]; p = 0.035). Among physician trainees with increased EHR use, those exposed to COVID-19 patients had significantly higher burnout (2.04, p < 0.001) and depression scores (14.13, p = 0.003).

CONCLUSION

Increased EHR use was associated with higher burnout, depression, and PTSD outcomes among physician trainees. Although preliminary, these findings have implications for creating systemic changes to manage the wellness and well-being of trainees.