Patient Outcomes when Housestaff Exceed 80 Hours per Week

Uses of Electronic Health Record Data to Measure the Clinical Learning Environment of Graduate Medical Education Trainees: A Systematic Review

PURPOSE

This study systematically reviews the uses of electronic health record (EHR) data to measure graduate medical education (GME) trainee competencies.

METHOD

In January 2022, the authors conducted a systematic review of original research in MEDLINE from database start to December 31, 2021. The authors searched for articles that used the EHR as their data source and in which the individual GME trainee was the unit of observation and/or unit of analysis. The database query was intentionally broad because an initial survey of pertinent articles identified no unifying Medical Subject Heading terms. Articles were coded and clustered by theme and Accreditation Council for Graduate Medical Education (ACGME) core competency.

RESULTS

The database search yielded 3,540 articles, of which 86 met the study inclusion criteria. Articles clustered into 16 themes, the largest of which were trainee condition experience (17 articles), work patterns (16 articles), and continuity of care (12 articles). Five of the ACGME core competencies were represented (patient care and procedural skills, practice-based learning and improvement, systems-based practice, medical knowledge, and professionalism). In addition, 25 articles assessed the clinical learning environment.

CONCLUSIONS

This review identified 86 articles that used EHR data to measure individual GME trainee competencies, spanning 16 themes and 6 competencies and revealing marked between-trainee variation. The authors propose a digital learning cycle framework that arranges sequentially the uses of EHR data within the cycle of clinical experiential learning central to GME. Three technical components necessary to unlock the potential of EHR data to improve GME are described: measures, attribution, and visualization. Partnerships between GME programs and informatics departments will be pivotal in realizing this opportunity.

Clinical outcome prediction using observational supervision with electronic health records and audit logs

OBJECTIVE

Audit logs in electronic health record (EHR) systems capture interactions of providers with clinical data. We determine if machine learning (ML) models trained using audit logs in conjunction with clinical data ("observational supervision") outperform ML models trained using clinical data alone in clinical outcome prediction tasks, and whether they are more robust to temporal distribution shifts in the data.

MATERIALS AND METHODS

Using clinical and audit log data from Stanford Healthcare, we trained and evaluated various ML models including logistic regression, support vector machine (SVM) classifiers, neural networks, random forests, and gradient boosted machines (GBMs) on clinical EHR data, with and without audit logs for two clinical outcome prediction tasks: major adverse kidney events within 120 days of ICU admission (MAKE-120) in acute kidney injury (AKI) patients and 30-day readmission in acute stroke patients. We further tested the best performing models using patient data acquired during different time-intervals to evaluate the impact of temporal distribution shifts on model performance.

RESULTS

Performance generally improved for all models when trained with clinical EHR data and audit log data compared with those trained with only clinical EHR data, with GBMs tending to have the overall best performance. GBMs trained with clinical EHR data and audit logs outperformed GBMs trained without audit logs in both clinical outcome prediction tasks: AUROC 0.88 (95% CI: 0.85-0.91) vs. 0.79 (95% CI: 0.77-0.81), respectively, for MAKE-120 prediction in AKI patients, and AUROC 0.74 (95% CI: 0.71-0.77) vs. 0.63 (95% CI: 0.62-0.64), respectively, for 30-day readmission prediction in acute stroke patients. The performance of GBM models trained using audit log and clinical data degraded less in later time-intervals than models trained using only clinical data.

CONCLUSION

Observational supervision with audit logs improved the performance of ML models trained to predict important clinical outcomes in patients with AKI and acute stroke, and improved robustness to temporal distribution shifts.

Strengthening the Clinical Learning Environment by Mandate-Implementing the ACGME Common Program Requirements

RATIONALE

Three years ago, the Accreditation Council for Graduate Medical Education (ACGME) introduced updated Common Program Requirements in recognition of the need to further promote resident and faculty member well-being and patient safety. The ACGME acknowledged residencies would need time to comply with new requirements. This grace period, however, concluded as of July 1, 2019, and programs now risk citations for failure to implement new requirements.

METHODS AND RESULTS

The authors, members of the Association of Program Directors in Radiology Common Program Requirements Ad Hoc committee, developed downloadable resources provided in the Appendix delineating the 2019 Common Program Requirements and offering sample resources as compliant solutions.

CONCLUSION

The resources offer a national standardized approach to educating trainees in these essential skills and should be especially helpful to programs with access to fewer resources. In addition to achieving compliance, incorporation of these resources into residency training will ensure the next generation of radiologists are equipped to add value while remaining physically and emotionally healthy.

Opportunities to use electronic health record audit logs to improve cancer care

The rapid adoption of electronic health records (EHRs) has created extensive repositories of digitized data that can be used to inform improvements in care delivery, processes, and patient outcomes. While the clinical data captured in EHRs are widely used for such efforts, EHRs also capture audit log data that reflect how users interact with the EHR to deliver care. Automatically collected audit log data provide a unique opportunity for new insights into EHR user behavior and decision‐making processes. Here, we provide an overview of audit log data and examples that could be used to improve oncology care and outcomes in four domains: diagnostic reasoning and consumption, care team collaboration and communication, patient outcomes and experience, and provider burnout/fatigue. This data source could identify gaps in performance and care, physician uptake of EHR features that enhance decision‐making, and integration of data trends for oncology. Ensuring researchers and oncologists are familiar with the data's potential and developing the data engineering capacity to utilize this rich data source, will expand the breadth of research to improve cancer care.

Characterizing styles of clinical note production and relationship to clinical work hours among first-year residents

OBJECTIVE

To characterize variation in clinical documentation production patterns, how this variation relates to individual resident behavior preferences, and how these choices relate to work hours.

MATERIALS AND METHODS

We used unsupervised machine learning with clinical note metadata for 1265 progress notes written for 279 patient encounters by 50 first-year residents on the Hospital Medicine service in 2018 to uncover distinct note-level and user-level production patterns. We examined average and 95% confidence intervals of median user daily work hours measured from audit log data for each user-level production pattern.

RESULTS

Our analysis revealed 10 distinct note-level and 5 distinct user-level production patterns (user styles). Note production patterns varied in when writing occurred and in how dispersed writing was through the day. User styles varied in which note production pattern(s) dominated. We observed suggestive trends in work hours for different user styles: residents who preferred producing notes in dispersed sessions had higher median daily hours worked while residents who preferred producing notes in the morning or in a single uninterrupted session had lower median daily hours worked.

DISCUSSION

These relationships suggest that note writing behaviors should be further investigated to understand what practices could be targeted to reduce documentation burden and derivative outcomes such as resident work hour violations.

CONCLUSION

Clinical note documentation is a time-consuming activity for physicians; we identify substantial variation in how first-year residents choose to do this work and suggestive trends between user preferences and work hours.

Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods

OBJECTIVE

To systematically review published literature and identify consistency and variation in the aims, measures, and methods of studies using electronic health record (EHR) audit logs to observe clinical activities.

MATERIALS AND METHODS

In July 2019, we searched PubMed for articles using EHR audit logs to study clinical activities. We coded and clustered the aims, measures, and methods of each article into recurring categories. We likewise extracted and summarized the methods used to validate measures derived from audit logs and limitations discussed of using audit logs for research.

RESULTS

Eighty-five articles met inclusion criteria. Study aims included examining EHR use, care team dynamics, and clinical workflows. Studies employed 6 key audit log measures: counts of actions captured by audit logs (eg, problem list viewed), counts of higher-level activities imputed by researchers (eg, chart review), activity durations, activity sequences, activity clusters, and EHR user networks. Methods used to preprocess audit logs varied, including how authors filtered extraneous actions, mapped actions to higher-level activities, and interpreted repeated actions or gaps in activity. Nineteen studies validated results (22%), but only 9 (11%) through direct observation, demonstrating varying levels of measure accuracy.

DISCUSSION

While originally designed to aid access control, EHR audit logs have been used to observe diverse clinical activities. However, most studies lack sufficient discussion of measure definition, calculation, and validation to support replication, comparison, and cross-study synthesis.

CONCLUSION

EHR audit logs have potential to scale observational research but the complexity of audit log measures necessitates greater methodological transparency and validated standards.

Analyses of electronic health records utilization in a large community hospital

Introduction

The Electronic Health Record (EHR) has become an integral component of healthcare delivery. Survey based studies have estimated that physicians spend 4–6 hours of their workday devoted to EHR. Our study was designed to use computer software to objectively obtain time spent on EHR.

Methods

We recorded EHR time for 248 physiciansover 2 time intervals. EHR active use was defined as more than 15 keystrokes, or 3 mouse clicks, or 1700 "mouse miles" per minute. We recorded total time and % of work hours spent on EHR, and differences in those based on seniority. Physicians reported duty hours using a standardized toolkit.

Results

Physicians spent 3.8 (±2) hours on EHR daily, which accounted for 37% (±17%), 41% (±14%), and 45% (±12%) of their day for all clinicians, residents, and interns, respectively. With the progression of training, there was a reduction in EHR time (all p values <0.01). During the first academic quarter, clinicians spent 38% (± 8%) of time on chart review, 17% (± 7%) on orders, 28% (±11%) on documentation (i.e. writing notes) and 17% (±7%) on other activities (i.e. physician hand-off and medication reconciliation). This pattern remained unchanged during the fourth quarter.

Conclusions

Physicians spend close to 40% of their work day on EHR, with interns spending the most time. There is a significant reduction in time spent on EHR with training and greater experience, although the overall amount of time spent on EHR remained high.

Electronic Health Record Use among Ophthalmology Residents while on Call

BACKGROUND

As electronic health record (EHR) use becomes more widespread, detailed records of how users interact with the EHR, known as EHR audit logs, are being used to characterize the clinical workflows of physicians including residents. After-hours EHR use is of particular interest given its known association with physician burnout. Several studies have analyzed EHR audit logs for residents in other fields, such as internal medicine, but none thus far in ophthalmology. Here, we focused specifically on EHR use during on-call shifts outside of normal clinic hours.

METHODS

In this retrospective study, we analyzed raw EHR audit log data from on-call shifts for 12 ophthalmology residents at a single institution over the course of a calendar year. Data were analyzed to characterize total time spent using the EHR, clinical volume, diagnoses of patients seen on call, and EHR tasks.

RESULTS

Across all call shifts, the median and interquartile range (IQR) of the time spent logged into the EHR per shift were 88 and 131 minutes, respectively. The median (IQR) unique patient charts accessed per shift was 7 (9) patients. When standardized to per-hour measures, weekday evening shifts were the busiest call shifts with regard to both EHR use time and clinical volume. Total EHR use time and clinical volume were greatest in the summer months (July to September). Chart review comprised a majority (63.4%) of ophthalmology residents' on-call EHR activities.

CONCLUSION

In summary, EHR audit logs demonstrate substantial call burden for ophthalmology residents outside of regular clinic hours. These data and future studies can be used to further characterize the clinical exposure and call burden of ophthalmology residents and could potentially have broader implications in the fields of physician burnout and education policy.

Physicians' working time restriction and its impact on patient safety: an integrative review

BACKGROUND

Although shift work is a part of the physicians' routine, there is controversy on the length of shifts and adequate rest for safe professional practice. If on the one hand long working hours might have negative impact on patient safety by interfering with the psychological and physical functioning of physicians, on the other shorter working hours might impair the safety of patients due to interference with the continuity of care.

OBJECTIVE

To analyze the impact of restrictions to physicians' working hours on patient safety.

METHOD

Integrative literature review in which we surveyed studies on restriction to physicians' working time and patient safety included in databases National Library of Medicine (PubMed) and Scientific Electronic Library Online (SciELO) until May 2018. Thirty-five studies which met the inclusion criteria were included.

RESULTS

Patient safety outcomes analyzed in the included studies were mortality, adverse events, continuity of care, in-hospital complications, readmission rate and length of stay at hospital. Restriction to working time was associated with variable impact on patient safety indicators, but often did not modify their performance.

CONCLUSION

Restrictions to physicians' working time did not always improved patient safety indicators. Focusing on interventions which only seek to limit the workload of physicians might be insufficient to bring consistent improvement to patient care.

Tracking resident pre-rounding electronic health record usage

PURPOSE

Residents collect information from the electronic health record (EHR) to present during rounds, but this crucial process is understudied. The purpose of this paper is to examine the feasibility of utilizing an EHR embedded time-tracking software to quantify resident pre-round EHR activity and how patient acuity impacts this activity.

DESIGN/METHODOLOGY/APPROACH

This was a retrospective observational study that quantified resident EHR activities (total time spent, tasks performed and patient encounters accessed) during pre-rounds on their pediatric intensive care unit rotation between May 2016 and December 2016. Patient encounters were reviewed to determine resident ownership and critical care resources provided.

FINDINGS

Allo 21 eligible participants were included. In total, 907 patient encounters were included to evaluate patient acuity impact. EHR usage per patient encounter (median in minutes (25th, 75th percentile)) was significantly affected by the critical care resources utilized. Total EHR time: both ventilator and vasoactive support (10.54 (6.68, 17.19)); neither ventilator nor vasoactive support (8.23 (5.07, 12.72)); invasive/noninvasive ventilator support (8.74 (5.69, 13.2)); and vasoactive support (10.37 (7.72, 11.65)), p<0.001. Chart review, order entry and documentation EHR times demonstrated similar trends.

PRACTICAL IMPLICATIONS

Residents spend more time utilizing the EHR to collect data on patients who require significant critical care resources. This information can be useful to determine optimal resident to patient workload. Future research is required to assess this EHR tool's ability to contribute to physician workflow study.

ORIGINALITY/VALUE

EHR embedded time-tracking software can offer insights into resident workflow.

Resident duty hours: Families' knowledge and perceptions in the paediatric intensive care unit

BACKGROUND AND OBJECTIVES

Resident duty hours remain a controversial topic in the literature. Competing interests include patient safety, resident education, and resident well-being. No studies, however, have sought family members' perspectives on duty hours in the paediatric context. This study aimed to explore family members' knowledge of trainee duty hours, and their perspectives on the balance between shift duration and hand-off frequency.

METHODS

We surveyed family members of patients admitted ≥ 24 hours in the paediatric intensive care unit at an academic center. We simultaneously collected daily logs of hours worked by trainees. Descriptive statistics were used to analyze survey responses and trainee duty hours.

RESULTS

One-hundred and one family members responded (75%). Respondents demonstrated knowledge of trainees working long duty hours but reported lower averages than the trainee logs (55 versus 66 hours per week and 16 versus 24 hours per shift). Elements related to both potential trainee fatigue and hand-offs raised concern in more than half of respondents. When asked to choose between a familiar trainee working a prolonged shift, or an unfamiliar trainee at the start of their shift, respondents were divided (52% versus 48%, respectively).

CONCLUSIONS

Family members of critically ill paediatric patients are aware that trainees provide patient care while working long duty hours with minimal sleep. Despite this awareness, long shifts retain value with some families, possibly due to continuity. Changes to duty hours and hand-off frequency may pose an unrealized harm on family-centered care, as well as patient-provider relationships, and further study is warranted.

ACGME Clinical and Educational Work Hour Standards: Perspectives and Recommendations from Emergency Medicine Educators

Introduction

The American College of Emergency Physicians (ACEP) and the Council of Emergency Medicine Residency Directors (CORD) were invited to contribute to the 2016 Accreditation Council for Graduate Medical Education’s (ACGME) Second Resident Duty Hours in the Learning and Working Environment Congress. We describe the joint process used by ACEP and CORD to capture the opinions of emergency medicine (EM) educators on the ACGME clinical and educational work hour standards, formulate recommendations, and inform subsequent congressional testimony.

Methods

In 2016 our joint working group of experts in EM medical education conducted a consensus-based, mixed-methods process using survey data from medical education stakeholders in EM and expert iterative discussions to create organizational position statements and recommendations for revisions of work hour standards. A 19-item survey was administered to a convenience sample of 199 EM residency training programs using a national EM educational listserv.

Results

A total of 157 educational leaders responded to the survey; 92 of 157 could be linked to specific programs, yielding a targeted response rate of 46.2% (92/199) of programs. Respondents commented on the impact of clinical and educational work-hour standards on patient safety, programmatic and personnel costs, resident caseload, and educational experience. Using survey results, comments, and iterative discussions, organizational recommendations were crafted and submitted to the ACGME.

Conclusion

EM educators believe that ACGME clinical and educational work hour standards negatively impact the learning environment and are not optimal for promoting patient safety or the development of resident professional citizenship.

Resident Workflow and Psychiatric Emergency Consultation: Identifying Factors for Quality Improvement in a Training Environment

OBJECTIVE

Quality improvement to optimize workflow has the potential to mitigate resident burnout and enhance patient care. This study applied mixed methods to identify factors that enhance or impede workflow for residents performing emergency psychiatric consultations.

METHODS

The study population consisted of all psychiatry program residents (55 eligible, 42 participating) at the Semel Institute for Neuroscience and Human Behavior, University of California, Los Angeles. The authors developed a survey through iterative piloting, surveyed all residents, and then conducted a focus group. The survey included elements hypothesized to enhance or impede workflow, and measures pertaining to self-rated efficiency and stress. Distributional and bivariate analyses were performed. Survey findings were clarified in focus group discussion.

RESULTS

This study identified several factors subjectively associated with enhanced or impeded workflow, including difficulty with documentation, the value of personal organization systems, and struggles to communicate with patients' families.

CONCLUSION

Implications for resident education are discussed.