Attributing Patients to Pediatric Residents Using Electronic Health Record Features Augmented with Audit Logs

Toward precision medical education: Characterizing individual residents' clinical experiences throughout training

BACKGROUND

Despite the central role of experiential learning in residency training, the actual clinical experiences residents participate in are not well characterized. A better understanding of the type, volume, and variation in residents' clinical experiences is essential to support precision medical education strategies.

OBJECTIVE

We sought to characterize the entirety of the clinical experiences had by individual internal medicine residents throughout their time in training.

METHOD

We evaluated the clinical experiences of medicine residents (n = 51) who completed training at NYU Grossman School of Medicine's Brooklyn campus between 2020 and 2023. Residents' inpatient and outpatient experiences were identified using notes written, orders placed, and care team sign-ins; principal ICD-10 codes for each encounter were converted into medical content categories using a previously described crosswalk tool.

RESULTS

Of 152,426 clinical encounters with available ICD-10 codes, 132,284 were mapped to medical content categories (94.5% capture). Residents' clinical experiences were particularly enriched in infectious and cardiovascular disease; most had very little exposure to allergy, dermatology, oncology, or rheumatology. Some trainees saw twice as many cases in a given content area as did others. There was little concordance between actual frequency of clinical experience and expected content frequency on the ABIM certification exam.

CONCLUSIONS

Individual residents' clinical experiences in training vary widely, both in number and in type. Characterizing these experiences paves the way for exploration of the relationships between clinical exposure and educational outcomes, and for the implementation of precision education strategies that could fill residents' experiential gaps and complement strengths with targeted educational interventions.

Uses of Electronic Health Record Data to Measure the Clinical Learning Environment of Graduate Medical Education Trainees: A Systematic Review

PURPOSE

This study systematically reviews the uses of electronic health record (EHR) data to measure graduate medical education (GME) trainee competencies.

METHOD

In January 2022, the authors conducted a systematic review of original research in MEDLINE from database start to December 31, 2021. The authors searched for articles that used the EHR as their data source and in which the individual GME trainee was the unit of observation and/or unit of analysis. The database query was intentionally broad because an initial survey of pertinent articles identified no unifying Medical Subject Heading terms. Articles were coded and clustered by theme and Accreditation Council for Graduate Medical Education (ACGME) core competency.

RESULTS

The database search yielded 3,540 articles, of which 86 met the study inclusion criteria. Articles clustered into 16 themes, the largest of which were trainee condition experience (17 articles), work patterns (16 articles), and continuity of care (12 articles). Five of the ACGME core competencies were represented (patient care and procedural skills, practice-based learning and improvement, systems-based practice, medical knowledge, and professionalism). In addition, 25 articles assessed the clinical learning environment.

CONCLUSIONS

This review identified 86 articles that used EHR data to measure individual GME trainee competencies, spanning 16 themes and 6 competencies and revealing marked between-trainee variation. The authors propose a digital learning cycle framework that arranges sequentially the uses of EHR data within the cycle of clinical experiential learning central to GME. Three technical components necessary to unlock the potential of EHR data to improve GME are described: measures, attribution, and visualization. Partnerships between GME programs and informatics departments will be pivotal in realizing this opportunity.

Measuring Training Disruptions Using an Informatics Based Tool

OBJECTIVE

Training disruptions, such as planned curricular adjustments or unplanned global pandemics, impact residency training in ways that are difficult to quantify. Informatics-based medical education tools can help measure these impacts. We tested the ability of a software platform driven by electronic health record data to quantify anticipated changes in trainee clinical experiences during the COVID-19 pandemic.

METHODS

We previously developed and validated the Trainee Individualized Learning System (TRAILS) to identify pediatric resident clinical experiences (i.e. shifts, resident provider-patient interactions (rPPIs), and diagnoses). We used TRAILS to perform a year-over-year analysis comparing pediatrics residents at a large academic children's hospital during March 15-June 15 in 2018 (Control #1), 2019 (Control #2), and 2020 (Exposure).

RESULTS

Residents in the exposure cohort had fewer shifts than those in both control cohorts (P < .05). rPPIs decreased an average of 43% across all PGY levels, with interns experiencing a 78% decrease in Continuity Clinic. Patient continuity decreased from 23% to 11%. rPPIs with common clinic and emergency department diagnoses decreased substantially during the exposure period.

CONCLUSIONS

Informatics tools like TRAILS may help program directors understand the impact of training disruptions on resident clinical experiences and target interventions to learners' needs and development.

Where Is the Digitally Silent Provider? Development and Validation of a Team-Centered Electronic Health Record Attribution Model for Supervising Residents

PROBLEM

Providing trainees with data and benchmarks on their own patient populations is an Accreditation Council for Graduate Medical Education core residency requirement. Leveraging electronic health records (EHRs) for this purpose relies on correctly attributing patients to the trainees responsible for their care. EHR activity logs are useful for attributing interns to inpatients but not for attributing supervising residents, who often have no inpatient EHR usage obligations, and therefore may generate no digital "footprints" on a given patient-day from which to ascertain attribution.

APPROACH

The authors developed and tested a novel team-centered binary logistic regression model leveraging EHR activity logs from July 1, 2018, to June 30, 2019, for pediatric hospital medicine (PHM) supervising residents at the University of California, San Francisco. Unlike patient-centered models that determine daily attribution according to the trainee generating the greatest relative activity in individual patients' charts, the team-centered approach predicts daily attribution based on the trainee generating EHR activity across the greatest proportion of a team's patients. To assess generalizability, the authors similarly modeled supervising resident attribution in adult hospital medicine (AHM) and orthopedic surgery (OS).

OUTCOMES

For PHM, AHM, and OS, 1,100, 1,399, and 803 unique patient encounters and 29, 62, and 10 unique supervising residents were included, respectively. Team-centered models outperformed patient-centered models for the 3 specialties, with respective accuracies of 85.4% versus 72.4% (PHM), 88.7% versus 75.4% (AHM), and 69.3% versus 51.6% (OS; P < .001 for all). AHM and PHM models demonstrated relative generalizability to one another while OS did not.

NEXT STEPS

Validation at other institutions will be essential to understanding the potential for generalizability of this approach. Accurately attributed data are likely to be trusted more by trainees, enabling programs to operationalize feedback for use cases including performance measurement, case mix assessment, and postdischarge opportunities for follow-up learning.

Using event logs to observe interactions with electronic health records: an updated scoping review shows increasing use of vendor-derived measures

OBJECTIVE

The aim of this article is to compare the aims, measures, methods, limitations, and scope of studies that employ vendor-derived and investigator-derived measures of electronic health record (EHR) use, and to assess measure consistency across studies.

MATERIALS AND METHODS

We searched PubMed for articles published between July 2019 and December 2021 that employed measures of EHR use derived from EHR event logs. We coded the aims, measures, methods, limitations, and scope of each article and compared articles employing vendor-derived and investigator-derived measures.

RESULTS

One hundred and two articles met inclusion criteria; 40 employed vendor-derived measures, 61 employed investigator-derived measures, and 1 employed both. Studies employing vendor-derived measures were more likely than those employing investigator-derived measures to observe EHR use only in ambulatory settings (83% vs 48%, P = .002) and only by physicians or advanced practice providers (100% vs 54% of studies, P < .001). Studies employing vendor-derived measures were also more likely to measure durations of EHR use (P < .001 for 6 different activities), but definitions of measures such as time outside scheduled hours varied widely. Eight articles reported measure validation. The reported limitations of vendor-derived measures included measure transparency and availability for certain clinical settings and roles.

DISCUSSION

Vendor-derived measures are increasingly used to study EHR use, but only by certain clinical roles. Although poorly validated and variously defined, both vendor- and investigator-derived measures of EHR time are widely reported.

CONCLUSION

The number of studies using event logs to observe EHR use continues to grow, but with inconsistent measure definitions and significant differences between studies that employ vendor-derived and investigator-derived measures.

Characterizing intensive care unit rounding teams using meta-data from the electronic health record

PURPOSE

Teamwork is an important determinant of outcomes in the intensive care unit (ICU), yet the nature of individual ICU teams remains poorly understood. We examined whether meta-data in the form of digital signatures in the electronic health record (EHR) could be used to identify and characterize ICU teams.

METHODS

We analyzed EHR data from 27 ICUs over one year. We linked intensivist physicians, nurses, and respiratory therapists to individual patients based on selected EHR meta-data. We then characterized ICU teams by their members' overall past experience and shared past experience; and used network analysis to characterize ICUs by their network's density and centralization.

RESULTS

We identified 2327 unique providers and 30,892 unique care teams. Teams varied based on their average team member experience (median and total range: 262.2 shifts, 9.0-706.3) and average shared experience (median and total range: 13.2 shared shifts, 1.0-99.3). ICUs varied based on their network's density (median and total range: 0.12, 0.07-0.23), degree centralization (0.50, 0.35-0.65) and closeness centralization (0.45, 0.11-0.60). In a regression analysis, this variation was only partially explained by readily observable ICU characteristics.

CONCLUSIONS

EHR meta-data can assist in the characterization of ICU teams, potentially providing novel insight into strategies to measure and improve team function in critical care.

A Model for Work Intensity in a Pediatric Training Program

BACKGROUND

The Accreditation Council for Graduate Medical Education (ACGME) requires residency programs to monitor scheduling, work intensity, and work compression.

OBJECTIVE

We aimed to create a model for assessing intern work intensity by examining patient and clinical factors in our electronic health systems using multiple linear regression.

METHODS

We identified measurable factors that may contribute to resident work intensity within our electronic health systems. In the spring of 2021, we surveyed interns on pediatric hospital medicine rotations each weekday over 5 blocks to rank their daily work intensity on a scale from -100 (bored) to +100 (exasperated). We queried our electronic systems to identify patient care activities completed by study participants on days they were surveyed. We used multiple linear regression to identify factors that correlate with subjective scores of work intensity.

RESULTS

Nineteen unique interns provided 102 survey responses (28.3% response rate) during the study period. The mean work intensity score was 9.82 (SD=44.27). We identified 19 candidate variables for the regression model. The most significantly associated variables from our univariate regression model were text messages (β=0.432, P<.0009, R²=0.105), orders entered (β=0.207, P<.0002, R²=0.128), and consults ordered (β=0.268, P=.022, R²=0.053). Stepwise regression produced a reduced model (R²=0.247) including text messages (β=0.379, P=.002), patient transfers (β=-1.405, P=.15), orders entered (β=0.186, P<.001), and national patients (β=-0.873, P=.035).

CONCLUSIONS

Our study demonstrates that data extracted from electronic systems can be used to estimate resident work intensity.

Design, Usability, and Acceptability of a Needs-Based, Automated Dashboard to Provide Individualized Patient-Care Data to Pediatric Residents

BACKGROUND AND OBJECTIVES

Pediatric residency programs are required by the Accreditation Council for Graduate Medical Education to provide residents with patient-care and quality metrics to facilitate self-identification of knowledge gaps to prioritize improvement efforts. Trainees are interested in receiving this data, but this is a largely unmet need. Our objectives were to (1) design and implement an automated dashboard providing individualized data to residents, and (2) examine the usability and acceptability of the dashboard among pediatric residents.

METHODS

We developed a dashboard containing individualized patient-care data for pediatric residents with emphasis on needs identified by residents and residency leadership. To build the dashboard, we created a connection from a clinical data warehouse to data visualization software. We allocated patients to residents based on note authorship and created individualized reports with masked identities that preserved anonymity. After development, we conducted usability and acceptability testing with 11 resident users utilizing a mixed-methods approach. We conducted interviews and anonymous surveys which evaluated technical features of the application, ease of use, as well as users' attitudes toward using the dashboard. Categories and subcategories from usability interviews were identified using a content analysis approach.

RESULTS

Our dashboard provides individualized metrics including diagnosis exposure counts, procedure counts, efficiency metrics, and quality metrics. In content analysis of the usability testing interviews, the most frequently mentioned use of the dashboard was to aid a resident's self-directed learning. Residents had few concerns about the dashboard overall. Surveyed residents found the dashboard easy to use and expressed intention to use the dashboard in the future.

CONCLUSION

Automated dashboards may be a solution to the current challenge of providing trainees with individualized patient-care data. Our usability testing revealed that residents found our dashboard to be useful and that they intended to use this tool to facilitate development of self-directed learning plans.

Predicting next-day discharge via electronic health record access logs

OBJECTIVE

Hospital capacity management depends on accurate real-time estimates of hospital-wide discharges. Estimation by a clinician requires an excessively large amount of effort and, even when attempted, accuracy in forecasting next-day patient-level discharge is poor. This study aims to support next-day discharge predictions with machine learning by incorporating electronic health record (EHR) audit log data, a resource that captures EHR users' granular interactions with patients' records by communicating various semantics and has been neglected in outcome predictions.

MATERIALS AND METHODS

This study focused on the EHR data for all adults admitted to Vanderbilt University Medical Center in 2019. We learned multiple advanced models to assess the value that EHR audit log data adds to the daily prediction of discharge likelihood within 24 h and to compare different representation strategies. We applied Shapley additive explanations to identify the most influential types of user-EHR interactions for discharge prediction.

RESULTS

The data include 26 283 inpatient stays, 133 398 patient-day observations, and 819 types of user-EHR interactions. The model using the count of each type of interaction in the recent 24 h and other commonly used features, including demographics and admission diagnoses, achieved the highest area under the receiver operating characteristics (AUROC) curve of 0.921 (95% CI: 0.919-0.923). By contrast, the model lacking user-EHR interactions achieved a worse AUROC of 0.862 (0.860-0.865). In addition, 10 of the 20 (50%) most influential factors were user-EHR interaction features.

CONCLUSION

EHR audit log data contain rich information such that it can improve hospital-wide discharge predictions.

Use of clinical data to augment healthcare worker contact tracing during the COVID-19 pandemic

OBJECTIVE

This work examined the secondary use of clinical data from the electronic health record (EHR) for screening our healthcare worker (HCW) population for potential exposures to patients with coronavirus disease 2019.

MATERIALS AND METHODS

We conducted a cross-sectional study at a free-standing, quaternary care pediatric hospital comparing first-degree, patient-HCW pairs identified by the hospital's COVID-19 contact tracing team (CTT) to those identified using EHR clinical event data (EHR Report). The primary outcome was the number of patient-HCW pairs detected by each process.

RESULTS

Among 233 patients with COVID-19, our EHR Report identified 4,116 patient-HCW pairs, including 2,365 (30.0%) of the 7,890 pairs detected by the CTT. The EHR Report also revealed 1,751 pairs not identified by the CTT. The highest number of patient-HCW pairs per patient was detected in the inpatient care venue. Nurses comprised the most frequently identified HCW role overall.

CONCLUSION

Automated methods to screen HCWs for potential exposure to patients with COVID-19 using clinical event data from the EHR are likely to improve epidemiologic surveillance by contact tracing programs and represent a viable and readily available strategy which should be considered by other institutions.

Resident Physician Experience and Duration of Electronic Health Record Use

BACKGROUND

Electronic health records (EHRs) demand a significant amount of physician time for documentation, orders, and communication during care delivery. Resident physicians already work long hours as they gain experience and develop both clinical and socio-technical skills.

OBJECTIVES

Measure how much time resident physicians spend in the EHR during clinic hours and after-hours, and how EHR usage changes as they gain experience over a 12-month period.

METHODS

Longitudinal descriptive study where participants were 622 resident physicians across postgraduate year cohorts (of 948 resident physicians at the institution, 65.6%) working in an ambulatory setting from July 2017 to June 2018. Time spent in the EHR per patient, patients records documented per day, and proportion of EHR time spent after-hours were the outcome, while the number of months of ambulatory care experience was the predictor.

RESULTS

Resident physicians spent an average of 45.6 minutes in the EHR per patient, with 13.5% of that time spent after-hours. Over 12 months of ambulatory experience, resident physicians reduced their EHR time per patient and saw more patients per day, but the proportion of EHR time after-hours did not change.

CONCLUSION

Resident physicians spend a significant amount of time working in the EHR, both during and after clinic hours. While residents improve efficiency in reducing EHR time per patient, they do not reduce the proportion of EHR time spent after-hours. Concerns over the impact of EHRs on physician well-being should include recognition of the burden of EHR usage on early-career physicians.

Improving ACGME Compliance for Obstetric Anesthesiology Fellows Using an Automated Email Notification System

BACKGROUND

The Accreditation Council for Graduate Medical Education establishes minimum case requirements for trainees. In the subspecialty of obstetric anesthesiology, requirements for fellow participation in nonobstetric antenatal procedures pose a particular challenge due to the physical location remote from labor and delivery and frequent last-minute scheduling.

OBJECTIVES

In response to this challenge, we implemented an informatics-based notification system, with the aim of increasing fellow participation in nonobstetric antenatal surgeries.

METHODS

In December 2014 an automated email notification system to inform obstetric anesthesiology fellows of scheduled nonobstetric surgeries in pregnant patients was initiated. Cases were identified via daily automated query of the preoperative evaluation database looking for structured documentation of current pregnancy. Information on flagged cases including patient medical record number, operating room location, and date and time of procedure were communicated to fellows via automated email daily. Median fellow participation in nonobstetric antenatal procedures per quarter before and after implementation were compared using an exact Wilcoxon-Mann-Whitney test due to low baseline absolute counts. The fraction of antenatal cases representing nonobstetric procedures completed by fellows before and after implementation was compared using a Fisher's exact test.

RESULTS

The number of nonobstetric antenatal cases logged by fellows per quarter increased significantly following implementation, from median 0[0,1] to 3[1,6] cases/quarter (p = 0.007). Additionally, nonobstetric antenatal cases completed by fellows as a percentage of total antenatal cases completed increased from 14% in preimplementation years to 52% in postimplementation years (p < 0.001).

CONCLUSION

Through an automated email system to identify nonobstetric antenatal procedures in pregnant patients, we were able to increase the number of these cases completed by fellows during 3 years following implementation.