An automated electronic case log: using electronic information systems to assess training in emergency medicine

Uses of Electronic Health Record Data to Measure the Clinical Learning Environment of Graduate Medical Education Trainees: A Systematic Review

PURPOSE

This study systematically reviews the uses of electronic health record (EHR) data to measure graduate medical education (GME) trainee competencies.

METHOD

In January 2022, the authors conducted a systematic review of original research in MEDLINE from database start to December 31, 2021. The authors searched for articles that used the EHR as their data source and in which the individual GME trainee was the unit of observation and/or unit of analysis. The database query was intentionally broad because an initial survey of pertinent articles identified no unifying Medical Subject Heading terms. Articles were coded and clustered by theme and Accreditation Council for Graduate Medical Education (ACGME) core competency.

RESULTS

The database search yielded 3,540 articles, of which 86 met the study inclusion criteria. Articles clustered into 16 themes, the largest of which were trainee condition experience (17 articles), work patterns (16 articles), and continuity of care (12 articles). Five of the ACGME core competencies were represented (patient care and procedural skills, practice-based learning and improvement, systems-based practice, medical knowledge, and professionalism). In addition, 25 articles assessed the clinical learning environment.

CONCLUSIONS

This review identified 86 articles that used EHR data to measure individual GME trainee competencies, spanning 16 themes and 6 competencies and revealing marked between-trainee variation. The authors propose a digital learning cycle framework that arranges sequentially the uses of EHR data within the cycle of clinical experiential learning central to GME. Three technical components necessary to unlock the potential of EHR data to improve GME are described: measures, attribution, and visualization. Partnerships between GME programs and informatics departments will be pivotal in realizing this opportunity.

Attributing Patients to Pediatric Residents Using Electronic Health Record Features Augmented with Audit Logs

OBJECTIVE

Patient attribution, or the process of attributing patient-level metrics to specific providers, attempts to capture real-life provider-patient interactions (PPI). Attribution holds wide-ranging importance, particularly for outcomes in graduate medical education, but remains a challenge. We developed and validated an algorithm using EHR data to identify pediatric resident PPIs (rPPIs).

METHODS

We prospectively surveyed residents in three care settings to collect self-reported rPPIs. Participants were surveyed at the end of primary care clinic, emergency department (ED), and inpatient shifts, shown a patient census list, asked to mark the patients with whom they interacted, and encouraged to provide a short rationale behind the marked interaction. We extracted routine EHR data elements, including audit logs, note contribution, order placement, care team assignment, and chart closure, and applied a logistic regression classifier to the data to predict rPPIs in each care setting. We also performed a comment analysis of the resident-reported rationales in the inpatient care setting to explore perceived patient interactions in a complicated workflow.

RESULTS

We surveyed 81 residents over 111 shifts and identified 579 patient interactions. Among EHR extracted data, time-in-chart was the best predictor in all three care settings (primary care clinic: odds ratio [OR] = 19.36, 95% confidence interval [CI]: 4.19-278.56; ED: OR = 19.06, 95% CI: 9.53-41.65' inpatient: OR = 2.95, 95% CI: 2.23-3.97). Primary care clinic and ED specific models had c-statistic values > 0.98, while the inpatient-specific model had greater variability (c-statistic = 0.89). Of 366 inpatient rPPIs, residents provided rationales for 90.1%, which were focused on direct involvement in a patient's admission or transfer, or care as the front-line ordering clinician (55.6%).

CONCLUSION

Classification models based on routinely collected EHR data predict resident-defined rPPIs across care settings. While specific to pediatric residents in this study, the approach may be generalizable to other provider populations and scenarios in which accurate patient attribution is desirable.

Pediatric Case Exposure During Emergency Medicine Residency

OBJECTIVE

While emergency medicine (EM) physicians treat the majority of pediatric EM (PEM) patients in the United States, little is known about their PEM experience during training. The primary objective was to characterize the pediatric case exposure and compare to established EM residency training curricula among EM residents across five U.S. residency programs.

METHODS

We performed a multicenter medical record review of all pediatric patients (aged < 18 years) seen by the 2015 graduating resident physicians at five U.S. EM training programs. Resident-level counts of pediatric patients were measured and specific counts were classified by the 2016 Model of Clinical Practice of Emergency Medicine (MCP) and Pediatric Emergency Care Applied Research Network (PECARN) diagnostic categories. We assessed variability between residents and between programs.

RESULTS

A total of 36,845 children were managed by 68 residents across all programs. The median age was 6 years. The median number of patients per resident was 660 with an interquartile range of 336. The most common PECARN diagnostic categories were trauma, gastrointestinal, and respiratory disease. Thirty-two core MCP diagnoses (43% of MCP list) were not seen by at least 50% of the residents. We found statistically significant variability between programs in both PECARN diagnostic categories (p < 0.01) and MCP diagnoses (p < 0.01).

CONCLUSION

There is considerable variation in the number of pediatric patients and the diagnostic case volume seen by EM residents. The relationship between this case variability and competence upon graduation is unknown; further investigation is warranted to better inform program-specific curricula and to guide training requirements in EM.

Automated Reporting of Trainee Metrics Using Electronic Clinical Systems

BACKGROUND

The Accreditation Council for Graduate Medical Education has called for increased emphasis on reporting objective performance measures to trainees and programs. However, reporting of objective measures, including clinical volume, is largely omitted from training programs.

OBJECTIVE

To use automated electronic medical systems at a tertiary pediatric care hospital to create a dashboard that reports objective trainee and program metrics, including clinical volume and diagnoses in a pediatrics residency.

METHODS

We queried an enterprise data warehouse that aggregates data daily from multiple hospital systems to identify patient encounters during which senior pediatrics residents at Boston Children's Hospital had entered documentation over a 9-month period. From this query, we created a filterable dashboard to display clinical volume and diagnosis data by individual resident and in aggregate.

RESULTS

A total of 44 of 45 senior residents (98%) in the program were included in analysis. We identified 12 198 patient encounters during which a senior pediatrics resident had entered documentation; these included a median of 332 inpatient encounters per resident, 122 emergency department encounters, and 84 outpatient encounters. The most common diagnoses stratified by clinical site were: inpatient - dehydration (median = 61); emergency department - long-term/current drug therapy (median = 16); and outpatient - encounter for immunization (median = 48).

CONCLUSIONS

We used electronic health record systems to generate performance dashboards for trainees in a pediatrics residency across different sites of care with reported volume by diagnosis. Our dashboards provide feedback to program leadership regarding individual and aggregate trainee experience and allow individual trainees to compare their clinical exposure to peers.

Using the Electronic Health Record to Identify Educational Gaps for Internal Medicine Interns

BACKGROUND

An important component of internal medicine residency is clinical immersion in core rotations to expose first-year residents to common diagnoses.

OBJECTIVE

Quantify intern experience with common diagnoses through clinical documentation in an electronic health record.

METHODS

We analyzed all clinical notes written by postgraduate year (PGY) 1, PGY-2, and PGY-3 residents on medicine service at an academic medical center July 1, 2012, through June 30, 2014. We quantified the number of notes written by PGY-1s at 1 of 3 hospitals where they rotate, by the number of notes written about patients with a specific principal billing diagnosis, which we defined as diagnosis-days. We used the International Classification of Diseases 9 (ICD-9) and the Clinical Classification Software (CCS) to group the diagnoses.

RESULTS

We analyzed 53 066 clinical notes covering 10 022 hospitalizations with 1436 different ICD-9 diagnoses spanning 217 CCS diagnostic categories. The 10 most common ICD-9 diagnoses accounted for 23% of diagnosis-days, while the 10 most common CCS groupings accounted for more than 40% of the diagnosis-days. Of 122 PGY-1s, 107 (88%) spent at least 2 months on the service, and 3% were exposed to all of the top 10 ICD-9 diagnoses, while 31% had experience with fewer than 5 of the top 10 diagnoses. In addition, 17% of PGY-1s saw all top 10 CCS diagnoses, and 5% had exposure to fewer than 5 CCS diagnoses.

CONCLUSIONS

Automated detection of clinical experience may help programs review inpatient clinical experiences of PGY-1s.

Quantifying internal medicine resident clinical experience using resident-selected primary diagnosis codes

BACKGROUND

While experiential learning is a desirable goal of residency education, little is known regarding the actual clinical experience of internal medicine residents during their training.

METHODS

We modified an electronic patient handoff tool to include a system for resident entry of a primary diagnosis for each of their patients. Using the International Classification of Diseases, Ninth Revision (ICD-9) system, we created two methods to select the code: 1) an organ system-based dropdown list containing frequently used codes; and 2) a search option for the complete ICD-9 database. The codes were then grouped using ICD-9 categorization.

RESULTS

A total of 7562 resident-patient diagnostic encounters were studied. A wide spectrum of clinical conditions was observed, with symptoms and ill-defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/immune disorders, and disorders of the skin and subcutaneous tissue accounting for about 86% of resident clinical experience. Symptoms and ill-defined conditions were noted to represent a sizable portion of resident clinical experience. Within this category, the most common conditions were fever; abdominal pain; and chest pain, unspecified.

CONCLUSIONS

Analysis of resident-selected ICD-9 codes might serve as a method to attempt to define resident clinical experience, and may be useful in the development of innovative experiential learning-based residency curricula. This might also be used to assess gaps in experiential learning at the program or resident level, and may serve to identify topics that require additional teaching supplementation.

Use of an automated electronic case log to assess fellowship training: tracking the pediatric emergency medicine experience

The Accreditation Council for Graduate Medical Education has mandated the assessment of medical training across 6 core competencies. The patient care competency is at the core of medical training. With the introduction of patient tracking systems used in emergency departments, patient-physician encounters can be systematically studied. The combination of tracking data with other clinical information systems can be used to create an electronic case log to quantify the experience of fellows, thereby offering a summative measure of the patient care competency. We used an automated case log to assess clinical exposure in our pediatric emergency medicine fellowship.