Automated Reporting of Trainee Metrics Using Electronic Clinical Systems

An Automated System for Physician Trainee Procedure Logging via Electronic Health Records

Importance

Procedural proficiency is a core competency for graduate medical education; however, procedural reporting often relies on manual workflows that are duplicative and generate data whose validity and accuracy are difficult to assess. Failure to accurately gather these data can impede learner progression, delay procedures, and negatively impact patient safety.

Objective

To examine accuracy and procedure logging completeness of a system that extracts procedural data from an electronic health record system and uploads these data securely to an application used by many residency programs for accreditation.

Design, Setting, and Participants

This quality improvement study of all emergency medicine resident physicians at University of California, San Diego Health was performed from May 23, 2023, to June 25, 2023.

Exposures

Automated system for procedure data extraction and upload to a residency management software application.

Main Outcomes and Measures

The number of procedures captured by the automated system when running silently compared with manually logged procedures in the same timeframe, as well as accuracy of the data upload.

Results

Forty-seven residents participated in the initial silent assessment of the extraction component of the system. During a 1-year period (May 23, 2022, to May 7, 2023), 4291 procedures were manually logged by residents, compared with 7617 procedures captured by the automated system during the same period, representing a 78% increase. During assessment of the upload component of the system (May 8, 2023, to June 25, 2023), a total of 1353 procedures and patient encounters were evaluated, with the system operating with a sensitivity of 97.4%, specificity of 100%, and overall accuracy of 99.5%.

Conclusions and Relevance

In this quality improvement study of emergency medicine resident physicians, an automated system demonstrated that reliance on self-reported procedure logging resulted in significant procedural underreporting compared with the use of data obtained at the point of performance. Additionally, this system afforded a degree of reliability and validity heretofore absent from the usual after-the-fact procedure logging workflows while using a novel application programming interface-based approach. To our knowledge, this system constitutes the first generalizable implementation of an automated solution to a problem that has existed in graduate medical education for decades.

Uses of Electronic Health Record Data to Measure the Clinical Learning Environment of Graduate Medical Education Trainees: A Systematic Review

PURPOSE

This study systematically reviews the uses of electronic health record (EHR) data to measure graduate medical education (GME) trainee competencies.

METHOD

In January 2022, the authors conducted a systematic review of original research in MEDLINE from database start to December 31, 2021. The authors searched for articles that used the EHR as their data source and in which the individual GME trainee was the unit of observation and/or unit of analysis. The database query was intentionally broad because an initial survey of pertinent articles identified no unifying Medical Subject Heading terms. Articles were coded and clustered by theme and Accreditation Council for Graduate Medical Education (ACGME) core competency.

RESULTS

The database search yielded 3,540 articles, of which 86 met the study inclusion criteria. Articles clustered into 16 themes, the largest of which were trainee condition experience (17 articles), work patterns (16 articles), and continuity of care (12 articles). Five of the ACGME core competencies were represented (patient care and procedural skills, practice-based learning and improvement, systems-based practice, medical knowledge, and professionalism). In addition, 25 articles assessed the clinical learning environment.

CONCLUSIONS

This review identified 86 articles that used EHR data to measure individual GME trainee competencies, spanning 16 themes and 6 competencies and revealing marked between-trainee variation. The authors propose a digital learning cycle framework that arranges sequentially the uses of EHR data within the cycle of clinical experiential learning central to GME. Three technical components necessary to unlock the potential of EHR data to improve GME are described: measures, attribution, and visualization. Partnerships between GME programs and informatics departments will be pivotal in realizing this opportunity.

User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard

OBJECTIVES

Clinical Competency Committee (CCC) members employ varied approaches to the review process. This makes the design of a competency assessment dashboard that fits the needs of all members difficult. This work details a user-centered evaluation of a dashboard currently utilized by the Internal Medicine Clinical Competency Committee (IM CCC) at the University of Cincinnati College of Medicine and generated design recommendations.

METHODS

Eleven members of the IM CCC participated in semistructured interviews with the research team. These interviews were recorded and transcribed for analysis. The three design research methods used in this study included process mapping (workflow diagrams), affinity diagramming, and a ranking experiment.

RESULTS

Through affinity diagramming, the research team identified and organized opportunities for improvement about the current system expressed by study participants. These areas include a time-consuming preprocessing step, lack of integration of data from multiple sources, and different workflows for each step in the review process. Finally, the research team categorized nine dashboard components based on rankings provided by the participants.

CONCLUSION

We successfully conducted user-centered evaluation of an IM CCC dashboard and generated four recommendations. Programs should integrate quantitative and qualitative feedback, create multiple views to display these data based on user roles, work with designers to create a usable, interpretable dashboard, and develop a strong informatics pipeline to manage the system. To our knowledge, this type of user-centered evaluation has rarely been attempted in the medical education domain. Therefore, this study provides best practices for other residency programs to evaluate current competency assessment tools and to develop new ones.

Using Resident-Sensitive Quality Measures Derived From Electronic Health Record Data to Assess Residents' Performance in Pediatric Emergency Medicine

PURPOSE

Traditional quality metrics do not adequately represent the clinical work done by residents and, thus, cannot be used to link residency training to health care quality. This study aimed to determine whether electronic health record (EHR) data can be used to meaningfully assess residents' clinical performance in pediatric emergency medicine using resident-sensitive quality measures (RSQMs).

METHOD

EHR data for asthma and bronchiolitis RSQMs from Cincinnati Children's Hospital Medical Center, a quaternary children's hospital, between July 1, 2017, and June 30, 2019, were analyzed by ranking residents based on composite scores calculated using raw, unadjusted, and case-mix adjusted latent score models, with lower percentiles indicating a lower quality of care and performance. Reliability and associations between the scores produced by the 3 scoring models were compared. Resident and patient characteristics associated with performance in the highest and lowest tertiles and changes in residents' rank after case-mix adjustments were also identified.

RESULTS

274 residents and 1,891 individual encounters of bronchiolitis patients aged 0-1 as well as 270 residents and 1,752 individual encounters of asthmatic patients aged 2-21 were included in the analysis. The minimum reliability requirement to create a composite score was met for asthma data (α = 0.77), but not bronchiolitis (α = 0.17). The asthma composite scores showed high correlations ( r = 0.90-0.99) between raw, latent, and adjusted composite scores. After case-mix adjustments, residents' absolute percentile rank shifted on average 10 percentiles. Residents who dropped by 10 or more percentiles were likely to be more junior, saw fewer patients, cared for less acute and younger patients, or had patients with a longer emergency department stay.

CONCLUSIONS

For some clinical areas, it is possible to use EHR data, adjusted for patient complexity, to meaningfully assess residents' clinical performance and identify opportunities for quality improvement.

A Model for Work Intensity in a Pediatric Training Program

BACKGROUND

The Accreditation Council for Graduate Medical Education (ACGME) requires residency programs to monitor scheduling, work intensity, and work compression.

OBJECTIVE

We aimed to create a model for assessing intern work intensity by examining patient and clinical factors in our electronic health systems using multiple linear regression.

METHODS

We identified measurable factors that may contribute to resident work intensity within our electronic health systems. In the spring of 2021, we surveyed interns on pediatric hospital medicine rotations each weekday over 5 blocks to rank their daily work intensity on a scale from -100 (bored) to +100 (exasperated). We queried our electronic systems to identify patient care activities completed by study participants on days they were surveyed. We used multiple linear regression to identify factors that correlate with subjective scores of work intensity.

RESULTS

Nineteen unique interns provided 102 survey responses (28.3% response rate) during the study period. The mean work intensity score was 9.82 (SD=44.27). We identified 19 candidate variables for the regression model. The most significantly associated variables from our univariate regression model were text messages (β=0.432, P<.0009, R²=0.105), orders entered (β=0.207, P<.0002, R²=0.128), and consults ordered (β=0.268, P=.022, R²=0.053). Stepwise regression produced a reduced model (R²=0.247) including text messages (β=0.379, P=.002), patient transfers (β=-1.405, P=.15), orders entered (β=0.186, P<.001), and national patients (β=-0.873, P=.035).

CONCLUSIONS

Our study demonstrates that data extracted from electronic systems can be used to estimate resident work intensity.

Design, Usability, and Acceptability of a Needs-Based, Automated Dashboard to Provide Individualized Patient-Care Data to Pediatric Residents

BACKGROUND AND OBJECTIVES

Pediatric residency programs are required by the Accreditation Council for Graduate Medical Education to provide residents with patient-care and quality metrics to facilitate self-identification of knowledge gaps to prioritize improvement efforts. Trainees are interested in receiving this data, but this is a largely unmet need. Our objectives were to (1) design and implement an automated dashboard providing individualized data to residents, and (2) examine the usability and acceptability of the dashboard among pediatric residents.

METHODS

We developed a dashboard containing individualized patient-care data for pediatric residents with emphasis on needs identified by residents and residency leadership. To build the dashboard, we created a connection from a clinical data warehouse to data visualization software. We allocated patients to residents based on note authorship and created individualized reports with masked identities that preserved anonymity. After development, we conducted usability and acceptability testing with 11 resident users utilizing a mixed-methods approach. We conducted interviews and anonymous surveys which evaluated technical features of the application, ease of use, as well as users' attitudes toward using the dashboard. Categories and subcategories from usability interviews were identified using a content analysis approach.

RESULTS

Our dashboard provides individualized metrics including diagnosis exposure counts, procedure counts, efficiency metrics, and quality metrics. In content analysis of the usability testing interviews, the most frequently mentioned use of the dashboard was to aid a resident's self-directed learning. Residents had few concerns about the dashboard overall. Surveyed residents found the dashboard easy to use and expressed intention to use the dashboard in the future.

CONCLUSION

Automated dashboards may be a solution to the current challenge of providing trainees with individualized patient-care data. Our usability testing revealed that residents found our dashboard to be useful and that they intended to use this tool to facilitate development of self-directed learning plans.

Resident Perceptions of a Publicly Disclosed Daily Productivity Dashboard

Introduction

Following resident requests, we created a public metrics dashboard to inform residents of their daily productivity. Our goal was to iteratively improve the dashboard based on resident feedback and to measure the impact of reviewing aggregate data on self-perceived productivity.

Methods

A 10-question anonymous survey was completed by our postgraduate year 1–3 residents. Residents answered questions on the dashboard and rated their own productivity before and after reviewing aggregate peer-comparison data. Using the Wilcoxon signed-rank test we calculated summary statistics for survey questions and compared distributions of pre- and post-test, self-rated productivity scores.

Results

All 43 eligible residents completed the survey (response rate 100%). Thirteen (30%) residents “rarely” or “never” reviewed the dashboard. No respondents felt the dashboard measured their productivity or quality of care “extremely accurately” or “very accurately.” Seven (16%) residents felt “very” or “extremely pressured” to change their practice patterns based on the metrics provided, and 28 (65%) would have preferred private over public feedback. Fifteen residents (35%) changed their self-perceived rank after viewing peer-comparison data, although not significantly in a particular direction (z = 0.71, P = 0.48).

Conclusion

Residents did not view the presented metrics as reflective of their productivity or quality of care. Viewing the dashboard did not lead to statistically significant changes in resident self-perception of productivity. This finding highlights the need for expanding the resident conversation and education on metrics, given their frequent inclusion in attending physician workforce payment and incentive models.

Use of clinical data to augment healthcare worker contact tracing during the COVID-19 pandemic

OBJECTIVE

This work examined the secondary use of clinical data from the electronic health record (EHR) for screening our healthcare worker (HCW) population for potential exposures to patients with coronavirus disease 2019.

MATERIALS AND METHODS

We conducted a cross-sectional study at a free-standing, quaternary care pediatric hospital comparing first-degree, patient-HCW pairs identified by the hospital's COVID-19 contact tracing team (CTT) to those identified using EHR clinical event data (EHR Report). The primary outcome was the number of patient-HCW pairs detected by each process.

RESULTS

Among 233 patients with COVID-19, our EHR Report identified 4,116 patient-HCW pairs, including 2,365 (30.0%) of the 7,890 pairs detected by the CTT. The EHR Report also revealed 1,751 pairs not identified by the CTT. The highest number of patient-HCW pairs per patient was detected in the inpatient care venue. Nurses comprised the most frequently identified HCW role overall.

CONCLUSION

Automated methods to screen HCWs for potential exposure to patients with COVID-19 using clinical event data from the EHR are likely to improve epidemiologic surveillance by contact tracing programs and represent a viable and readily available strategy which should be considered by other institutions.

"EMERGing" Electronic Health Record Data Metrics: Insights and Implications for Assessing Residents' Clinical Performance in Emergency Medicine

OBJECTIVES

Competency-based medical education requires that residents are provided with frequent opportunities to demonstrate competence as well as receive effective feedback about their clinical performance. To meet this goal, we investigated how data collected by the electronic health record (EHR) might be used to assess emergency medicine (EM) residents' independent and interdependent clinical performance and how such information could be represented in an EM resident report card.

METHODS

Following constructivist grounded theory methodology, individual semistructured interviews were conducted in 2017 with 10 EM faculty and 11 EM residents across all 5 postgraduate years. In addition to open-ended questions, participants were presented with an emerging list of EM practice metrics and asked to comment on how valuable each would be in assessing resident performance. Additionally, we asked participants the extent to which each metric captured independent or interdependent performance. Data collection and analysis were iterative; analysis employed constant comparative inductive methods.

RESULTS

Participants refined and eliminated metrics as well as added new metrics specific to the assessment of EM residents (e.g., time between signup and first orders). These clinical practice metrics based on data from our EHR database were organized along a spectrum of independent/interdependent performance. We conclude with discussions about the relationship among these metrics, issues in interpretation, and implications of using EHR for assessment purposes.

CONCLUSIONS

Our findings document a systematic approach for developing EM resident assessments, based on EHR data, which incorporate the perspectives of both clinical faculty and residents. Our work has important implications for capturing residents' contributions to clinical performances and distinguishing between independent and interdependent metrics in collaborative workplace-based settings.

A Descriptive Analysis of the Cumulative Experiences of Emergency Medicine Residents in the Pediatric Emergency Department

OBJECTIVES

Most children seeking emergency care are evaluated in general emergency departments (EDs). The cumulative pediatric clinical experiences of emergency medicine (EM) residents are largely unknown. This study examined EM resident pediatric clinical experience through the lens of the Accreditation Council for Graduate Medical Education requirements and the Model of the Clinical Practice of Emergency Medicine.

METHODS

Retrospective, observational study of the cumulative clinical experience of two classes of EM residents from a 4-year training program at two pediatric EDs of a quaternary care pediatric center. A database of resident patient encounters was generated from the electronic medical record. Experiences classified included: diagnosis categories per the Model of the Clinical Practice of Emergency Medicine, procedures, and resuscitations. Results were stratified by age, acuity, and disposition.

RESULTS

Twenty-five EM residents evaluated 17,642 patients (median = 723). Most patients (73.5%) were emergent acuity (Emergency Severity Index triage level 2 or 3 or non-intensive care admission); 2% were critical. Residents participated in 598 (median = 22) medical resuscitations and 483 (median = 19) trauma resuscitations. Minor procedures (e.g., laceration repair) were commonly performed; critical procedures (e.g., intubation) were rare. Exposure to neonates was infrequent and pediatric deaths were rare. Abdominal pain (5.7%), asthma exacerbation (4.6%), and fever (3.8%) were the most common diagnoses.

CONCLUSIONS

Emergency medicine residents encountered a wide array of pediatric diagnoses throughout training and performed a substantial number of common pediatric procedures. Exposure to critical acuity and procedures, neonatal pathology, and certain pediatric-specific diagnoses, such as congenital heart disease, was limited despite training in a large, quaternary care children's hospital. Curriculum development and collaboration should focus on these areas.

Attributing Patients to Pediatric Residents Using Electronic Health Record Features Augmented with Audit Logs

OBJECTIVE

Patient attribution, or the process of attributing patient-level metrics to specific providers, attempts to capture real-life provider-patient interactions (PPI). Attribution holds wide-ranging importance, particularly for outcomes in graduate medical education, but remains a challenge. We developed and validated an algorithm using EHR data to identify pediatric resident PPIs (rPPIs).

METHODS

We prospectively surveyed residents in three care settings to collect self-reported rPPIs. Participants were surveyed at the end of primary care clinic, emergency department (ED), and inpatient shifts, shown a patient census list, asked to mark the patients with whom they interacted, and encouraged to provide a short rationale behind the marked interaction. We extracted routine EHR data elements, including audit logs, note contribution, order placement, care team assignment, and chart closure, and applied a logistic regression classifier to the data to predict rPPIs in each care setting. We also performed a comment analysis of the resident-reported rationales in the inpatient care setting to explore perceived patient interactions in a complicated workflow.

RESULTS

We surveyed 81 residents over 111 shifts and identified 579 patient interactions. Among EHR extracted data, time-in-chart was the best predictor in all three care settings (primary care clinic: odds ratio [OR] = 19.36, 95% confidence interval [CI]: 4.19-278.56; ED: OR = 19.06, 95% CI: 9.53-41.65' inpatient: OR = 2.95, 95% CI: 2.23-3.97). Primary care clinic and ED specific models had c-statistic values > 0.98, while the inpatient-specific model had greater variability (c-statistic = 0.89). Of 366 inpatient rPPIs, residents provided rationales for 90.1%, which were focused on direct involvement in a patient's admission or transfer, or care as the front-line ordering clinician (55.6%).

CONCLUSION

Classification models based on routinely collected EHR data predict resident-defined rPPIs across care settings. While specific to pediatric residents in this study, the approach may be generalizable to other provider populations and scenarios in which accurate patient attribution is desirable.

Trainee and Program Director Perspectives on Meaningful Patient Attribution and Clinical Outcomes Data

BACKGROUND

The Accreditation Council for Graduate Medical Education specifies that trainees must receive clinical outcomes and quality benchmark data at specific levels related to institutional patient populations. Program directors (PDs) are challenged to identify meaningful data and provide them in formats acceptable to trainees.

OBJECTIVE

We sought to understand what types of patients, data/metrics, and data delivery systems trainees and PDs prefer for supplying trainees with clinical outcomes data.

METHODS

Trainees (n = 21) and PDs (n = 12) from multiple specialties participated in focus groups during academic year 2017-2018. They described key themes for providing clinical outcomes data to trainees.

RESULTS

Trainees and PDs differed in how they identified patients for clinical outcomes data for trainees. Trainees were interested in encounters where they felt a sense of responsibility or had autonomy/independent decision-making opportunities, continuity, or learned something new; PDs used broader criteria including all patients cared for by their trainees. Both groups thought trainees should be given trainee-level metrics and consistently highlighted the importance of comparison to peers and/or benchmarks. Both groups found value in "push" and "pull" data systems, although trainees wanted both, while PDs wanted one or the other. Both groups agreed that trainees should review data with specific faculty. Trainees expressed concern about being judged based on their patients' clinical outcomes.

CONCLUSIONS

Trainee and PD perspectives on which patients they would like outcomes data for differed, but they overlapped for types of metrics, formats, and review processes for the data.

A Feasibility Study to Attribute Patients to Primary Interns on Inpatient Ward Teams Using Electronic Health Record Data

PURPOSE

To inform graduate medical education (GME) outcomes at the individual resident level, this study sought a method for attributing care for individual patients to individual interns based on "footprints" in the electronic health record (EHR).

METHOD

Primary interns caring for patients on an internal medicine inpatient service were recorded daily by five attending physicians of record at University of Cincinnati Medical Center in August 2017 and January 2018. These records were considered gold standard identification of primary interns. The following EHR variables were explored to determine representation of primary intern involvement in care: postgraduate year, progress note author, discharge summary author, physician order placement, and logging clicks in the patient record. These variables were turned into quantitative attributes (e.g., progress note author: yes/no), and informative attributes were selected and modeled using a decision tree algorithm.

RESULTS

A total of 1,511 access records were generated; 116 were marked as having a primary intern assigned. All variables except discharge summary author displayed at least some level of importance in the models. The best model achieved 78.95% sensitivity, 97.61% specificity, and an area under the receiver-operator curve of approximately 91%.

CONCLUSIONS

This study successfully predicted primary interns caring for patients on inpatient teams using EHR data with excellent model performance. This provides a foundation for attributing patients to primary interns for the purposes of determining patient diagnoses and complexity the interns see as well as supporting continuous quality improvement efforts in GME.

Documenting or Operating: Where Is Time Spent in General Surgery Residency?

OBJECTIVE

The utilization of electronic health records (EHR) has become essential in the daily activities of physicians for documentation and as an information source. However, the amount of time spent by residents utilizing the EHR has not been thoroughly evaluated, particularly within surgical specialties. This study aims to analyze EHR usage by general surgery residents and to assess the association between this use and case volume at a single academic institution.

DESIGN

For general surgery residents in clinical years (CY) 1-5, de-identified login and logout time data between September 2016 and June 2017 were retrospectively extracted from the Epic EHR (Verona, WI). A binary time series was created for each resident to indicate and track over time whether he or she was utilizing the EHR system. Comparisons between categorical variables were performed with Fisher's exact test. Continuous variables were compared using Wilcoxon rank sum test. Longitudinal linear mixed-effects models were used to assess the EHR usage among the surgery residents. The association between EHR time and the number of operative cases logged was evaluated with Pearson's correlation coefficient.

SETTING

This study was performed by the Department of Surgery in conjunction with the Office of Graduate Medical Education at Duke University Health System.

PARTICIPANTS

All active general surgery residents during the 2016-2017 academic year.

RESULTS

Thirty-six general surgery residents (28 males, 8 females) spent a median of 2.4 hours per day and 23.7 hours per week using the EHR. CY2 had the highest median usage per week (28.9 hours), while CY3 had the lowest (16.7 hours) but no significant difference based on EHR usage was found among the analyzed CYs (p = 0.164). Residents spent significantly more time logged into the EHR during the week compared to weekends and during the day compared to nights (all p < 0.001). For the residency program as a whole, a median of 151.5 total work hours per day was dedicated to documentation. On average, interns on dedicated night rotations spent 7% of their login time outside regularly scheduled duty hours while interns on dedicated day rotations spent 27%. There was no overall correlation between monthly case logs and EHR usage (r = 0.06, p = 0.30); however, CY2 had a significant negative correlation (r = -0.2, p = 0.038).

CONCLUSIONS

In the era of a maximum 80-hour work week, general surgery residents spend a substantial portion, at least 30%, of their time utilizing the EHR. One third of EHR usage by interns occurred outside the scheduled 12-hour shift, demonstrating the difficulties of completing paperwork as part of the scheduled work day. Additionally, the lack of correlation to case logs is likely due to an underestimation of the documentation burden associated with operating, which includes preparatory effort and operative notes. Ultimately, these quantitative EHR usage results will be correlated to burnout prior to implementing programs to improve efficiency and decrease the burden of charting.

Pediatric Case Exposure During Emergency Medicine Residency

OBJECTIVE

While emergency medicine (EM) physicians treat the majority of pediatric EM (PEM) patients in the United States, little is known about their PEM experience during training. The primary objective was to characterize the pediatric case exposure and compare to established EM residency training curricula among EM residents across five U.S. residency programs.

METHODS

We performed a multicenter medical record review of all pediatric patients (aged < 18 years) seen by the 2015 graduating resident physicians at five U.S. EM training programs. Resident-level counts of pediatric patients were measured and specific counts were classified by the 2016 Model of Clinical Practice of Emergency Medicine (MCP) and Pediatric Emergency Care Applied Research Network (PECARN) diagnostic categories. We assessed variability between residents and between programs.

RESULTS

A total of 36,845 children were managed by 68 residents across all programs. The median age was 6 years. The median number of patients per resident was 660 with an interquartile range of 336. The most common PECARN diagnostic categories were trauma, gastrointestinal, and respiratory disease. Thirty-two core MCP diagnoses (43% of MCP list) were not seen by at least 50% of the residents. We found statistically significant variability between programs in both PECARN diagnostic categories (p < 0.01) and MCP diagnoses (p < 0.01).

CONCLUSION

There is considerable variation in the number of pediatric patients and the diagnostic case volume seen by EM residents. The relationship between this case variability and competence upon graduation is unknown; further investigation is warranted to better inform program-specific curricula and to guide training requirements in EM.