Have First-Year Emergency Medicine Residents Achieved Level 1 on Care-Based Milestones?

Early Emergency Medicine Milestone Assessment for Predicting First-Year Resident Performance

Introduction

The Accreditation Council for Graduate Medical Education (ACGME) requires emergency medicine (EM) residency training programs to monitor residents' progress using standardized milestones. The first assessment of PGY 1 resident milestones occurs midway through the first year and could miss initial deficiencies. Early assessment of PGY 1 EM resident milestones has potential to identify at-risk residents prior to standard midyear evaluations. We developed an orientation syllabus for PGY 1 residents followed by a milestone assessment. Assessment scores helped predict future milestone scores and American Board of Emergency Medicine (ABEM) In-Training Examination (ITE) scores for PGY 1 residents.

Methods

From 2013 to 2020, we developed and implemented Milestone Evaluation Day (MED), a simulation-based day and written exam assessing PGY 1 EM residents during their first month on the 23 ACGME 1.0 milestones. MED stations included a history and physical with verbal presentation, patient simulation, vascular access, wound management, and airway management. MED, Clinical Competency Committee-generated (CCC-generated) milestone, and ABEM ITE scores were averaged and compared utilizing Pearson's correlation coefficient.

Results

Of 112 PGY 1 EM residents, 110 (98%) were analyzed over an 8-year period. We observed a moderate positive correlation of MED and CCC-generated milestone scores (r = .34, p < .001). There was a nonstatistically significant weak positive correlation of MED and ABEM ITE scores (r = .13, p = .17).

Discussion

An early assessment of EM milestones in the PGY 1 year can assist in the prediction of CCC-generated milestone scores for PGY 1 residents.

Only as Strong as the Weakest Link: Resident Perspectives on Entrustable Professional Activities and Their Impact on Learning

Background Core to competency-based medical education (CBME) is the use of frequent low-stakes workplace-based assessments. In the Canadian context, these observations of performance are framed around entrustable professional activities (EPAs). Objective We aimed to explore residents' real-world perspectives of EPAs and their perceived impact on learning, because assessments perceived to be "inauthentic," or not truly reflective of their lived experiences, may interfere with learning. Methods Using constructivist grounded theory, we conducted 18 semistructured interviews in 2021 with residents from all programs that had implemented CBME at one tertiary care academic center in Canada. Participants were recruited via email through respective program administrators. Data collection and analysis occurred iteratively, and categories were identified using constant comparative analysis. Results Residents were strikingly polarized, perceiving EPAs as either a valuable opportunity for professional growth or as an onerous requirement that interfered with learning. Regardless of what view participants held, all perspectives were informed by: (1) the program administration and the perceived messaging from program to residents; (2) faculty assessors and their perceived degree of engagement, or "buy-in" with the EPA system; and ultimately (3) learner behavior. We theorized from these findings that all 3 aspects must be working in tandem for the assessment system to function as intended. Conclusions From the learners' perspective, there exists a dynamic, interdependent relationship between the 3 CBME stakeholders. As such, the perceived value of the EPA assessment system can only be as strong as the weakest link in the chain.

Are First-Year Emergency Medicine Residents Still Behind on Level 1 Care-Based Milestones?

Background The Accreditation Council for Graduate Medical Education defines Level 1 as "the resident demonstrates milestones expected of an incoming resident," yet a previous study of emergency medicine (EM) interns showed most were not meeting Level 1 milestones. In addition, previous research indicates that residents often provide more favorable self-assessments when compared to faculty assessments. Our study, performed in July 2022, aims to determine whether incoming EM residents remain behind on Level 1 care-based milestones and if resident self-assessments are consistent with faculty assessments. Methodology This is an observational study involving five distinct EM residency programs. Incoming interns were directly assessed by faculty for behaviors associated with the care-based milestones for EM using a standardized survey. Interns were asked to complete this same survey regarding their own performance. Results Faculty completed a total of 101 assessments on 49 residents. Of the 49 residents, 39 completed self-evaluations (80%). Achievement of Level 1 ranged from 25% to 82%. Residents had significantly higher self-assessments than faculty assessments on PC-1, PC-5a, and PC-6a. Faculty assessments were significantly higher than resident self-assessments on PC-6b. Conclusions Greater than 75% of incoming interns were able to meet Level 1 milestones in three of seven care-based milestones. However, there is a generalized trend toward overall improvement when compared to previous studies. Residents continue to demonstrate higher self-assessments than faculty in three separate care-based milestones and faculty rated residents significantly higher in one care-based milestone. This is consistent with previous studies.

Post-Clerkship Curricular Reform: Specialty-Specific Tracks and Entrustable Professional Activities to Guide the Transition to Residency

The development of core entrustable professional activities (EPA) for entering residency and Accreditation Council for Graduate Medical Education's milestones have spurred thinking about the fourth year of medical school as a transition to residency. In this monograph, we lay out our specialty focused post-clerkship curriculum and report learner and residency director perceptions over the first three years of implementation. Ongoing curricular monitoring has reinforced core principles but has also informed actionable quality improvement efforts. EPA-focused learning experiences, integration of specialty-specific milestones, addition of the feedforward process, and accessible mentorships have been key curricular elements to guide the transition to residency.

Senior-Year Curriculum in U.S. Medical Schools: A Scoping Review

Phenomenon: Many U.S. medical schools have responded to the adoption of competency-based medical education (CBME) frameworks by renewing their final-year curricula and including internship preparatory courses. The purpose of this scoping review was to map the published literature regarding the final year to discern how medical schools have responded to this paradigm change. Approach: A structured 5-step approach was used to conduct this scoping review. Electronic searches of PubMed, ERIC, Scopus, MedEdPortal, and 8 medical education journals were conducted to identify relevant articles published from 2006 to 2016. Four authors screened articles for inclusion using standardized eligibility criteria; interrater agreement was discussed and calculated. Authors extracted data elements, and a consensus-based approach was used to categorize, sort, and structure information gathered. Findings: Among 6,485 articles retrieved, 817 articles were included in the study. From 2007-2011 to 2012-2016, articles addressing the final year of medical school increased 93%, whereas articles describing internship preparatory courses increased 218%. The majority of articles did not reference a CBME framework (572/817; 70%), the frequency of mentions increased 268% from 2007-2011 to 2012-2016. Nearly three fourths of preparatory course-related papers reference a CBME framework (37/50; 74%). Insights: Our findings may reflect a movement in U.S. medical schools toward using shared assessment metrics to support 4th-year medical students' preparation for their postgraduate medical education. Despite an increased reference to CBME, there is relatively little use of competency-based assessment frameworks to evaluate learning outcomes. This review also found a substantial increase in the frequency of articles referencing senior-year preparatory courses in U.S. medical schools. Articles tended to describe institution-specific research or experiences. Opportunities likely exist for multi-institutional and organizational collaboration in determining the structure of the final year of medical school.

Use of Emergency Department Pharmacists in Emergency Medicine Resident Milestone Assessment

Introduction

The use of competency-based milestones for emergency medicine (EM) was mandated by the Accreditation Council for Graduate Medical Education in 2013. However, clinical competency committees (CCC) may lack diverse, objective data to assess these new competencies. To remedy the lack of objective data when assessing the pharmacotherapy sub-competency (PC5) we introduced a unique approach that actively involves departmental clinical pharmacists in determining the milestone level achieved by the resident.

Methods

Our pharmacists assess the pharmacotherapy knowledge of the residents through multiple methods: direct observation of orders, communication with the residents while performing patient care within the emergency department (ED), and real-time chart review. This observation occurs informally on a daily basis in the ED and is incorporated into the routine work of the pharmacist. The pharmacists use the PC5 sub-competency as their standard evaluation tool in this setting to keep all assessments consistent.

Results

Since our residency program introduced pharmacist assessment of resident pharmacotherapy knowledge, the CCC has conducted seven biannual meetings. Of the 120 separate PC5 sub-competency assessments made during those meetings there was 100% agreement between the pharmacist’s assessment and the CCC’s final assessment of the trainee. A survey of the CCC members concluded that the pharmacists’ assessments were useful and aided in accurate resident evaluation.

Conclusion

The use of ED pharmacists in assessing the pharmacotherapy sub-competency provides important information used in resident assessment of the PC5 milestone.

What to Do About the Transition to Residency? Exploring Problems and Solutions From Three Perspectives

Transitions are both a blessing and a curse for learning and professional development. While transitions can afford valuable opportunities for growth, they can also burden learners in ways that stymie performance and development. In this Invited Commentary, the author focuses on the transition from medical school to residency-a transition that many believe has tipped too far toward burden. The commentary explores three ways of problematizing the transition to residency. The first is as a transaction problem involving a complex exchange of information among programs and people. The second is as a transfer problem in which learners must apply knowledge in new contexts. The third is a trajectory problem concerning each learner's progression through medical education. The author describes current efforts aligned with each framing of the problem and discusses potential gaps in these efforts that might be addressed through an overarching framing of transitions in medical education as transformative processes. This framing may help the medical educational community think more holistically about the problems and solutions with transitions.

Clinical Assessment of Medical Students in Emergency Medicine Clerkships: A Survey of Current Practice

BACKGROUND

Assessment practices in emergency medicine (EM) clerkships have not been previously described. Clinical assessment frequently relies on global ratings of clinical performance, or "shift cards," although these tools have not been standardized or studied.

OBJECTIVE

We sought to characterize assessment practices in EM clerkships, with particular attention to shift cards.

METHODS

A survey regarding assessment practices was administered to a national sample of EM clerkship directors (CDs). Descriptive statistics were compiled and regression analyses were performed.

RESULTS

One hundred seventy-two CDs were contacted, and 100 (58%) agreed to participate. The most heavily weighted assessment methods in final grades were shift cards (66%) and written examinations (21-26%), but there was considerable variability in grading algorithms. EM faculty (100%) and senior residents (69%) were most commonly responsible for assessment, and assessors were often preassigned (71%). Forty-four percent of CDs reported immediate completion of shift cards, 27% within 1 to 2 days, and 20% within a week. Only 40% reported return rates >75%. Thirty percent of CDs do not permit students to review individual evaluations, and 54% of the remainder deidentify evaluations before student review. Eighty-six percent had never performed psychometric analysis on their assessment tools. Sixty-five percent of CDs were satisfied with their shift cards, but 90% supported the development of a national tool.

CONCLUSION

There is substantial variability in assessment practices between EM clerkships, raising concern regarding the comparability of grades between institutions. CDs rely on shift cards in grading despite the lack of evidence of validity and inconsistent process variables. Standardization of assessment practices may improve the assessment of EM students.