Programmatic assessment of level 1 milestones in incoming interns

Early Emergency Medicine Milestone Assessment for Predicting First-Year Resident Performance

Introduction

The Accreditation Council for Graduate Medical Education (ACGME) requires emergency medicine (EM) residency training programs to monitor residents' progress using standardized milestones. The first assessment of PGY 1 resident milestones occurs midway through the first year and could miss initial deficiencies. Early assessment of PGY 1 EM resident milestones has potential to identify at-risk residents prior to standard midyear evaluations. We developed an orientation syllabus for PGY 1 residents followed by a milestone assessment. Assessment scores helped predict future milestone scores and American Board of Emergency Medicine (ABEM) In-Training Examination (ITE) scores for PGY 1 residents.

Methods

From 2013 to 2020, we developed and implemented Milestone Evaluation Day (MED), a simulation-based day and written exam assessing PGY 1 EM residents during their first month on the 23 ACGME 1.0 milestones. MED stations included a history and physical with verbal presentation, patient simulation, vascular access, wound management, and airway management. MED, Clinical Competency Committee-generated (CCC-generated) milestone, and ABEM ITE scores were averaged and compared utilizing Pearson's correlation coefficient.

Results

Of 112 PGY 1 EM residents, 110 (98%) were analyzed over an 8-year period. We observed a moderate positive correlation of MED and CCC-generated milestone scores (r = .34, p < .001). There was a nonstatistically significant weak positive correlation of MED and ABEM ITE scores (r = .13, p = .17).

Discussion

An early assessment of EM milestones in the PGY 1 year can assist in the prediction of CCC-generated milestone scores for PGY 1 residents.

Sailing the boat together: Co-creation of a model for learning during transition

PURPOSE

Medical students' transition to postgraduate training, given the complexity of new roles and responsibilities, requires the engagement of all involved stakeholders. This study aims to co-create a transition curriculum and determine the value of involving the key stakeholders throughout such transition in its design process.

METHODS

We conducted a mixed-methods study involving faculty/leaders (undergraduate/postgraduate), final-year medical students, and chief residents. It commenced with eight co-creation sessions (CCS), qualitative results of which were used to draft a quantitative survey sent to non-participants, followed by two consensus-building CCS with the original participants. We applied thematic analysis for transcripts of all CCS, and mean scores with standard deviations for survey analysis.

RESULTS

We identified five themes: adaptation, authenticity, autonomy, connectedness, and continuity, embedded in the foundation of a supportive environment, to constitute a Model of Learning during Transition (MOLT). Inclusion of various stakeholders and optimizing their representation brought rich perspectives to the design process. This was reinforced through active students' participation enabling a final consensus.

CONCLUSIONS

Bringing perspectives of key stakeholders in the transition spectrum enriches transition curricula. The proposed MOLT can provide a guide for curriculum designers to optimize the final year of undergraduate medical training in preparing students for postgraduate training with essential competencies to be trained.

Competency-based medical education across the continuum: How well aligned are medical school EPAs to residency milestones?

INTRODUCTION

Competency-based medical education (CBME) provides a framework for describing learner progression throughout training. However, specific approaches to CBME implementation vary widely across educational settings. Alignment between various methods used across the continuum is critical to support transitions and assess learner performance. The purpose of this study was to investigate alignment between CBME frameworks used in undergraduate medical education (UME) and graduate medical education (GME) settings using the US context as a model.

METHOD

The authors analyzed content from the core entrustable professional activities for entering residency (Core EPAs; UME model) and residency milestones (GME model). From that analysis, they performed a series of cross-walk activities to investigate alignment between frameworks. After independent review, authors discussed findings until consensus was reached.

RESULTS

Some alignment was found for activities associated with history taking, physical examination, differential diagnosis, patient safety, and interprofessional care; however, there were far more examples of misalignment.

CONCLUSIONS

These findings highlight challenges creating alignment of assessment frameworks across the continuum of training. The importance of these findings includes implications for assessment and persistence of the educational gap across UME and GME. The authors provide four next steps to improve upon the continuum of education.

Implementation of a pilot novel objective peer comparison evaluation system in an emergency medicine residency program

OBJECTIVES

Emergency medicine (EM) residents are currently evaluated via The Milestones, which have been shown to be imperfect and subjective. There is also a need for residents to achieve competency in patient safety and quality improvement processes, which can be accomplished through provision of peer comparison metrics. This pilot study aimed to evaluate the implementation of an objective peer comparison system for metrics that quantified aspects of quality and safety, efficiency and throughput, and utilization.

METHODS

This pilot study took place at an academic, tertiary care center with a 3-year residency and 14 residents per postgraduate year (PGY) class. Metrics were compared within each PGY class using Wilcoxon signed-rank and rank-order analyses.

RESULTS

Significant changes were seen in the majority of the metrics for all PGY classes. PGY3s accounted for the significant change in EKG and X-ray reads, while PGY1s and PGY2s accounted for the significant change in disposition to final note share. Physician evaluation to disposition decision was the only metric that did not reach significance in any class.

CONCLUSIONS

These preliminary data suggest that providing objective metrics is possible. Peer comparison metrics could provide an effective objective addition to the milestone evaluation system currently in use.

The Emergency Medicine Milestones 2.0: Setting the stage for 2025 and beyond

INTRODUCTION

Beginning in 1999, residents in emergency medicine have been expected to demonstrate competence in the six Accreditation Council on Graduate Medical Education (ACGME) Core Competencies. Expectations were further refined and clarified through the introduction of the Milestones in 2013. Emerging research and data from milestone reporting has illustrated the need for modification of the original milestones. Against this backdrop, the ACGME convened a committee to review and revise the original milestones.

METHODS

The working group was convened in December 2018 and consisted of representatives from the American Board of Emergency Medicine, American Osteopathic Association, Council of Residency Directors in Emergency Medicine, Association of American Medical Colleges, ACGME-Emergency Medicine Review Committee, three community members, a resident member, and a public member. This group also included members from both academic and community emergency medicine programs. The group was overseen by the ACGME vice president for milestones development and met in person one time followed by four virtual sessions to revise and draft the Emergency Medicine Milestones and Supplemental Guide as part of the ACGME Milestones 2.0 Project.

RESULTS

Using data from milestones reporting, needs assessment data, stakeholder interviews, and community commentary, the working group engaged in revisions and updates for the Emergency Medicine Milestones and created a supplemental guide to aid programs in the design of programmatic assessment for the milestones.

CONCLUSION

The Emergency Medicine Milestones 2.0 provide updated specialty-specific, competency-based behavioral anchors to guide the assessment of residents, the design of curricula, and the advancement of emergency medicine training programs.

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

OBJECTIVES

Direct observation is important for assessing the competency of medical learners. Multiple tools have been described in other fields, although the degree of emergency medicine-specific literature is unclear. This review sought to summarize the current literature on direct observation tools in the emergency department (ED) setting.

METHODS

We searched PubMed, Scopus, CINAHL, the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, ERIC, PsycINFO, and Google Scholar from 2012 to 2020 for publications on direct observation tools in the ED setting. Data were dual extracted into a predefined worksheet, and quality analysis was performed using the Medical Education Research Study Quality Instrument.

RESULTS

We identified 38 publications, comprising 2,977 learners. Fifteen different tools were described. The most commonly assessed tools included the Milestones (nine studies), Observed Structured Clinical Exercises (seven studies), the McMaster Modular Assessment Program (six studies), Queen's Simulation Assessment Test (five studies), and the mini-Clinical Evaluation Exercise (four studies). Most of the studies were performed in a single institution, and there were limited validity or reliability assessments reported.

CONCLUSIONS

The number of publications on direct observation tools for the ED setting has markedly increased. However, there remains a need for stronger internal and external validity data.

Where the rubber meets the road - An integrative review of programmatic assessment in health care professions education

INTRODUCTION

Programmatic assessment was introduced as an approach to design assessment programmes with the aim to simultaneously optimize the decision-making and learning function of assessment. An integrative review was conducted to review and synthesize results from studies investigating programmatic assessment in health care professions education in practice.

METHODS

The authors systematically searched PubMed, Web of Science, and ERIC to identify studies published since 2005 that reported empirical data on programmatic assessment. Characteristics of the included studies were extracted and synthesized, using descriptive statistics and thematic analysis.

RESULTS

Twenty-seven studies were included, which used quantitative methods (n = 10), qualitative methods (n = 12) or mixed methods (n = 5). Most studies were conducted in clinical settings (77.8%). Programmatic assessment was found to enable meaningful triangulation for robust decision-making and used as a catalyst for learning. However, several problems were identified, including overload in assessment information and the associated workload, counterproductive impact of using strict requirements and summative signals, lack of a shared understanding of the nature and purpose of programmatic assessment, and lack of supportive interpersonal relationships. Thematic analysis revealed that the success and challenges of programmatic assessment were best understood by the interplay between quantity and quality of assessment information, and the influence of social and personal aspects on assessment perceptions.

CONCLUSION

Although some of the evidence may seem compelling to support the effectiveness of programmatic assessment in practice, tensions will emerge when simultaneously stimulating the development of competencies and assessing its result. The identified factors and inferred strategies provide guidance for navigating these tensions.

A Standardized Patient Experience: Elevating Interns to Expected Level of Clinical Competency

Introduction

Medical students transition to intern year with significant variability in prior clinical experience depending on their medical school education. This leads to notable differences in the interns’ ability to perform focused histories and physical exams, develop reasoned differentials, and maximize care plans. Providing a foundational experience for these essential skills will help to establish standardized expectations despite variable medical school experiences.

Methods

During an orientation block, interns participated in a standardized patient experience. Interns were presented with three common chief complaints: abdominal pain; chest pain; and headache. Faculty observed the three patient encounters and provided immediate verbal and written feedback to the interns based on a standardized grading rubric.

Results

All residents that participated “agreed” or “strongly agreed” that the experience was a meaningful educational experience. 90% of the interns reported the experience would change their clinical practice. Additionally, 75% of residents survyed one year after the experience felt the experience changed their clinical practice. Faculty felt the learning experience allowed them to address knowledge gaps early and provide early guidance where needed.

Conclusion

This article describes an emergency medicine residency program’s effort to provide a foundational experience for interns in evaluating emergency department patients. The intent was to “level the playing field” and establish “good habits” early in intern year with the realization that prior experiences vary significantly in July of intern year.

Videotaped Unannounced Standardized Patient Encounters to Evaluate Interpersonal and Communication Skills in Emergency Medicine Residents

BACKGROUND

Communication and interpersonal skills are important for effective patient care but are difficult to measure. Unannounced standardized patient encounters (USPEs) have the benefit of providing a standardized situation and provide a representation of usual care rather than best behavior, while also allowing for video recording without violating patient privacy. We conducted a feasibility pilot study to examine the use of videotaped USPEs in resident education of interpersonal and communication skills and specifically empathy.

METHODS

This was a pilot study conducted at an urban community teaching hospital. About 16 first-year emergency medicine engaged in four videotaped USPEs during their normal clinical shifts during 1 academic year. All visible recording equipment was concealed from the residents. The standardized patients completed two assessment forms after each encounter to measure empathy.

RESULTS

All 16 residents engaged in four videotaped USPEs for a total of 64 encounters.

CONCLUSIONS

We were able to successfully demonstrate the feasibility of using USPEs for assessment of resident interpersonal and communication skills.

Trainees' Perceptions of the Transition From Medical School to Residency

BACKGROUND

There is emerging evidence that learners may be suboptimally prepared for the expectations of residency. In order to address these concerns, many medical schools are implementing residency preparation courses (RPCs).

OBJECTIVE

We aimed to determine trainees' perceptions of their transition to residency and whether they felt that they benefited from participation in an RPC.

METHODS

All residents and fellows at the University of Michigan (n = 1292) received an electronic survey in July 2018 that queried respondents on demographics, whether medical school had prepared them for intern year, and whether they had participated in an RPC.

RESULTS

The response rate was 44% (563 of 1292) with even distribution across gender and postgraduate years (PGYs). Most (78%, 439 of 563) felt that medical school prepared them well for intern year. There were no differences in reported preparedness for intern year across PGY, age, gender, or specialty. Overall, 28% (156 of 563) of respondents participated in an RPC and endorsed feeling prepared for intern year, which was more than RPC non-participants (85% [133 of 156] vs 70% [306 of 439], P = .029). Participation in longer RPCs was also associated with higher perceived preparedness for residency.

CONCLUSIONS

This study found that residents from multiple specialties reported greater preparedness for residency if they participated in a medical school fourth-year RPC, with greater perceptions of preparedness for longer duration RPCs, which may help to bridge the medical school to residency gap.

A Responsible Educational Handover: Improving Communication to Improve Learning

An important tenet of competency-based medical education is that the educational continuum should be seamless. The transition from undergraduate medical education (UME) to graduate medical education (GME) is far from seamless, however. Current practices around this transition drive students to focus on appearing to be competitively prepared for residency. A communication at the completion of UME-an educational handover-would encourage students to focus on actually preparing for the care of patients. In April 2018, the American Medical Association's Accelerating Change in Medical Education consortium meeting included a debate and discussion on providing learner performance measures as part of a responsible educational handover from UME to GME. In this Perspective, the authors describe the resulting 5 recommendations for developing such a handover: (1) The purpose of the educational handover should be to provide medical school performance data to guide continued improvement in learner ability and performance, (2) the process used to create an educational handover should be philosophically and practically aligned with the learner's continuous quality improvement, (3) the educational handover should be learner driven with a focus on individualized learning plans that are coproduced by the learner and a coach or advisor, (4) the transfer of information within an educational handover should be done in a standardized format, and (5) together, medical schools and residency programs must invest in adequate infrastructure to support learner improvement. These recommendations are shared to encourage implementation of the educational handover and to generate a potential research agenda that can inform policy and best practices.

Comparison of Male and Female Resident Milestone Assessments During Emergency Medicine Residency Training: A National Study

PURPOSE

A previous study found that milestone ratings at the end of training were higher for male than for female residents in emergency medicine (EM). However, that study was restricted to a sample of 8 EM residency programs and used individual faculty ratings from milestone reporting forms that were designed for use by the program's Clinical Competency Committee (CCC). The objective of this study was to investigate whether similar results would be found when examining the entire national cohort of EM milestone ratings reported by programs after CCC consensus review.

METHOD

This study examined longitudinal milestone ratings for all EM residents (n = 1,363; 125 programs) reported to the Accreditation Council for Graduate Medical Education every 6 months from 2014 to 2017. A multilevel linear regression model was used to estimate differences in slope for all subcompetencies, and predicted marginal means between genders were compared at time of graduation.

RESULTS

There were small but statistically significant differences between males' and females' increase in ratings from initial rating to graduation on 6 of the 22 subcompetencies. Marginal mean comparisons at time of graduation demonstrated gender effects for 4 patient care subcompetencies. For these subcompetencies, males were rated as performing better than females; differences ranged from 0.048 to 0.074 milestone ratings.

CONCLUSIONS

In this national dataset of EM resident milestone assessments by CCCs, males and females were rated similarly at the end of their training for the majority of subcompetencies. Statistically significant but small absolute differences were noted in 4 patient care subcompetencies.

Assessment in the context of problem-based learning

Arguably, constructive alignment has been the major challenge for assessment in the context of problem-based learning (PBL). PBL focuses on promoting abilities such as clinical reasoning, team skills and metacognition. PBL also aims to foster self-directed learning and deep learning as opposed to rote learning. This has incentivized researchers in assessment to find possible solutions. Originally, these solutions were sought in developing the right instruments to measure these PBL-related skills. The search for these instruments has been accelerated by the emergence of competency-based education. With competency-based education assessment moved away from purely standardized testing, relying more heavily on professional judgment of complex skills. Valuable lessons have been learned that are directly relevant for assessment in PBL. Later, solutions were sought in the development of new assessment strategies, initially again with individual instruments such as progress testing, but later through a more holistic approach to the assessment program as a whole. Programmatic assessment is such an integral approach to assessment. It focuses on optimizing learning through assessment, while at the same gathering rich information that can be used for rigorous decision-making about learner progression. Programmatic assessment comes very close to achieving the desired constructive alignment with PBL, but its wide adoption-just like PBL-will take many years ahead of us.

Does Performance on Evidence-Based Medicine and Urgent Clinical Scenarios Assessments Deteriorate During the Fourth Year of Medical School? Findings From One Institution

PURPOSE

The fourth year of medical school (M4) should prepare students for residency yet remains generally unstructured, with ill-defined goals. The primary aim of this study was to determine whether there were performance changes in evidence-based medicine (EBM) and urgent clinical scenarios (UCS) assessments before and after M4 year.

METHOD

University of Michigan Medical School graduates who matched into internship at Michigan Medicine completed identical assessments on EBM and UCS at the beginning of M4 year and 13 months later during postgraduate year 1 (PGY1) orientation. Individual scores on these assessments were compared using paired t test analysis. The associations of academic performance, residency specialty classification, and initial performance on knowledge changes were analyzed.

RESULTS

During academic years 2014 and 2015, 76 students matched into a Michigan Medicine internship; 52 completed identical EBM stations and 53 completed UCS stations. Learners' performance on the EBM assessment decreased from M4 to PGY1 (mean 93% [SD = 7%] vs. mean 80% [SD = 13%], P < .01), while performance on UCS remained stable (mean 80% [SD = 9%] vs. mean 82% [SD = 8%], P = .22). High M4 performers experienced a greater rate of decline in knowledge level compared with low M4 performers for EBM (-20% vs. -4%, P = .01). Residency specialty and academic performance did not affect performance.

CONCLUSIONS

This study demonstrated degradation of performance in EBM during the fourth year and adds to the growing literature that highlights the need for curricular reform during this year.

What to Do About the Transition to Residency? Exploring Problems and Solutions From Three Perspectives

Transitions are both a blessing and a curse for learning and professional development. While transitions can afford valuable opportunities for growth, they can also burden learners in ways that stymie performance and development. In this Invited Commentary, the author focuses on the transition from medical school to residency-a transition that many believe has tipped too far toward burden. The commentary explores three ways of problematizing the transition to residency. The first is as a transaction problem involving a complex exchange of information among programs and people. The second is as a transfer problem in which learners must apply knowledge in new contexts. The third is a trajectory problem concerning each learner's progression through medical education. The author describes current efforts aligned with each framing of the problem and discusses potential gaps in these efforts that might be addressed through an overarching framing of transitions in medical education as transformative processes. This framing may help the medical educational community think more holistically about the problems and solutions with transitions.

Ethics of Virtual Reality in Medical Education and Licensure

The aim of this section is to expand and accelerate advances in curriculum developments and in methods of teaching bioethics.

Programmatic Assessment in Emergency Medicine: Implementation of Best Practices

BACKGROUND

Programmatic assessment is the intentional collection of key data from multiple sources for both assessment of learning and assessment for learning.

OBJECTIVE

We developed a system of programmatic assessment (PA) to identify competency progression (summative) and assessment for learning to assist residents in their formative development.

METHODS

The programmatic assessment was designed iteratively from 2014 through 2016. All assessments were first categorized by competency domain and source of assessment. The number of assessment modalities for each competency domain was collected. These multisource assessments were then mapped by program leadership to the milestones to develop a master PA blueprint. A resident learning management system provided the platform for aggregating formative and summative data, allowing residents and faculty ongoing access to guide learning and assessment. A key component of programmatic assessment was to support resident integration of assessment information through feedback by faculty after shifts and during monthly formal assessments, semiannual resident reviews, and summative judgments by the Clinical Competency Committee.

RESULTS

Through the PA, the 6 competency domains are assessed through multiple modalities: patient care (22 different assessments), professionalism (18), systems-based practice (17), interprofessional and communication skills (16), medical knowledge (11), and practice-based learning and improvement (6). Each assessment provides feedback to the resident in various formats. Our programmatic assessment has been utilized for more than 2 years with iterative improvements.

CONCLUSIONS

The implementation of programmatic assessment allowed our program to organize diverse, multisourced feedback to drive both formative and summative assessments.

Bringing the Flipped Classroom to Day 1: A Novel Didactic Curriculum for Emergency Medicine Intern Orientation

Most emergency medicine (EM) residency programs provide an orientation program for their incoming interns, with the lecture being the most common education activity during this period. Our orientation program is designed to bridge the gap between undergraduate and graduate medical education by ensuring that all learners demonstrate competency on Level 1 Milestones, including medical knowledge (MK). To teach interns core medical knowledge in EM, we reformulated orientation using the flipped-classroom model by replacing lectures with small group, case-based discussions. Interns demonstrated improvement in medical knowledge through higher scores on a posttest. Evaluation survey results were also favorable for the flipped-classroom teaching format.

Patient Handoff Education: Are Medical Schools Catching Up?

Communication errors during shift-to-shift handoffs are a leading cause of preventable adverse events. Nevertheless, handoff skills are variably taught at medical schools. The authors administered questionnaires on handoffs to interns during orientation. Questions focused on medical school handoff education, experiences, and perceptions. The majority (546/718) reported having some form of education on handoffs during medical school, with 48% indicating this was 1 hour or less. Most respondents (98%) reported that they believe patients experience adverse events because of inadequate handoffs, and more than one third had witnessed a patient safety issue. Results show that medical school graduates are not receiving adequate handoff training. Yet graduates are expected to conduct safe patient handoffs at the start of residency. Given that ineffective handoffs pose a significant patient safety risk, medical school graduates should have a baseline competency in handoff skills. This will require medical schools to develop, implement, and study handoff education.

Critical Deficiency Ratings in Milestone Assessment: A Review and Case Study

PURPOSE

The Accreditation Council for Graduate Medical Education (ACGME) requires programs to report learner progress using specialty-specific milestones. It is unclear how milestones can best identify critical deficiencies (CDs) in trainee performance. Specialties developed milestones independently of one another; not every specialty included CDs within milestones ratings. This study examined the proportion of ACGME milestone sets that include CD ratings, and describes one residency program's experiences using CD ratings in assessment.

METHOD

The authors reviewed ACGME milestones for all 99 specialties in November 2015, determining which rating scales contained CDs. The authors also reviewed three years of data (July 2012-June 2015) from the University of Cincinnati Medical Center (UCMC) internal medicine residency assessment system based on observable practice activities mapped to ACGME milestones. Data were analyzed by postgraduate year, assessor type, rotation, academic year, and core competency. The Mantel-Haenszel chi-square test was used to test for changes over time.

RESULTS

Specialties demonstrated heterogeneity in accounting for CDs in ACGME milestones, with 22% (22/99) of specialties having no language describing CDs in milestones assessment. Thirty-three percent (63/189) of UCMC internal medicine residents received at least one CD rating, with CDs accounting for 0.18% (668/364,728) of all assessment ratings. The authors identified CDs across multiple core competencies and rotations.

CONCLUSIONS

Despite some specialties not accounting for CDs in milestone assessment, UCMC's experience demonstrates that a significant proportion of residents may be rated as having a CD during training. Identification of CDs may allow programs to develop remediation and improvement plans.

How Effective are New Milestones Assessments at Demonstrating Resident Growth? 1 Year of Data

OBJECTIVE

Assessment tools that accrue data for the Accreditation Council for Graduate Medical Education Milestones must evaluate residents across multiple dimensions, including medical knowledge, procedural skills, teaching, and professionalism. Our objectives were to: (1) develop an assessment tool to evaluate resident performance in accordance with the Milestones and (2) review trends in resident achievements during the inaugural year of Milestone implementation.

DESIGN

A novel venue and postgraduate year (PGY) specific assessment tool was built, tested, and implemented for both operating room and labor and delivery "venues." Resident development of competence and independence was captured over time. To account for variable rotation schedules, the year was divided into thirds and compared using two-tailed Fisher's exact test.

SETTING

Brigham and Women's and Massachusetts General Hospitals, Boston MA.

PARTICIPANTS

Faculty evaluators and obstetrics and gynecology residents.

RESULTS

A total of 822 assessments of 44 residents were completed between 9/2014 and 6/2015. The percentage of labor and delivery tasks completed "independently" increased monotonically across the start of all years: 8.4% for PGY-1, 60.3% for PGY-2, 73.7% for PGY-3, and 87.5% for PGY-4. Assessments of PGY-1 residents demonstrated a significant shift toward "with minimal supervision" and "independent" for the management of normal labor (p = 0.03). PGY-3 residents demonstrated an increase in "able to be primary surgeon" in the operating room, from 36% of the time in the first 2/3 of the year, to 62.3% in the last 1/3 (p < 0.01).

CONCLUSION

Assessment tools developed to assist with Milestone assignments capture the growth of residents over time and demonstrate quantifiable differences in achievements between PGY classes. These tools will allow for targeted teaching opportunities for both individual residents and residency programs.

Emergency Medicine Resident Orientation: How Training Programs Get Their Residents Started

Introduction

The first formal orientation program for incoming emergency medicine (EM) residents was started in 1976. The last attempt to describe the nature of orientation programs was by Brillman in 1995. Now almost all residencies offer orientation to incoming residents, but little is known about the curricular content or structure of these programs. The purpose of this project was to describe the current composition and purpose of EM resident orientation programs in the United States.

Methods

In autumn of 2014, we surveyed all U.S. EM residency program directors (n=167). We adapted our survey instrument from one used by Brillman (1995). The survey was designed to assess the orientation program’s purpose, structure, content, and teaching methods.

Results

The survey return rate was 63% (105 of 167). Most respondents (77%) directed three-year residencies, and all but one program offered intern orientation. Orientations lasted an average of nine clinical (Std. Dev.=7.3) and 13 non-clinical days (Std. Dev.=9.3). The prototypical breakdown of program activities was 27% lectures, 23% clinical work, 16% skills training, 10% administrative activities, 9% socialization and 15% other activities. Most orientations included activities to promote socialization among interns (98%) and with other members of the department (91%). Many programs (87%) included special certification courses (ACLS, ATLS, PALS, NRP). Course content included the following: use of electronic medical records (90%), physician wellness (75%), and chief complaint-based lectures (72%). Procedural skill sessions covered ultrasound (94%), airway management (91%), vascular access (90%), wound management (77%), splinting (67%), and trauma skills (62%).

Conclusion

Compared to Brillman (1995), we found that more programs (99%) are offering formal orientation and allocating more time to them. Lectures remain the most common educational activity. We found increases in the use of skills labs and specialty certifications. We also observed increases in time dedicated to clinical work during orientation. Only a few programs reported engaging in baseline or milestone assessments, an activity that could offer significant benefits to the residency program.

A Pilot Study to Gauge Effectiveness of Standardized Patient Scenarios in Assessing General Surgery Milestones

PURPOSE

Some General Surgery Milestones can be difficult to assess in traditional clinical settings and especially difficult to assess in junior residents. The purpose of this pilot study was to METHODS: A total of 9 categorical interns participated in a comprehensive, 4-module, SP scenario designed to evaluate and manage right upper quadrant pain. SP checklist scores (SP%) were converted to Milestone-equivalent scores for direct comparison (SP-C). Milestone scores were analyzed from 3 different sources: SP, faculty (FAC), and CCC. Interns completed course evaluations at the end of each session. Spearman's rho was used to determine correlations. Wilcoxon signed rank tests were used to test for differences between scores from different sources.

RESULTS

Individual intern Milestone scores from the 3 sources (SP-C, FAC, and CCC) did not correlate. All 7 mean Milestone scores from SPs were significantly higher than from FAC and CCC. FAC and CCC scores were statistically equivalent except for Systems-Based Practice 1 (SBP1) and Patient Care 3 (PC3) where CCC scores were significantly higher than FAC. Mean SP% scores for PC1 were significantly lower than for PROF1, MK1, MK2, and ICS1 (p < 0.05). Interns felt the modules were moderately to very useful.

CONCLUSIONS

Developing an SP scenario for Milestones evaluation is feasible. SPs, faculty observers, and CCC each use different data to provide a unique source of Milestone assessment. SP scenarios may be ideally suited to assess specific resident strengths and weaknesses and provide individualized feedback, thus augmenting traditional evaluations. Additional SP scenarios, assessing a broader range of skills and Milestones, are advisable for more reliable estimates of resident performance.

Clinical Assessment of Medical Students in Emergency Medicine Clerkships: A Survey of Current Practice

BACKGROUND

Assessment practices in emergency medicine (EM) clerkships have not been previously described. Clinical assessment frequently relies on global ratings of clinical performance, or "shift cards," although these tools have not been standardized or studied.

OBJECTIVE

We sought to characterize assessment practices in EM clerkships, with particular attention to shift cards.

METHODS

A survey regarding assessment practices was administered to a national sample of EM clerkship directors (CDs). Descriptive statistics were compiled and regression analyses were performed.

RESULTS

One hundred seventy-two CDs were contacted, and 100 (58%) agreed to participate. The most heavily weighted assessment methods in final grades were shift cards (66%) and written examinations (21-26%), but there was considerable variability in grading algorithms. EM faculty (100%) and senior residents (69%) were most commonly responsible for assessment, and assessors were often preassigned (71%). Forty-four percent of CDs reported immediate completion of shift cards, 27% within 1 to 2 days, and 20% within a week. Only 40% reported return rates >75%. Thirty percent of CDs do not permit students to review individual evaluations, and 54% of the remainder deidentify evaluations before student review. Eighty-six percent had never performed psychometric analysis on their assessment tools. Sixty-five percent of CDs were satisfied with their shift cards, but 90% supported the development of a national tool.

CONCLUSION

There is substantial variability in assessment practices between EM clerkships, raising concern regarding the comparability of grades between institutions. CDs rely on shift cards in grading despite the lack of evidence of validity and inconsistent process variables. Standardization of assessment practices may improve the assessment of EM students.

Using Milestones as Evaluation Metrics During an Emergency Medicine Clerkship

BACKGROUND

The Accreditation Council for Graduate Medical Education's (ACGME) Milestones presumes graduating medical students will enter residency proficient at Milestone level 1 for 23 skills. The Next Accreditation System now includes Milestones for each postgraduate specialty, and it is unlikely that schools will document every emergency medicine (EM) applicant's EM-specific skills in their performance evaluation.

OBJECTIVES

The goals of this research were to determine if assessment of the Milestones was feasible during a medical student clerkship and examine the proportion of medical students performing at Milestone level 1.

METHODS

This study was conducted at a center with Liaison Committee on Medical Education-approved medical training and a 4-year EM residency. Using traditional clerkship, we studied the feasibility of an ACGME EM Milestones-based clerkship assessment. Data led to redesign of the clerkship and its evaluation process, including all level 1 anchor(s) to add "occasionally" (>60%), "usually" (>80%), and "always" (100%) on a Likert scale to on-shift assessment forms.

RESULTS

During the feasibility phase (2013-14), 75 students rotated though the clerkship; 55 evaluations were issued and 50 contained the Milestone summary. Eight deficiencies were noted in Milestone 12 and three in Milestone 14. After changes, 49 students rotated under the new evaluation rubric. Of 575 completed on-shift evaluations, 16 Milestone deficiencies were noted. Of 41 institutional evaluations issued, only one student had deficiencies noted, all of which pertained to patient care. All evaluations in this second cohort contained each student's Milestone proficiency.

CONCLUSIONS

Assessment of the Milestones is feasible. Communication of ACGME EM Milestone proficiency may identify students who require early observation or remediation. The majority of students meet the anchors for the Milestones, suggesting that clerkship assessment with the ACGME EM Milestones does not adequately differentiate students.

Mapping Direct Observations From Objective Structured Clinical Examinations to the Milestones Across Specialties

BACKGROUND

Little is known about residents' performance on the milestones at the institutional level. Our institution formed a work group to explore this using an institutional-level curriculum and residents' evaluation of the milestones.

OBJECTIVE

We assessed whether beginner-level milestones for interpersonal and communication skills (ICS) related to observable behaviors in ICS-focused objective structured clinical examinations (OSCEs) for postgraduate year (PGY) 1 residents across specialties.

METHODS

The work group compared ICS subcompetencies across 12 programs to identify common beginner-level physician-patient communication milestones. The selected ICS milestone sets were compared for common language with the ICS-OSCE assessment tool-the Kalamazoo Essential Elements of Communication Checklist-Adapted (KEECC-A). To assess whether OSCE scores related to ICS milestone scores, all PGY-1 residents from programs that were part of Next Accreditation System Phase 1 were identified; their OSCE scores from July 2013 to June 2014 and ICS subcompetency scores from December 2014 were compared.

RESULTS

The milestones for 10 specialties and the transitional year had at least 1 ICS subcompetency that related to physician-patient communication. The language of the ICS beginner-level milestones appears similar to behaviors outlined in the KEECC-A. All 60 residents with complete data received at least a beginner-level ICS subcompetency score and at least a satisfactory score on all 3 OSCEs.

CONCLUSIONS

The ICS-OSCE scores for PGY-1 residents appear to relate to beginner-level milestones for physician-patient communication across multiple specialties.

A pilot study of orthopaedic resident self-assessment using a milestones' survey just prior to milestones implementation

OBJECTIVE

To pilot test if Orthopaedic Surgery residents could self-assess their performance using newly created milestones, as defined by the Accreditation Council on Graduate Medical Education.

METHODS

In June 2012, an email was sent to Program Directors and administrative coordinators of the 154 accredited Orthopaedic Surgery Programs, asking them to send their residents a link to an online survey. The survey was adapted from the Orthopaedic Surgery Milestone Project. Completed surveys were aggregated in an anonymous, confidential database. SAS 9.3 was used to perform the analyses.

RESULTS

Responses from 71 residents were analyzed. First and second year residents indicated through self-assessment that they had substantially achieved Level 1 and Level 2 milestones. Third year residents reported they had substantially achieved 30/41, and fourth year residents, all Level 3 milestones. Fifth year, graduating residents, reported they had substantially achieved 17 Level 4 milestones, and were extremely close on another 15. No milestone was rated at Level 5, the maximum possible. Earlier in training, Patient Care and Medical Knowledge milestones were rated lower than the milestones reflecting the other four competencies of Practice Based Learning and Improvement, Systems Based Practice, Professionalism, and Interpersonal Communication. The gap was closed by the fourth year.

CONCLUSIONS

Residents were able to successfully self-assess using the 41 Orthopaedic Surgery milestones. Respondents' rate improved proficiency over time. Graduating residents report they have substantially, or close to substantially, achieved all Level 4 milestones. Milestone self-assessment may be a useful tool as one component of a program's overall performance assessment strategy.

Critical Appraisal of Emergency Medicine Education Research: The Best Publications of 2014

OBJECTIVES

The objective was to critically appraise and highlight rigorous education research study articles published in 2014 whose outcomes advance the science of emergency medicine (EM) education.

METHODS

A search of the English language literature in 2014 querying Education Resources Information Center (ERIC), PsychINFO, PubMed, and Scopus identified 243 EM-related articles using either quantitative (hypothesis-testing or observational investigations of educational interventions) or qualitative (exploring important phenomena in EM education) methods. Two reviewers independently screened all of the publications using previously established exclusion criteria. Six reviewers then independently scored the 25 selected publications using either a qualitative or a quantitative scoring system. Each scoring system consisted of nine criteria. Selected criteria were based on accepted educational review literature and chosen a priori. Both scoring systems use parallel scoring metrics and have been used previously within this annual review.

RESULTS

Twenty-five medical education research papers (22 quantitative, three qualitative) met the criteria for inclusion and were reviewed. Five quantitative and two qualitative studies were ranked most highly by the reviewers as exemplary and are summarized in this article.

CONCLUSIONS

This annual critical appraisal series highlights seven excellent EM education research studies, meeting a priori criteria and published in 2014. Methodologic strengths in the 2014 papers are noted, and current trends in medical education research in EM are discussed.

Use of Emergency Medicine Milestones as Items on End-of-Shift Evaluations Results in Overestimates of Residents' Proficiency Level

BACKGROUND

The emergency medicine milestones were developed to provide more objective resident assessment than current methods. However, little is known about the best practices for applying the milestones in resident assessment.

OBJECTIVE

We examined the utility of end-of-shift evaluations (ESEs) constructed using the milestones in resident assessment.

METHODS

We developed 14 daily ESEs, each of which included 9 or 10 emergency medicine milestones. Postgraduate year (PGY)-1 and PGY-2 residents were assessed on milestone levels 1 through 3; PGY-3 and PGY-4 residents were assessed on levels 3 through 5. Each milestone was rated on a nominal scale (yes, no, or not applicable). The Clinical Competency Committee combined the ESE data with data from other assessments to determine each resident's proficiency level for the emergency medicine subcompetencies. We used descriptive statistics to summarize resident ESEs and milestone levels. We analyzed differences in ESE score across PGY levels using t tests and analyses of variance.

RESULTS

Faculty completed 763 ESEs on 33 residents with a range of 2 to 54 (median=22) ESEs per resident. Faculty rarely (8%, 372 of 4633) rated a resident as not achieving a milestone on the ESEs. Analyses of variance revealed that ESE scores on level 3 milestones did not differ significantly by PGY level. There was poor agreement between ESE scores and Clinical Competency Committee ratings.

CONCLUSIONS

The ESEs constructed using the milestones resulted in grade or milestone inflation. Our results do not support using milestones as a stand-alone assessment tool.