The Milestones Passport: A Learner-Centered Application of the Milestone Framework to Prompt Real-Time Feedback in the Emergency Department

Implementation of an Un-Pairing Passport to Improve the Transition From Intern to Resident During a Critical Period of Anesthesiology Residency Training

Background

The transition from intern year to the first year of clinical anesthesiology residency (CA-1) is a challenging period for residents and their supervisors. Orientation methods and instructional material targeting this transition vary across U.S. residency programs. An un-pairing passport was implemented during the 2021-2022 transition to guide and provide expectations for interns, senior residents, and staff. The objective of this quality improvement project was to assess the effectiveness of the passport in improving the transition period and overall preparedness of the new CA-1s.

Methods

We surveyed 3 groups (CA-1s, CA-2s/CA-3s, and staff anesthesiologists) 6 months after the completion of passport implementation to retrospectively assess the 2021-2022 CA-1 class's preparedness across 7 domains compared with those who transitioned before passport implementation. Mann-Whitney U statistics and median effect sizes were used to compare pre- and postintervention.

Results

Self-reflected preparedness scores of the CA-1s were higher across all domains compared with the senior resident group (r = 0.328-0.548). Overall level of comfort and preparedness for the start of the CA-1 year was higher in the postintervention group (r = 0.162- 0.514). Staff anesthesiologists' perceived preparedness of the residents was also higher across all domains for the postintervention group (r = 0.197-0.387).

Conclusion

The un-pairing passport improved residents' and staff anesthesiologists' subjective assessments of the readiness of new CA-1 residents after a critical transition in their training. Similar tools can be more broadly applied to other anesthesiology residency and possibly fellowship programs as well as subspecialty rotations within those programs.

Development and Implementation of DIGEST: The Digital Interactive Gastroenterology Education Suite for Trainees

INTRODUCTION

Clinical clerkship curricula should exist to provide rotating learners on subspecialty rotations with consistent exposure to specific topics geared toward the discipline of interest, such as pediatric gastroenterology (GI). We aim to describe our experience developing and implementing DIGEST: the Digital Interactive Gastroenterology Education Suite for Trainees, a novel, online GI curriculum delivered to virtual, rotating learners during the coronavirus (COVID-19) pandemic stay-at-home order.

MATERIALS AND METHODS

A general needs assessment in 2019 identified a lack of standardized educational experience amongst the rotating learners on pediatric GI service. The COVID-19 pandemic compelled us to transition our curriculum from our institution's secure share drive to the GOOGLE classroom. A program evaluation was undertaken and included learner responses to content and confidence questionnaires and a health care professions education (HPE) expert's response to a course quality assessment rubric.

RESULTS

Feasibility-the final DIGEST product was free of charge to create but incurred direct and indirect costs of time and training on behalf of the authors. Acceptance-7 possible learners participated and responded to the questionnaires (100% response rate). Learners reported a superior learning experience and increased confidence with DIGEST. An HPE expert reported that the course design of DIGEST met or exceeded expectations in all categories.

CONCLUSIONS

DIGEST is a novel pediatric GI curriculum for rotating learners that could be rapidly deployed, or adapted, for a wide range of clinical disciplines within the Military Health System.

PICU Passport: Pilot study of a handheld resident curriculum

BACKGROUND

To explore the impact of an educational tool designed to streamline resident learning during their pediatric intensive care (PICU) rotations.

METHODS

Topics and procedures were chosen for inclusion based on national requirements for pediatric residents. Residents received a PICU Passport at the beginning of their rotations. PICU faculty were provided learning objectives for each topic. Residents and faculty were surveyed before and after starting use of the Passport.

RESULTS

Twenty-two residents pre-Passport and 38 residents post-Passport were compared. Residents were more satisfied with their educational experiences (27 % vs. 79 %; P < 0.001), more likely to report faculty targeted teaching towards knowledge gaps (5 % vs. 63 %; P < 0.001) and felt more empowered to ask faculty to discuss specific topics (27 % vs. 76 %; P = 0.002). The median number of teaching sessions increased from 3 to 10 (Z = 4.2; P < 0.001). Most residents (73 %) felt the Passport helped them keep track of their learning and identify gaps in their knowledge.

CONCLUSIONS

The PICU Passport helps residents keep track of their learning and identify gaps in their knowledge. Passport use increases resident satisfaction with education during their PICU rotation and empowers residents to ask PICU faculty to address specific knowledge gaps.

Feedback Frequency in Competence by Design: A Quality Improvement Initiative

BACKGROUND

Otolaryngology-head and neck surgery is in the first wave of residency training programs in Canada to adopt Competence by Design (CBD), a model of competency-based medical education. CBD is built on frequent, low-stakes assessments and requires an increase in the number of feedback interactions. The University of Toronto otolaryngology-head and neck surgery residents piloted the CBD model but were completing only 1 assessment every 4 weeks, which was insufficient to support CBD.

OBJECTIVE

This project aimed to increase assessment completion to once per resident per week using quality improvement methodology.

METHODS

Stakeholder engagement activities had residents and faculty characterize barriers to assessment completion. Brief electronic assessment forms were completed by faculty on residents' personal mobile devices in face-to-face encounters, and the number completed per resident was tracked for 10 months during the 2016-2017 pilot year. Response to the intervention was analyzed using statistical process control charts.

RESULTS

The first bundled intervention-a rule set dictating which clinical instance should be assessed, combined with a weekly reminder implemented for 10 weeks-was unsuccessful in increasing the frequency of assessments. The second intervention was a leaderboard, designed on an audit-and-feedback system, which sent weekly comparison e-mails of each resident's completion rate to all residents and the program director. The leaderboard demonstrated significant improvement from baseline over 10 weeks, increasing the assessment completion rate from 0.22 to 2.87 assessments per resident per week.

CONCLUSIONS

A resident-designed audit-and-feedback leaderboard system improved the frequency of CBD assessment completion.

Feedback With Performance Metric Scorecards Improves Resident Satisfaction but Does Not Impact Clinical Performance

OBJECTIVES

The Emergency Medicine Milestone Project, a framework for assessing competencies, has been used as a method of providing focused resident feedback. However, the emergency medicine milestones do not include specific objective data about resident clinical efficiency and productivity, and studies have shown that milestone-based feedback does not improve resident satisfaction with the feedback process. We examined whether providing performance metric reports to resident physicians improves their satisfaction with the feedback process and their clinical performance.

METHODS

We conducted a three-phase stepped-wedge randomized pilot study of emergency medicine residents at a single, urban academic site. In phase 1, all residents received traditional feedback; in phase 2, residents were randomized to receive traditional feedback (control group) or traditional feedback with performance metric reports (intervention group); and in phase 3, all residents received monthly performance metric reports and traditional feedback. To assess resident satisfaction with the feedback process, surveys using 6-point Likert scales were administered at each study phase and analyzed using two-sample t-tests. Analysis of variance in repeated measures was performed to compare impact of feedback on resident clinical performance, specifically patient treatment time (PTT) and patient visits per hour.

RESULTS

Forty-one residents participated in the trial of which 21 were randomized to the intervention group and 20 in the control group. Ninety percent of residents liked receiving the report and 74% believed that it better prepared them for expectations of becoming an attending physician. Additionally, residents randomized to the intervention group reported higher satisfaction (p = 0.01) with the quality of the feedback compared to residents in the control group. However, receiving performance metric reports, regardless of study phase or postgraduate year status, did not affect clinical performance, specifically PTT (183 minutes vs. 177 minutes, p = 0.34) or patients visits per hour (0.99 vs. 1.04, p = 0.46).

CONCLUSIONS

While feedback with performance metric reports did not improve resident clinical performance, resident physicians were more satisfied with the feedback process, and a majority of residents expressed liking the reports and felt that it better prepared them to become attending physicians. Residency training programs could consider augmenting feedback with performance metric reports to aide in the transition from resident to attending physician.

myTIPreport and Training for Independent Practice: A Tool for Real-Time Workplace Feedback for Milestones and Procedural Skills

BACKGROUND

Few tools currently exist for effective, accessible delivery of real-time, workplace feedback in the clinical setting.

OBJECTIVE

We developed and implemented a real-time, web-based tool for performance-based feedback in the clinical environment.

METHODS

The tool (myTIPreport) was designed for performance-based feedback to learners on the Accreditation Council for Graduate Medical Education (ACGME) Milestones and procedural skills. "TIP" stands for "Training for Independent Practice." We implemented myTIPreport in obstetrics and gynecology (Ob-Gyn) and female pelvic medicine and reconstructive surgery (FPMRS) programs between November 2014 and May 2015. Residents, fellows, teachers, and program directors completed preimplementation and postimplementation surveys on their perceptions of feedback.

RESULTS

Preimplementation surveys were completed by 656 participants of a total of 980 learners and teachers in 19 programs (12 Ob-Gyn and 7 FPMRS). This represented 72% (273 of 378) of learners and 64% (383 of 602) of teachers. Seventy percent of participants (381 of 546) reported having their own individual processes for real-time feedback; the majority (79%, 340 of 430) described these processes as informal discussions. Over 6 months, one-third of teachers and two-thirds of learners used the myTIPreport tool a total of 4311 times. Milestone feedback was recorded 944 times, and procedural feedback was recorded 3367 times. Feedback addressed all ACGME Milestones and procedures programmed into myTIPreport. Most program directors reported that tool implementation was successful.

CONCLUSIONS

The majority of learners successfully received workplace feedback using myTIPreport. This web-based tool, incorporating procedures and ACGME Milestones, may be an important transition from other feedback formats.

Rapid Web-Based Platform for Assessment of Orthopedic Surgery Patient Care Milestones: A 2-Year Validation

OBJECTIVE

To determine the validity, feasibility, and responsiveness of a new web-based platform for rapid milestone-based evaluations of orthopedic surgery residents.

SETTING

Single academic medical center, including a trauma center and pediatrics tertiary hospital.

PARTICIPANTS

Forty residents (PG1-5) in an orthopedic residency program and their faculty evaluators.

METHODS

Residents and faculty were trained and supported in the use of a novel trainee-initiated web-based evaluation system. Residents were encouraged to use the system to track progress on patient care subcompetencies. Two years of prospectively collected data were reviewed from residents at an academic program. The primary outcome was Spearman's rank correlation between postgraduate year (PGY) and competency level achieved as a measure of validity. Secondary outcomes assessed feasibility, resident self-evaluation versus faculty evaluation, the distributions among subcompetencies, and responsiveness over time.

RESULTS

Between February 2014 and February 2016, 856 orthopedic surgery patient care subcompetency evaluations were completed (1.2 evaluations per day). Residents promptly requested feedback after a procedure (median = 0 days, interquartile range: 0-2), and faculty responded within 2 days in 51% (median = 2 days, interquartile range: 0-13). Primary outcome showed a correlation between PGY and competency level (r = 0.78, p < 0.001), with significant differences in competency among PGYs (p < 0.001 by Kruskal-Wallis rank sum test). Self-evaluations by residents substantially agreed with faculty-assigned competency level (weighted Cohen's κ = 0.72, p < 0.001). Resident classes beginning the study as PGY1, 2, and 3 separately demonstrated gains in competency over time (Spearman's rank correlation 0.39, 0.60, 0.59, respectively, each p < 0.001). There was significant variance in the number of evaluations submitted per subcompetency (median = 43, range: 6-113) and competency level assigned (p < 0.01).

CONCLUSIONS

Rapid tracking of trainee competency with milestone-based evaluations in a learner-centered mobile platform demonstrated validity, feasibility, and responsiveness. Next Accreditation System-mandated data may be efficiently collected and used for trainee and program self-study.

Medical Student Perceptions of Learner-Initiated Feedback Using a Mobile Web Application

Feedback, especially timely, specific, and actionable feedback, frequently does not occur. Efforts to better understand methods to improve the effectiveness of feedback are an important area of educational research. This study represents preliminary work as part of a plan to investigate the perceptions of a student-driven system to request feedback from faculty using a mobile device and Web-based application. We hypothesize that medical students will perceive learner-initiated, timely feedback to be an essential component of clinical education. Furthermore, we predict that students will recognize the use of a mobile device and Web application to be an advantageous and effective method when requesting feedback from supervising physicians. Focus group data from 18 students enrolled in a 4-week anesthesia clerkship revealed the following themes: (1) students often have to solicit feedback, (2) timely feedback is perceived as being advantageous, (3) feedback from faculty is perceived to be more effective, (4) requesting feedback from faculty physicians poses challenges, (5) the decision to request feedback may be influenced by the student's clinical performance, and (6) using a mobile device and Web application may not guarantee timely feedback. Students perceived using a mobile Web-based application to initiate feedback from supervising physicians to be a valuable method of assessment. However, challenges and barriers were identified.

Are All Competencies Equal in the Eyes of Residents? A Multicenter Study of Emergency Medicine Residents' Interest in Feedback

Introduction

Feedback, particularly real-time feedback, is critical to resident education. The emergency medicine (EM) milestones were developed in 2012 to enhance resident assessment, and many programs use them to provide focused resident feedback. The purpose of this study was to evaluate EM residents’ level of interest in receiving real-time feedback on each of the 23 competencies/sub-competencies.

Methods

This was a multicenter cross-sectional study of EM residents. We surveyed participants on their level of interest in receiving real-time on-shift feedback on each of the 23 competencies/sub-competencies. Anonymous paper or computerized surveys were distributed to residents at three four-year training programs and three three-year training programs with a total of 223 resident respondents. Residents rated their level of interest in each milestone on a six-point Likert-type response scale. We calculated average level of interest for each of the 23 sub-competencies, for all 223 respondents and separately by postgraduate year (PGY) levels of training. One-way analyses of variance were performed to determine if there were differences in ratings by level of training.

Results

The overall survey response rate across all institutions was 82%. Emergency stabilization had the highest mean rating (5.47/6), while technology had the lowest rating (3.24/6). However, we observed no differences between levels of training on any of the 23 competencies/sub-competencies.

Conclusion

Residents seem to ascribe much more value in receiving feedback on domains involving high-risk, challenging procedural skills as compared to low-risk technical and communication skills. Further studies are necessary to determine whether residents’ perceived importance of competencies/sub-competencies needs to be considered when developing an assessment or feedback program based on these 23 EM competencies/sub-competencies.

Characteristics of Real-Time, Non-Critical Incident Debriefing Practices in the Emergency Department

INTRODUCTION

Benefits of post-simulation debriefings as an educational and feedback tool have been widely accepted for nearly a decade. Real-time, non-critical incident debriefing is similar to post-simulation debriefing; however, data on its practice in academic emergency departments (ED), is limited. Although tools such as TeamSTEPPS® (Team Strategies and Tools to Enhance Performance and Patient Safety) suggest debriefing after complicated medical situations, they do not teach debriefing skills suited to this purpose. Anecdotal evidence suggests that real-time debriefings (or non-critical incident debriefings) do in fact occur in academic EDs;, however, limited research has been performed on this subject. The objective of this study was to characterize real-time, non-critical incident debriefing practices in emergency medicine (EM).

METHODS

We conducted this multicenter cross-sectional study of EM attendings and residents at four large, high-volume, academic EM residency programs in New York City. Questionnaire design was based on a Delphi panel and pilot testing with expert panel. We sought a convenience sample from a potential pool of approximately 300 physicians across the four sites with the goal of obtaining >100 responses. The survey was sent electronically to the four residency list-serves with a total of six monthly completion reminder emails. We collected all data electronically and anonymously using SurveyMonkey.com; the data were then entered into and analyzed with Microsoft Excel.

RESULTS

The data elucidate various characteristics of current real-time debriefing trends in EM, including its definition, perceived benefits and barriers, as well as the variety of formats of debriefings currently being conducted.

CONCLUSION

This survey regarding the practice of real-time, non-critical incident debriefings in four major academic EM programs within New York City sheds light on three major, pertinent points: 1) real-time, non-critical incident debriefing definitely occurs in academic emergency practice; 2) in general, real-time debriefing is perceived to be of some value with respect to education, systems and performance improvement; 3) although it is practiced by clinicians, most report no formal training in actual debriefing techniques. Further study is needed to clarify actual benefits of real-time/non-critical incident debriefing as well as details on potential pitfalls of this practice and recommendations for best practices for use.

Learnings From the Pilot Implementation of Mobile Medical Milestones Application

BACKGROUND

Implementation of the educational milestones benefits from mobile technology that facilitates ready assessments in the clinical environment. We developed a point-of-care resident evaluation tool, the Mobile Medical Milestones Application (M3App), and piloted it in 8 North Carolina family medicine residency programs.

OBJECTIVE

We sought to examine variations we found in the use of the tool across programs and explored the experiences of program directors, faculty, and residents to better understand the perceived benefits and challenges of implementing the new tool.

METHODS

Residents and faculty completed presurveys and postsurveys about the tool and the evaluation process in their program. Program directors were interviewed individually. Interviews and open-ended survey responses were analyzed and coded using the constant comparative method, and responses were tabulated under themes.

RESULTS

Common perceptions included increased data collection, enhanced efficiency, and increased perceived quality of the information gathered with the M3App. Residents appreciated the timely, high-quality feedback they received. Faculty reported becoming more comfortable with the tool over time, and a more favorable evaluation of the tool was associated with higher utilization. Program directors reported improvements in faculty knowledge of the milestones and resident satisfaction with feedback.

CONCLUSIONS

Faculty and residents credited the M3App with improving the quality and efficiency of resident feedback. Residents appreciated the frequency, proximity, and specificity of feedback, and faculty reported the app improved their familiarity with the milestones. Implementation challenges included lack of a physician champion and competing demands on faculty time.

Using Transitional Year Milestones to Assess Graduating Medical Students' Skills During a Capstone Course

BACKGROUND

Undergraduate medical education (UME) follows the lead of graduate medical education (GME) in moving to competency-based assessment. The means for and the timing of competency-based assessments in UME are unclear.

OBJECTIVE

We explored the feasibility of using the Accreditation Council for Graduate Medical Education Transitional Year (TY) Milestones to assess student performance during a mandatory, fourth-year capstone course.

METHODS

Our single institution, observational study involved 99 medical students who completed the course in the spring of 2014. Students' skills were assessed by self, peer, and faculty assessment for 6 existing course activities using the TY Milestones. Evaluation completion rates and mean scores were calculated.

RESULTS

Students' mean milestone levels ranged between 2.2 and 3.6 (on a 5-level scoring rubric). Level 3 is the performance expected at the completion of a TY. Students performed highest in breaking bad news and developing a quality improvement project, and lowest in developing a learning plan, working in interdisciplinary teams, and stabilizing acutely ill patients. Evaluation completion rates were low for some evaluations, and precluded use of the data for assessing student performance in the capstone course. Students were less likely to complete separate online evaluations. Faculty were less likely to complete evaluations when activities did not include dedicated time for evaluations.

CONCLUSIONS

Assessment of student competence on 9 TY Milestones during a capstone course was useful, but achieving acceptable evaluation completion rates was challenging. Modifications are necessary if milestone scores from a capstone are intended to be used as a handoff between UME and GME.

Genetic Counseling Milestones: A Framework for Student Competency Evaluation

Graduate medical education has recently increased focus on the development of medical specialty competency milestones to provide a targeted tool for medical resident evaluation. Milestones provide developmental assessment of the attainment of competencies over the course of an educational program. An educational framework is described to explore the development of Genetic Counseling Milestones for the evaluation of the development of genetic counseling competencies by genetic counseling students. The development of Genetic Counseling Milestones may provide a valuable tool to assess genetic counseling students across all program activities. Historical educational context, current practices, and potential benefits and challenges in the development of Genetic Counseling Milestones are discussed.