Using the ACGME Milestones for Resident Self-Evaluation and Faculty Engagement

Using ACGME General Surgery Milestones to Define the Competent Foundational Surgical Resident

OBJECTIVE

In transitioning to competency-based surgical training, the need to clearly define competency is paramount. The purpose of this study is to define the well-prepared foundational resident using the ACGME General Surgery Milestones as our conceptual framework.

DESIGN

Participants reflected on their expectations of a well-prepared resident at the end of PGY1, then assigned milestone levels reflecting this level of competence for General Surgery Milestones 1.0 and 2.0. Subcompetency scores were averaged among residents and faculty. The level of the well-prepared foundational resident was determined based on the highest level within one standard deviation of faculty, resident, and total group averages.

SETTING

This took place during a dedicated education retreat at a single, large academic general surgery residency program.

PARTICIPANTS

Key faculty stakeholders and a representative sample of residents (PGY 1-5) within our institution participated.

RESULTS

Eight faculty and five residents completed Milestones 1.0 and 2.0 scoring. Mean scores between faculty and residents were compared. For 1.0, mean scores for Practice-Based Learning and Improvement 3 (PBLI 3) and Interpersonal Communication Skills 3 (ICS 3) were discernably lower for residents than for faculty (PBLI 3 1.3 (0.3) v 0.9 (0.2), p = 0.01; ICS3 1.6 (0.6) v 1.1 (1), p = 0.01). Scores of 2.0 were comparable across all subcompetency domains. With this broad agreement, Milestone-based competency standards were determined. Descriptive narratives of the KSAs were created for each subcompetency, combining the determined Milestones 1.0 and 2.0 levels.

CONCLUSIONS

We were able to clearly define the competent foundational resident using the ACGME Milestones as a conceptual framework. These Milestone levels reflect the culture and expectations in our department, providing a foundation upon which to build a program of assessment. This methodology can be readily replicated in other programs to reflect specific expectations of the program within the larger ACGME frameworks of competency.

Differences between resident self-assessments and faculty- assessments on Anesthesiology Milestones and associated factors: a cross-sectional study

BACKGROUND

Accurate self-assessment is crucial for the professional development of physicians. There has been sparse data on the accuracy of self-assessments on Anesthesiology Milestones. The aim of this study was to investigate the differences between resident self-assessments and faculty-assessments on Anesthesiology Milestones and the associated factors.

METHODS

This was a cross-sectional study conducted in a general tertiary university-affiliated hospital. We included anesthesia residents who were enrolled in the standardized residency training program in postgraduate year two and three at the time of the Milestone evaluation. We requested evaluations of competencies from both the Clinical Competency Committee faculty and the anesthesia residents themselves, utilizing the Chinese version of Anesthesiology Milestones in January 2023 and January 2024. The primary outcome was the differences between self- and faculty-assessments, calculated by subtracting the faculty-rated score from the self-rated score on each subcompetency.

RESULTS

A total of 46 and 42 residents were evaluated in year 2023 and 2024, respectively. The self-rated sum score was significantly higher than that rated by faculty [mean (standardized deviation): 120.39 (32.41) vs. 114.44 (23.71), P = 0.008 in paired t test] with an intraclass correlation coefficient of 0.55 [95% confidence interval (CI): 0.31 to 0.70]. The Bland-Altman plots revealed significant overestimation in patient care (bias 0.32, 95% CI: 0.05 to 0.60), practice-based learning and improvement (bias 0.45, 95% CI: 0.07 to 0.84), and professionalism (bias 0.37, 95% CI: 0.02 to 0.72). Ratings from residents with master's degrees (mean difference: -1.06, 95% CI: -1.80 to -0.32, P = 0.005) and doctorate degrees (mean difference: -1.14, 95% CI: -1.91 to -0.38, P = 0.003) were closer to the faculty-assessments than residents with bachelor's degrees. Compared with patient care, the differences between self- and faculty- rated scores were smaller in medical knowledge (mean difference: -0.18, 95% CI: -0.35 to -0.02, P = 0.031) and interpersonal and communication skills (mean difference: -0.41, 95% CI: -0.64 to -0.19, P < 0.001) in the generalized estimating equation logistic regression model.

CONCLUSIONS

This study revealed that residents tended to overestimate themselves, emphasizing the need to improve the accuracy of Milestones self-assessment. The differences between self- and faculty-assessments were associated with residents' degrees and domains of competency.

Assessing the impact of adopting a competency-based medical education framework and ACGME-I accreditation on educational outcomes in a family medicine residency program in Abu Dhabi Emirate, United Arab Emirates

Background

Competency-Based Medical Education (CBME) is now mandated by many graduate and undergraduate accreditation standards. Evaluating CBME is essential for quantifying its impact, finding supporting evidence for the efforts invested in accreditation processes, and determining future steps. The Ambulatory Healthcare Services (AHS) family medicine residency program has been accredited by the Accreditation Council of Graduate Medical Education-International (ACGME-I) since 2013. This study aims to report the Abu Dhabi program's experience in implementing CBME and accreditation.

Objectives

Compare the two residents' cohorts' performance pre-and post-ACGME-I accreditation.Study the bi-annually reported milestones as a graduating residents' performance prognostic tool.

Methods

All residents in the program from 2008 to 2019 were included. They are called Cohort one-the intake from 2008 to 2012, before the ACGME accreditation, and Cohort two-the intake from 2013 to 2019, after the ACGME accreditation, with the milestones used. The mandatory annual in-training exam was used as an indication of the change in competency between the two cohorts. Among Cohort two ACGME-I, the biannual milestones data were studied to find the correlation between residents' early and graduating milestones.

Results

A total of 112 residents were included: 36 in Cohort one and 76 in Cohort two. In Cohort one, before the ACGME accreditation, no significant associations were identified between residents' graduation in-training exam and their early performance indicators, while in Cohort two, there were significant correlations between almost all performance metrics. Early milestones are correlated with the graduation in-training exam score. Linear regression confirmed this relationship after controlling the residents' undergraduate Grade Point Average (GPA). Competency development continues to improve even after residents complete training at Post Graduate Year, PGY4, as residents' achievement in PGY5 continues to improve.

Conclusion

Improved achievement of residents after the introduction of the ACGME-I accreditation is evident. Additionally, the correlation between the graduation in-training exam and graduation milestones, with earlier milestones, suggests a possible use of early milestones in predicting outcomes.

An Easily Implemented, Inexpensive, Useful Novel Surgical Intern Wellness Initiative

OBJECTIVE

Determine whether use of reflective questions asked on a twice monthly basis is a useful addition to our intern wellness curriculum, with a goal of longitudinal development. Prior studies have demonstrated the use of reflection in processing educational experiences toward professional growth at both the medical student and resident level.

DESIGN

During the first year, every 2 weeks, the 13 interns were asked and answered 2 reflective prompts by email. Their responses went to a single faculty member and were then blinded for analysis. The second year of the program, prompts were discussed by participants in a closed group setting. Participation was voluntary. The questions fell into 6 major categories: role expectations, role assessment, role affirmation, role reflection, emotional self-assessment, work-life integration, and boundaries. Thematic analysis of the responses was performed using an inductive approach by 2 independent expert reviewers.

SETTING

Brown General Surgery Residency Program, academic years 2021 to 2022 and 2022 to 2023.

CONCLUSIONS

Use of reflective questions is a valuable tool as part of an intern wellness curriculum and can be easily implemented. It is inexpensive, does not require a huge time commitment, and is easily adaptable to a program's specific needs. It encourages developing surgeons to recognize and share in their emotions as they encounter the new and stressful experiences inherent in residency and may help to prevent burnout. Sustained participation through the year and robust responses suggest good resident engagement and acceptance of the program.

Relationship between epa level of supervision with their associated subcompetency milestone levels in pediatric fellow assessment

BACKGROUND

Entrustable Professional Activities (EPA) and competencies represent components of a competency-based education framework. EPAs are assessed based on the level of supervision (LOS) necessary to perform the activity safely and effectively. The broad competencies, broken down into narrower subcompetencies, are assessed using milestones, observable behaviors of one's abilities along a developmental spectrum. Integration of the two methods, accomplished by mapping the most relevant subcompetencies to each EPA, may provide a cross check between the two forms of assessment and uncover those subcompetencies that have the greatest influence on the EPA assessment.

OBJECTIVES

We hypothesized that 1) there would be a strong correlation between EPA LOS ratings with the milestone levels for the subcompetencies mapped to the EPA; 2) some subcompetencies would be more critical in determining entrustment decisions than others, and 3) the correlation would be weaker if the analysis included only milestones reported to the Accreditation Council for Graduate Medical Education (ACGME).

METHODS

In fall 2014 and spring 2015, the Subspecialty Pediatrics Investigator Network asked Clinical Competency Committees to assign milestone levels to each trainee enrolled in a pediatric fellowship for all subcompetencies mapped to 6 Common Pediatric Subspecialty EPAs as well as provide a rating for each EPA based upon a 5-point LOS scale.

RESULTS

One-thousand forty fellows were assessed in fall and 1048 in spring, representing about 27% of all fellows. For each EPA and in both periods, the average milestone level was highly correlated with LOS (rho range 0.59-0.74; p < 0.001). Correlations were similar when using a weighted versus unweighted milestone score or using only the ACGME reported milestones (p > 0.05).

CONCLUSIONS

We found a strong relationship between milestone level and EPA LOS rating but no difference if the subcompetencies were weighted, or if only milestones reported to the ACGME were used. Our results suggest that representative behaviors needed to effectively perform the EPA, such as key subcompetencies and milestones, allow for future language adaptations while still supporting the current model of assessment. In addition, these data provide additional validity evidence for using these complementary tools in building a program of assessment.

Autonomy Maps: Building a Shared Mental Model for Progressive Resident Operative Autonomy on the Road to Entrustable Professional Activities (EPAs)

Our residents expressed dissatisfaction with operative autonomy and faculty feedback regarding technical skills. They reported variability among faculty regarding allowed operative autonomy. Our goals were to establish a shared mental model among residents and faculty regarding intraoperative performance expectations. We asked faculty to assign a level of expected autonomy (Zwisch scale) for various steps of common procedures according to the resident post-graduate year. Through an iterative process, the maps were standardized across service lines. The resulting "Autonomy Maps" were distributed to the faculty and residents. We held educational sessions and set expectations for use. Selected benchmarks were incorporated into resident end-of-rotation assessment forms. Initial operative case mapping identified variability in faculty expectations for a given post-graduate year and procedure. Residents reported improved satisfaction with understanding expectations regarding operative performance. Establishing autonomy benchmarks facilitated more specific feedback regarding residents' technical skills. Faculty expectations for resident operative autonomy are variable. Autonomy Maps provide structure for a shared mental model between faculty and residents for progressive operative autonomy and serve as a framework for expectations that improve resident satisfaction. Case-specific technical benchmarks are useful tools for assessing residents' technical milestones.

New ACGME Clinician Educator Milestones as a Roadmap for Faculty Development: a Position Paper from the Society of General Internal Medicine Education Committee

Traditionally, clinician educators are tasked with the responsibility of training future physician workforce. However, there is limited identification of skills required to fulfill this responsibility and a lack of consensus on effective faculty development for career growth as a clinician educator. The newly released Accreditation Council of Graduate Medical Education (ACGME) Clinician Educator (CE) Milestones framework outlines important skills for clinician educators and provides the opportunity to create robust faculty development. In this paper, members of the Society of General Internal Medicine Education Committee discuss the importance of these CE Milestones, outline the novel themes highlighted in the project, and provide recommendations for proper application on both the individual and institutional levels to optimize faculty development. The paper discusses strategies for how to apply the CE Milestones as a tool to create a culture of professional growth and self-directed learning. Using a reflective approach, CE faculty and mentors can identify areas of proficiency and opportunities for growth, thereby creating individualized professional development plans for career success. Institutions should use aggregate CE Milestones data as a needs assessment of their faculty "population" to create targeted faculty development. Most importantly, institutions should not use CE Milestones for high-stakes assessments but rather encourage reflection by CE faculty and create subsequent robust faculty development programs. The ACGME CE Milestones present an exciting opportunity and lay an important foundation for future CE faculty development.

Bias in Surgical Residency Evaluations: A Scoping Review

OBJECTIVE

Given widespread disparities in the surgical workforce and the advent of competency-based training models that rely on objective evaluations of resident performance, this review aims to describe the landscape of bias in the evaluation methods of residents in surgical training programs in the United States.

DESIGN

A scoping review was conducted within PubMed, Embase, Web of Science, and ERIC in May 2022, without a date restriction. Studies were screened and reviewed in duplicate by 3 reviewers. Data were described descriptively.

SETTING/PARTICIPANTS

English-language studies conducted in the United States that assessed bias in the evaluation of surgical residents were included.

RESULTS

The search yielded 1641 studies, of which 53 met inclusion criteria. Of the included studies, 26 (49.1%) were retrospective cohort studies, 25 (47.2%) were cross-sectional studies, and 2 (3.8%) were prospective cohort studies. The majority included general surgery residents (n = 30, 56.6%) and nonstandardized examination modalities (n = 38, 71.7%), such as video-based skills evaluations (n = 5, 13.2%). The most common performance metric evaluated was operative skill (n = 22, 41.5%). Overall, the majority of studies demonstrated bias (n = 38, 73.6%) and most investigated gender bias (n = 46, 86.8%). Most studies reported disadvantages for female trainees regarding standardized examinations (80.0%), self-evaluations (73.7%), and program-level evaluations (71.4%). Four studies (7.6%) assessed racial bias, of which all reported disadvantages for trainees underrepresented in surgery.

CONCLUSIONS

Evaluation methods for surgery residents may be prone to bias, particularly with regard to female trainees. Research is warranted regarding other implicit and explicit biases, such as racial bias, as well as for nongeneral surgery subspecialties.

Perioperative Reflection: An Understudied Yet Common Practice Among Surgical Residents

OBJECTIVE

Directed reflection, a descriptive, analytical, and critical reasoning process, has been implemented in several professional fields as a method of developing expertise. In surgery, investigation of reflection has been limited. To date, we are unaware of any study that has sought to define or characterize the independent reflective practices of surgical trainees. In this small pilot study, we characterize the independent operative reflection activities among surgical residents at a single academic institution.

DESIGN

An unvalidated survey was developed through cognitive interviews and pilot testing to assess 5 aspects of operative reflection. An IRB approved electronic survey was developed and distributed to all general surgery residents at an academic institution. Responses were divided by post-graduate year into junior residents (JR; PGY 1-3) and senior residents (SR; PGY 4-5).

RESULTS

A response rate of 46% (23/50) was attained. All respondents self-identified as operative reflectors. Three primary methods of reflection were identified: mental, written, and oral. Mental reflection was the primary reflection method of 75% of our respondents as identified by step-by-step mental rehearsal, internal narration, and question series. Reflection was practiced everyday by 42% of respondents. Topics of reflection focused on weaknesses (50%), self-improvement (35%) and judgment (15%). Several respondents (57%) indicated that reflection-based learning tools would be useful. No statistically significant changes in the method or frequency were established between junior and senior residents (p = 0.378, p = 0.464, respectively). Variation in mental reflection activities among JR and SR groups were found (p = 0.029, p = 0.00004, respectively).

CONCLUSIONS

The enduring role of reflection in experiential learning has been established in many professional fields but remains under-utilized in surgical education. Our study suggests surgical residents reflect independently of curriculum in individual and categorizable ways. Further investigation is required to adequately characterize reflective habits of the surgical resident and evaluate its impact on competency.

Radiation Oncology Resident Evaluations: Current Methods and Resident Perceptions

Purpose

This study analyzes assessments within radiation oncology (RO) to determine characteristics of existing assessment methods and then report resident perceptions of these methods. We hypothesize familiarity with evaluation methods is predictive of the perceived utility of evaluations and behavioral changes.

Methods and materials

This study was conducted in 2 phases. Phase 1 involved requesting resident evaluation forms from RO residency programs to assess the Accreditation Council for Graduate Medical Education 6 Core Competencies. Analysis of variance was used to determine any significant differences between institutions or categories of questions. In phase 2, RO residents were surveyed about familiarity with the Accreditation Council for Graduate Medical Education Milestones and their perceptions about the current methods. Responses to questions were further analyzed using linear regression models.

Results

In phase 1, forms were obtained from 13 institutions, and they were based on the 6 Core Competencies with an average of 19 questions (standard deviation, 11; range, 5-47) in total. Analysis of variance did not show significant variation in the number of questions between the categories (F = 0.78, P = .6). A significant difference in the mean number of questions used for assessing each of the competencies was found between institutions (F = 6.6, P < .01). In phase 2, a majority of surveyed residents reported being "not" or only "slightly familiar" with the competencies (59.6%) and the factors used to assess them (73.1%). Resident-reported familiarity with the evaluation methods was not found to be a significant predictor of likelihood of changing postevaluation (coefficient = 0.41, P = .204), intimidation from receiving evaluations (coefficient = -0.06, P = .792), stress of receiving evaluations (coefficient = -0.11, P = .62), or usefulness of evaluations (coefficient = -0.02, P = .83).

Conclusions

Familiarity with evaluation methods is not correlated with perceptions or behavioral changes necessitating further investigation of alternative predictor variables. Despite the low familiarity with evaluation tools, most residents reported that evaluations were useful and likely to elicit changes in their behaviors and practice, highlighting the value of current evaluation methods.

Reimagining the Clinical Competency Committee to Enhance Education and Prepare for Competency-Based Time-Variable Advancement

Assessing residents and clinical fellows is a high-stakes activity. Effective assessment is important throughout training so that identified areas of strength and weakness can guide educational planning to optimize outcomes. Assessment has historically been underemphasized although medical education oversight organizations have strengthened requirements in recent years. Growing acceptance of competency-based medical education and its logical extension to competency-based time-variable (CB-TV) graduate medical education (GME) further highlights the importance of implementing effective evidence-based approaches to assessment. The Clinical Competency Committee (CCC) has emerged as a key programmatic structure in graduate medical education. In the context of launching a multi-specialty pilot of CB-TV GME in our health system, we have examined several program's CCC processes and reviewed the relevant literature to propose enhancements to CCCs. We recommend that all CCCs fulfill three core goals, regularly applied to every GME trainee: (1) discern and describe the resident's developmental status to individualize education, (2) determine readiness for unsupervised practice, and (3) foster self-assessment ability. We integrate the literature and observations from GME program CCCs in our institutions to evaluate how current CCC processes support or undermine these goals. Obstacles and key enablers are identified. Finally, we recommend ways to achieve the stated goals, including the following: (1) assess and promote the development of competency in all trainees, not just outliers, through a shared model of assessment and competency-based advancement; (2) strengthen CCC assessment processes to determine trainee readiness for independent practice; and (3) promote trainee reflection and informed self-assessment. The importance of coaching for competency, robust workplace-based assessments, feedback, and co-production of individualized learning plans are emphasized. Individual programs and their CCCs must strengthen assessment tools and frameworks to realize the potential of competency-oriented education.

Using the Psychiatry Resident-In-Training Examination (PRITE) to Assess the Psychiatry Medical Knowledge Milestones in Psychiatry

OBJECTIVE

The introduction of the Milestone Project underscored the need for objective assessments of resident progress across the competencies. Therefore, the authors examined the Psychiatry Resident-In-Training Examination (PRITE) utility for measuring improvements in medical knowledge (MK).

METHODS

The authors compared the mean performance for each MK subcompetency by resident year for all residents taking the PRITE from 2015 to 2017 (18,175 examination administrations). In addition, they surveyed psychiatry residency program directors regarding how well they thought they teach these subcompetencies.

RESULTS

Increases in MK subcompetencies by resident year were significant for Psychopathology (p < 0.003), Psychotherapy (p < 0.002), and Somatic Therapies (p < 0.000). Development, Clinical Neuroscience, and Practice of Psychiatry did not show statistically significant differences between postgraduate years. Eighty psychiatry program directors responded to the survey and felt optimistic about their ability to teach the Psychopathology, Psychotherapy, Somatic Therapies, and Practice of Psychiatry subcompetencies.

CONCLUSIONS

The PRITE measured significant improvements in medical knowledge for several of the core subcompetencies. The program director's responses would suggest that the lack of statistically significant differences found for Development and Clinical Neuroscience reflects areas in need of curricular development. The disparity between PRITE performance and program director perception of the Practice of Psychiatry subcompetency may reflect difficulties in defining the scope of this subcompetency. Overall, this suggests that structured examinations help measure improvements in certain subcompetencies and may also help identify curricular needs. However, there may be potential problems with the definition of some subcompetencies.

The Accreditation Council for Graduate Medical Education Milestones in Integrated Plastic Surgery Programs: How Competency-Based Assessment Has Been Implemented

BACKGROUND

The plastic surgery milestones are a central component of resident assessment. The authors performed a survey to evaluate how milestones have been implemented across integrated plastic surgery programs and how faculty perceive the Milestones Project has impacted their program.

METHODS

A 25-question survey was directed to the head of the clinical competency committee of all 82 integrated plastic surgery programs through the American Council of Academic Plastic Surgeons. The survey queried the composition of the committee, how ratings are generated, and the data used in generating these ratings.

RESULTS

Committee leaders from 57 programs (69.5 percent) responded to the survey. For most programs (73 percent), one faculty member completed milestone ratings for each resident and reviewed them with the residents. To determine milestone ratings, 92 percent utilized resident assessment after every rotation, and 63 percent utilized in-service examination scores. For documented resident assessment overall, 96 percent of programs assess residents after every rotation; 37 percent asses after every procedure. Feedback is most frequently provided to residents after every rotation (52 percent), rather than only during committee reviews (32 percent) or after every documented assessment (16 percent). Sixty-four percent of respondents did not believe that milestones have helped in the mentorship role.

CONCLUSIONS

Implementation of the milestones has varied among integrated plastic surgery programs. Structured training for core faculty directed to methods of assessment will make milestones a more effective a tool by which to improve resident evaluation and education. These findings provide a key data set by which to revise the milestones for their second iteration.

A Multi-Institutional Collaborative To Assess the Knowledge and Skills of Medicine-Pediatrics Residents in Health Care Transition

Background Pediatric to adult health care transition (HCT) is an essential process in the care of youth with special health care needs (YSHCN). Many internal medicine-pediatrics (med-peds) residency programs have developed curricula to teach transition knowledge and skills for the care of YSHCN. Objective Using a national med-peds program director quality improvement collaborative to improve transition curriculum, we aim to identify curricular content areas of improvement by describing baseline trainee knowledge and skills taught through existing transition curricula in med-peds programs. Methods We analyzed data collected during the 2018-2019 national med-peds program director quality improvement collaborative to improve transition curriculum. Program directors assessed their programs, and trainees assessed themselves on five transition goals by completing a Likert-scale questionnaire. In addition, trainees received an objective assessment of their knowledge through a multiple-choice questionnaire (MCQ). Results All 19 programs in the collaborative, and 193 of 316 trainees from these programs, completed the questionnaires. Most programs were based at academic centers (68%) and provided transition training via didactics (63%) and/or subspecialty rotations (58%). More programs had high confidence (95%) than trainees (58%) in goal 1 (knowledge and skills of the issues around transition), whereas more trainees had high confidence (60%) than programs (47%) in goal 2 (understanding the developmental and psychosocial aspects of transition). Programs and trainees self-assessed lower in goals related to health insurance, educational and vocational needs, and application of health care system knowledge to the practice environment (goals 3, 4, and 5, respectively). Conclusions Using the assessments of the program directors and resident trainees, we identified subject areas for improvement of transition curricula, including health insurance and the application of health care system knowledge to the practice environment.

Metrics of Resident Achievement for Defining Program Aims

Introduction

Resident achievement data is a powerful but underutilized means of program evaluation, allowing programs to empirically measure whether they are meeting their program aims, facilitate refinement of curricula and improve resident recruitment efforts. The goal was to provide an overview of available metrics of resident achievement and how these metrics can be used to inform program aims.

Methods

A literature search was performed using PubMed and Google Scholar between May and November of 2020. Publications were eligible for inclusion if they discussed or assessed “excellence” or “success” during residency training. A narrative review structure was chosen due to the intention to provide an examination of the literature on available resident achievement metrics.

Results

57 publications met inclusion criteria and were included in the review. Metrics of excellence were grouped into larger categories, including success defined by program factors, academics, national competencies, employer factors, and possible new metrics.

Conclusions

Programs can best evaluate whether they are meeting their program aims by creating a list of important resident-level metrics based on their stated goals and values using one or more of the published definitions as a foundation. Each program must define which metrics align best with their individual program aims and mission.

Can ACGME Milestones predict surgical specialty board passage: an example in Obstetrics and Gynecology

Background:

Multiple tools including Accreditation Council for Graduate Medical Education (ACGME) standardized milestones can be utilized to assess trainee and residency program performance. However, little is known regarding the objective validation of these tools in predicting written board passage.

Methods:

In this retrospective study, data was gathered on n = 45 Wayne State University Obstetrics and Gynecology program graduates over the five-year period ending July 2018. United States Medical Licensing Examination (USMLE) scores, Council on Resident Education in Obstetrics and Gynecology (CREOG) in-training scores and ACGME milestones were used to predict American Board of Obstetrics and Gynecology (ABOG) board passage success on first attempt. Significance was set at p < 0.05.

Results:

Written board passage was associated with average CREOGs (p = 0.01) and milestones (p = 0.008) while USMLE1 was not significantly associated (p = 0.055). USMLE1 <217 (Positive predictive value (PPV) = 96%). CREOGs <197 (PPV = 100%) and milestones <3.25 (PPV = 100%), particularly practice-based learning and systems-based practice milestones were most strongly correlated with board failure. Using a combination of these two milestones, it is possible to correctly predict board passage using our model (PPV = 86%).

Discussion:

This study is the first validating the utility of milestones in a surgical specialty by demonstrating their ability to predict board passage. Residents with CREOGs or milestones below thresholds are at risk for board failure and may warrant early intervention.

How Do We Choose? A Review of Residency Application Scoring Systems

INTRODUCTION

As the USMLE Step 1 Board exam moves to a pass/fail system there will be fewer objective measurements available to evaluate students applying to residency programs. Thus, there is a need for a reliable, validated method of screening applicants based on all areas of their application. To this end, we conducted a literature review search to examine previously described residency application screening tools.

METHODS

A PubMed search was conducted using the keywords "residency," "applicant," "scoring," "algorithm," and "ranking." The search was limited to the last 10 years, and only articles written in English with the full-text available were included. The initial search yielded 512 results. Titles and abstracts were evaluated for inclusion and 11 articles met criteria for full-text evaluation. An additional 6 articles were excluded with reason following the full-text evaluation.

RESULTS

A total of 5 papers were included in our descriptive analysis. Villwock et al. used the open-source STAR algorithm to create an initial interview list based on program-specific desirable attributes. Bowe et al. attempted to develop a screening tool based on the 6 ACGME competencies that could accurately predict a resident's performance. Similarly, Lyons et al. worked with an outside consulting firm to develop a screening tool that used several situational judgment questions to assess desired competencies for first year residents. Schenker et al. developed an evaluation process that used a combination of a standardized screening tool and semistructured interviews to produce a final rank list. Hu et al. created a screening tool for pharmacology residency applicants based on specific domains.

CONCLUSIONS

Several residency application scoring systems have been evaluated for use in the initial screening process, but there is no consensus on which system is superior and whether or not those systems are successful in selecting the "best" candidates.

Application of mentorship program for another aspect of surgical residency training: The importance of academia in surgical training

Traditionally, surgical residency training is more focused on obtaining surgical skills through a well-established coaching system worldwide. However, constant advances in medical science require surgeons to learn not only surgical skills but also the ability of scientific research to improve clinical practice and future professional development. The study aims to emphasize that professional education in terms of scientific research is also significant for surgical residency training.All residents who had been recruited in a medical center for the surgery residency program between years 2006 and 2015 were evaluated in the study. Generally, every resident is assigned to a mentor since the first year of residency. Then, the mentor would help the resident qualify a 2-step evaluation in terms of scientific research during the residency training program.A total of 193 residents were evaluated in the study. All of them had completed the first step regarding oral presentation of their designated research, and the majority of residents obtained 80 to 90 points that were rated by referees. Overall, 102 residents (52.8%) had completed the second step with the publication of a research manuscript. The percentage of residents who had fulfilled the criteria of this 2-step assessment ranged from 35.3% to 81.8% by year.The continuing education for surgical residents should not be limited in coaching clinical practice. Scientific research is also essential for current surgical residency training, and a formal mentorship program may be beneficial for the future professional development of surgical residents. However, the success of the 2-step evaluation could possibly depend on the career choices of the residents instead of the mentorship program.

Reliability and validity of the international standards for neurological classification of spinal cord injury in patients with non-traumatic spinal cord lesions

STUDY DESIGN

Prospective, observational study.

OBJECTIVES

The International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) represent the gold standard for the assessment of patients with spinal cord injury (SCI) and their measurement properties have been evaluated in patients with traumatic lesions. Albeit the ISNCSCI are widely used also for the assessment and prognosis of patients with non-traumatic SCI, a validation of this grading system in this sample has never been performed. Therefore, the aim of this study is to evaluate the measurement properties of the ISNCSCI in a population of persons with non-traumatic SCI.

SETTING

Three Italian rehabilitation hospitals.

METHODS

The sample included 140 patients with non-traumatic SCI of different etiology, level and grade, for a total of 169 evaluations performed by two examiners. Cronbach's Alpha was used to evaluate the internal consistency of the ISNCSCI various components. The agreement between two examiners of each center in the definition of different components was used to assess the inter-rater reliability. The construct validity was evaluated through the correlation of the ISNCSCI with the Spinal Cord Independence Measure (SCIM).

RESULTS

The ISNCSCI showed substantial internal consistency, and substantial inter-rater agreement for AIS grade, cumulative motor and sensory scores. The motor scores for upper and lower extremity showed fair to moderate correlation with SCIM self-care and motility subscores, respectively. The ISNCSCI total motor score correlated with the total SCIM score.

CONCLUSIONS

Our study demonstrates that the ISNCSCI are a valid and reliable tool for the assessment of patients with non-traumatic SCI.

"When in Doubt, Ask the Patient": A Quantitative, Patient-Oriented Approach to Formative Assessment of CanMEDS Roles

INTRODUCTION

Since the introduction of competency-based frameworks into postgraduate medical curricula, educators have struggled to implement robust assessment tools that document the progression of necessary skills. The global movement towards competency-based medical education demands validated assessment tools. Our objective was to provide validity evidence for the Ottawa CanMEDS Competency Assessment Tool (OCCAT), designed to assess clinical performance in the communicator, professional, and health advocate CanMEDS roles.

METHODS

We developed the OCCAT, a 29-item questionnaire informed by specialty-specific Entrustable Professional Activities and consultation with stakeholders, including patients. Our sample included nine neonatal-perinatal medicine and maternal fetal medicine fellows rotating through antenatal high-risk clinics at the Ottawa Hospital. Following 70 unique encounters, the OCCAT was completed by patients and learners. Generalizability theory was used to determine overall reliability of scores. Differences in self and patient ratings were assessed using analyses of variance.

RESULTS

Generalizability analysis demonstrated that both questionnaires produced reliable scores (G-coefficient > 0.9). Self-scores were significantly lower than patient scores across all competencies, F(1, 6) = 13.9, p = .007. Variability analysis demonstrated that trainee scores varied across all competencies, suggesting both groups were able to recognize competencies as distinct and discriminate favorable behaviors belonging to each.

DISCUSSION

Our findings lend support to the movement to integrate self-assessment and patient feedback in formal evaluations for the purpose of enriched learner experiences and improved patient outcomes. We anticipate that the OCCAT will facilitate bridging to competency-based medical education.

Trainee Sex and Accreditation Council for Graduate Medical Education Milestone Assessments During General Surgery Residency

Importance

In evaluating the effectiveness of general surgery (GS) training, an unbiased assessment of the progression of residents with attention to individual learner factors is imperative.

Objective

To evaluate the role of trainee sex in milestone achievement over the course of GS residency using national data from the Accreditation Council for Graduate Medical Education (ACGME).

Design, Setting, and Participants

This cross-sectional study evaluated female and male GS residents enrolled in ACGME-accredited programs in the US from 2014 to 2018 with reported variation in milestones performance across years in training and representation. Data were analyzed from November 2019 to June 2021.

Main Outcomes and Measures

Mean reported milestone score at initial and final assessment, and predicted time-to-attainment of equivalent performance by sex.

Results

Among 4476 GS residents from 250 programs who had milestone assessments at any point in their clinical training, 1735 were female (38.8%). Initially, female and male residents received similar mean (SD) milestone scores (1.95 [0.50] vs 1.94 [0.50]; P = .69). At the final assessment, female trainees received overall lower mean milestone scores than male trainees (4.25 vs 4.31; P < .001). Significantly lower mean milestone scores were reported for female residents at the final assessment for several subcompetencies in both univariate and multivariate analyses, with only medical knowledge 1 (pathophysiology, diagnosis, and initial management) common to both. Multilevel mixed-effects linear modeling demonstrated that female trainees had significantly lower rates of monthly milestone attainment in the subcompetency of medical knowledge 1, which was associated with a significant difference in training time of approximately 1.8 months.

Conclusions and Relevance

Both female and male GS trainees achieved the competency scores necessary to transition to independence after residency as measured by the milestones assessment system. Initially, there were no sex differences in milestone score. By graduation, there were differences in the measured assessment of female and male trainees across several subcompetencies. Careful monitoring for sex bias in the evaluation of trainees and scrutiny of the training process is needed to ensure that surgical residency programs support the educational needs of both female and male trainees.

Female Residents Give Themselves Lower Scores Than Male Colleagues and Faculty Evaluators on ACGME Milestones

OBJECTIVE

Orthopedic surgery is one of the specialties with the lowest number of women residents and practicing surgeons. The gender discrepancy in orthopedic residency training may drive a competency bias. We asked whether female orthopedic surgery residents score themselves lower on the Accreditation Council for Graduate Medical Education (ACGME) Milestones than their male counterparts, and lower than their faculty evaluators.

DESIGN

We conducted a retrospective review of ACGME Milestone data from faculty and residents over a 4-year period. The data were analyzed using a snapshot of PGY2 (n = 20 residents) and PGY4 (n = 19 residents) scores, and using a Generalized Estimation Equation (GEE) to account for additional data points from the same residents over the 4-year data collection period.

SETTING

Assessment scores were compiled from a single orthopedic surgery residency at Oregon Health & Science University from 2014 to 2017.

PARTICIPANTS

The residency program has 5 residents in each program year (PGY1 through PGY5); a total of 25 residents during each year of the study were included.

RESULTS

On average, female residents scored themselves lower than both their male counterparts and their faculty mentors. Female PGY2 self-evaluation scores were lower than males in both patient care (p = 0.005) and medical knowledge (p < 0.001). When the GEE model was applied to 99 responses from 41 residents over a 4-year period, there were no gender-related differences in resident self-evaluation scores and in faculty scores of male and female residents, with the exception of meniscal tear. For this milestone, faculty rated female residents lower than males. Furthermore, the differences between faculty evaluation scores and resident self-evaluation scores were significantly lower for males than for females for 4 of the clinical domains, as well as the systems-based practice domains of cost and communication.

CONCLUSIONS

Our results indicate female residents are at risk for a competency bias during training, as reflected by evaluations using the ACGME Milestones.

Association Between Entrustable Professional Activities and Milestones Evaluations: Real-time Assessments Correlate With Semiannual Reviews

OBJECTIVE

Entrustable professional activities (EPAs) have been developed to refine competency-based education. The American Board of Surgery has initiated a 2-year pilot study to evaluate the impact of EPAs on the evaluation and feedback of surgical residents. The ACGME Milestones in Surgery is a semiannual competency-based evaluation program to measure resident progression through 16 professional attributes across 8 practice domains. The correlation between these 2 evaluation tools remains unclear. The purpose of this study is to evaluate this correlation through comparison of an EPA with the corresponding elements of the ACGME Milestones.

DESIGN

From July, 2018 to October, 2019, all residents submitting EPA evaluations for gall bladder disease were evaluated for preoperative, intraoperative, and/or postoperative entrustability. The ratings were converted to a numerical rank from 0 to 4. Milestones scores from May 2019 and November 2019 were obtained for each resident, with scores ranging from 0 to 4. The gall bladder EPA incorporates the operative PC3 and MK2 and nonoperative PC1, PC2, and ICS3 components. Spearman rank correlation was conducted to evaluate the association between each resident's median EPA ranking and his/her milestones scores.

SETTING

SUNY Upstate Medical University, Syracuse, NY, a university-based hospital.

PARTICIPANTS

General surgery residents.

RESULTS

Among 24 residents, 106 intraoperative EPA evaluations were. For both the May and November milestones, significant positive correlations were noted for PC3 (correlation coefficient ρ = 0.690, p < 0.001; ρ = 0.876, p < 0.001). Similarly, for MK2, a significant positive correlation was noted (ρ = 0.882, p < 0.001; ρ = 0.759, p < 0.001). Interestingly, significant positive correlations were also identified between the 3 nonoperative milestones and the intraoperative entrustability ranking.

CONCLUSIONS

We observed significant correlations between EPAs for cholecystectomy and associated milestones evaluation scores. These findings indicate that EPAs may provide more timely and specific feedback than existing tools and, on aggregate, may improve upon existing formative feedback practices provided through the biannual evaluation of surgical residents.

Implementation and Initial Construct Validity Evidence of a Tool, myTIPreport, for Interactive Workplace Feedback on ACGME Milestones

OBJECTIVE

To describe implementation of myTIPreport for milestone feedback and to initiate construct validity testing of myTIPreport for milestones.

DESIGN

myTIPreport was used to provide workplace feedback on Accreditation Council for Graduate Medical Education required milestone sets. Performance of senior learners (postgraduate year [PGY]-4s) was compared to that of junior learners (PGY-1s) to begin the process of construct validity testing for myTIPreport.

SETTING

A convenience-based site selection of Obstetrics and Gynecology (OBGYN) residency programs.

PARTICIPANTS

OBGYN residents and faculty.

RESULTS

Amongst the 12 participating OBGYN residency programs, there were 444 unique learners and 343 unique faculty teachers. A total of 5293 milestone feedback encounters were recorded. Mean PGY-4 performance was rated higher than mean PGY-1 performance on all 25 of the compared milestone sets, with statistically significant differences seen for 19 (76%) of these 25 milestone sets and nonsignificant differences in the predicted direction observed for the other 6 milestone sets.

CONCLUSIONS

myTIPreport detected differences between senior and junior learners for the majority of compared feedback encounters for OBGYN residents. Findings support the emerging construct validity of myTIPreport for milestone feedback.

Lessons from the implementation of developmental progress assessment: A scoping review

OBJECTIVES

Educators and researchers recently implemented developmental progress assessment (DPA) in the context of competency-based education. To reap its anticipated benefits, much still remains to be understood about its implementation. In this study, we aimed to determine the nature and extent of the current evidence on DPA, in an effort to broaden our understanding of the major goals and intended outcomes of DPA as well as the lessons learned from how it has been executed in, or applied across, educational contexts.

METHODS

We conducted a scoping study based on the methodology of Arksey and O'Malley. Our search strategy yielded 2494 articles. These articles were screened for inclusion and exclusion (90% agreement), and numerical and qualitative data were extracted from 56 articles based on a pre-defined set of charting categories. The thematic analysis of the qualitative data was completed with iterative consultations and discussions until consensus was achieved for the interpretation of the results.

RESULTS

Tools used to document DPA include scales, milestones and portfolios. Performances were observed in clinical or standardised contexts. We identified seven major themes in our qualitative thematic analysis: (a) underlying aims of DPA; (b) sources of information; (c) barriers; (d) contextual factors that can act as barriers or facilitators to the implementation of DPA; (e) facilitators; (f) observed outcomes, and (g) documented validity evidences.

CONCLUSIONS

Developmental progress assessment seems to fill a need in the training of future competent health professionals. However, moving forward with a widespread implementation of DPA, factors such as lack of access to user-friendly technology and time to observe performance may render its operationalisation burdensome in the context of competency-based medical education.

Multimodal Cardiopulmonary Bypass Skills Assessment Within a High-Fidelity Simulation Environment

BACKGROUND

A high-fidelity simulator that uses a perfused porcine heart, cannulae, and tubing has been demonstrated to be a useful training adjunct. We hypothesized that multimodal assessment of cardiopulmonary bypass (CPB) skills within this high-fidelity simulated environment could discern expert from trainee performance.

METHODS

Three traditional fellows (postgraduate year 6-8) and 3 attending surgeons each performed 3 aortic cannulations. The third sequence included venous cannulation, commencement of CPB, and placement of a cardioplegia catheter and aortic cross-clamp. Performance across 20 cognitive and 21 technical domains was evaluated. Surgeon and assistant hand movements and economy of motion were assessed by electromagnetic motion sensors worn under sterile gloves.

RESULTS

Analysis showed a significant difference in cognitive (6.7 ± 2.3 vs 4.6 ± 2.7, P = .03) but not technical (6.2 ± 2.5 vs 5.8 ± 2.2, P = .7) scores favoring the experts. In addition, experts showed higher efficiency by spending 64 ± 14 seconds to construct a nonpledgeted aortic purse-string suture and secure it with a Rummel, while trainees spent 82 ± 30 seconds to complete this task (P = .03). Motion analysis revealed similar path lengths between experts and trainees for cannulation and CPB but significantly shorter path lengths for experts in cross-clamp (47.5 ± 15.5 m vs 91.9 ± 20.3 m, P = .04).

CONCLUSIONS

Multimodal assessment using cognitive, technical, and motion analysis of basic CPB tasks using a high-fidelity simulation environment is a valid system to measure performance and discriminate experts from trainees. This construct may allow for development of "competence thresholds" with important implications for training and certification in cardiothoracic surgery.

From the lens of the clinical residents: undergraduate to postgraduate medical education and the transition process

The concept of continuity in medical education reflects the progressive professional and personal development that physicians need in education. The aim of this study is investigating the views of the residents about the adequacy of undergraduate and postgraduate education in the context of preparing them for the next stage and their perceptions about the transition period. This phenomenological study was conducted at Hacettepe University Medical School. The study group consisted of medical and surgical sciences residents in the first year and last year of postgraduate medical education. Four focus group interviews were held with the participation of 21 residents. The participants emphasised that practising with real patients under supervision by taking an active role in healthcare teams was important for their preparation for the next stage in their carrier. However, their educational experiences during undergraduate medical education differed in community-based education, scientific research training, learning in small groups, internship and guidance of clinical educators. The transition period has been expressed with the concepts of identity change, high responsibilities and expectations required by the new identity, adaptation to the healthcare team, institution, and health system, meeting the expectations in an overly busy work environment, and feelings of incompetence. Participants pointed out that curriculum, which was declared and taught, educational environments, assessment approaches, consultancy systems and practices differed between the clinical departments. In line with the principles of competency-based education, practices related to the development and assessment of the competencies with all professional aspects in postgraduate medical education can be strengthened.

Best Approaches to Evaluation and Feedback in Post-Graduate Medical Education

PURPOSE OF REVIEW

The objectives of this literature review are to appraise current approaches and assess new technologies that have been utilized for evaluation and feedback of residents, with focus on surgical trainees.

RECENT FINDINGS

In 1999, the Accreditation Council for Graduate Medical Education introduced the Milestone system as a tool for summative evaluation. The organization allows individual program autonomy on how evaluation and feedback are performed. In the past, questionnaire evaluations and informal verbal feedback were employed. However, with the advent of technology, they have taken a different shape in the form of crowdsourcing, mobile platforms, and simulation. Limited data is available on new methods but studies show promise citing low cost and positive impact on resident education. No one "best approach" exists for evaluation and feedback. However, it is apparent that a multimodal approach that is based on the ACGME Milestones can be effective and aid in guiding programs.

Guided Self-Assessment and Action Plans: What Do Residents Need to Succeed?

Resident feedback and program evaluation are essential to ACGME-accredited training programs. We sought to integrate these requirements into our program by creating a systematic process for program improvement focusing on personal learning plans (PLPs). Residents completed a PLP tool every 6 months, followed by an evaluation completed with the program director. Among respondents, 96% reported the PLP process provided useful feedback. A majority found the PLP process useful in developing learning strategies and modeling lifelong learning. The integrated PLP/program improvement process serves as an effective strategy for quickly identifying and capitalizing on both individual and program opportunities for improvement.

Competency-based medical education for the clinician-educator: the coming of Milestones version 2

Competency-based medical education is emphasized by institutions overseeing medical school and postgraduate training worldwide. The high rate of preventable errors in medicine underscores this need. Expanding physician competency beyond the domains of patient care and medical knowledge towards goals that emphasize a more holistic view of the healthcare system is one aspect of this emphasis. The Accreditation Council on Graduate Medical Education (ACGME), which oversees postgraduate training programs in the USA, has recently expanded to oversee training programs internationally. The original ACGME Milestones effort unveiled in 2013 was met with skepticism. Nevertheless, other outcomes-based education programs worldwide, including the CanMEDS framework (Canada), Tomorrow's Doctor (UK), and Scottish Doctor (Scotland), have suggested that milestones do offer advantages. Missing from the milestone rollout, however, was collaborative buy-in from multiple stakeholders such as from clinician-educators. Consequently, Milestones version 2 is in development. Specifically, these will address the need for specialty-specific milestones, and the usage of harmonized milestones. A concise history of the push towards outcomes-based medical education is presented and contextualized for physicians who must embrace the transition from teacher-based to learner-based outcomes.

Information within residency monthly evaluation forms at two institutions

Periodic review of resident performance is an important aspect of residency training. Amongst allopathic residency programs, it is expected that the performance of resident physicians which can be grouped based on the ACGME core competencies, be assessed so as to allow for effective feedback and continuous improvement. Review of monthly evaluation forms for residents in the core ACGME programs at Marshall University and the University of Toledo demonstrated a wide spread in the number of Likert questions that faculty were asked to complete. This number ranged from a low of 7 in Surgery to a high of 65 in Psychiatry (both Marshall Programs). Correlation and network analysis were performed on these data. High degrees of correlations were noted between answers to questions (controlled for each resident) on these forms at both institutions. In other words, although evaluation scores varied tremendously amongst the different residents in all the programs studied, scores addressing different competencies tended to be very similar for the same resident, especially in some of the programs which were studied. Network analysis suggested that there were clusters of questions that produced essentially the same answer for a given resident, and these clusters were bigger in some of the different residency program assessment forms. This seemed to be more the rule in the residency programs with large numbers of Likert questions. The authors suggest that reducing the number of monthly questions used to address the core competencies in some programs may be possible without substantial loss of information.

Benefits and Limitations of Transurethral Resection of the Prostate Training With a Novel Virtual Reality Simulator

PURPOSE

Profound endourological skills are required for optimal postoperative outcome parameters after transurethral resection of the prostate (TURP). We investigated the Karl Storz (Tuttlingen, Germany) UroTrainer for virtual simulation training of the TURP.

MATERIALS AND METHODS

Twenty urologists underwent a virtual reality (VR) TURP training. After a needs analysis, performance scores and self-rated surgical skills were compared before and after the curriculum, the realism of the simulator was assessed, and the optimal level of experience for VR training was evaluated. Statistical testing was done with SPSS 25.

RESULTS

Forty percent of participants indicated frequent intraoperative overload during real-life TURP and 80% indicated that VR training might be beneficial for endourological skills development, underlining the need to advance classical endourological training. For the complete cohort, overall VR performance scores (P = 0.022) and completeness of resection (P < 0.001) significantly improved. Self-rated parameters including identification of anatomical structures (P = 0.046), sparing the sphincter (P = 0.002), and handling of the resectoscope (P = 0.033) became significantly better during the VR curriculum. Participants indicated progress regarding handling of the resectoscope (70%), bleeding control (55%), and finding the correct resection depth (50%). Although overall realism and handling of the resectoscope was good, virtual bleeding control and correct tissue feedback should be improved in future VR simulators. Seventy percent of participants indicated 10 to 50 virtual TURP cases to be optimal and 80% junior residents to be the key target group for VR TURP training.

CONCLUSIONS

There is a need to improve training the TURP and VR simulators might be a valuable supplement, especially for urologists beginning with the endourological desobstruction of the prostate.

Ability of Ophthalmology Residents to Self-Assess Their Performance Through Established Milestones

OBJECTIVES

Accurate self-assessment is an important aspect of practice-based learning and improvement and a critical skill for resident growth. The Accreditation Council for Graduate Medical Education mandates semiannual milestones assessments by a clinical competency committee (CCC) for all ophthalmology residents. There are six core competencies: patient care (PC), medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and interpersonal communication skills. These competencies are assessed by the milestones rubric, which has detailed behavioral anchors and are also used for trainee self-assessments. This study compares resident self-assessed (SA) and faculty CCC milestones scores.

DESIGN

Residents completed milestones self-assessments prior to receiving individual score reports from the CCC. Correlation coefficients were calculated comparing the SA and CCC scores. In addition, statistical models were used to determine predictors of disparities and differences between the SA and CCC scores.

SETTING

Wilmer Eye Institute, Johns Hopkins Hospital.

PARTICIPANTS

Twenty-one residents in the Wilmer Ophthalmology Residency program from July 2014 to June 2016.

RESULTS

Fifty-seven self-assessments were available for the analysis. For each resident's first assessment, SA and CCC scores were strongly correlated (r ≥ 0.6 and p < 0.05) for four milestones, and not correlated for the remaining 20 milestones. In multivariable models, the SA and CCC scores are less disparate for medical knowledge and systems-based practice competencies compared to practice-based learning and improvement. Higher year of training, PC and professionalism competencies were predictive of statistically significant resident overestimation of scores relative to the CCC. In addition, higher CCC scores predicted statistically significant lower SA-CCC disparities and differences. SA-CCC differences did not lower to a significant extent with repeated assessments or modification to the end-of-rotation evaluation forms.

CONCLUSIONS

Self-assessments by ophthalmology residents are not well-correlated with faculty assessments, emphasizing the need for improved and frequent timely feedback. Residents have the greatest difficulty self-assessing their professionalism and PC competency. In general, senior residents and underperforming residents have more inaccurate self-assessments.

Who moved my fellow: changes to Accreditation Council for Graduate Medical Education fellowships in pediatric surgery and what may be yet to come

PURPOSE OF REVIEW

Over the past 15 years, the Accreditation Council for Graduate Medical Education (ACGME) has significantly altered the regulatory framework governing fellowship training in pediatric surgery. The daily experiences of pediatric surgical trainees have been impacted by these changes, but training program directors and faculty have not developed a consistent approach to managing this shift. This review highlights the changes, which have occurred, analyzes the current state of fellowship training, and proposes potential strategies for management.

RECENT FINDINGS

The implementation of work hour restrictions, increased supervision requirements, the milestone evaluation program and most recently, enforcement of required critical care experience, have caused significant changes in the curriculum. Pediatric surgical trainees record more total cases, and more minimally invasive surgical (MIS) cases, in particular, than ever before. A subset of this increase may result from trainees performing cases previously assigned to general surgery residents. Teaching cases performed by fellows have decreased. Although the relationship between these shifts in training experience and the didactic curriculum is not clear, we also note that the Pediatric Surgery Certifying Examination failure rate has increased, approaching 20% in recent years.

SUMMARY

It is unclear whether the changes in Pediatric Surgery training programs have been effective, or (conversely) have led to unintended consequences. Paradigm shifts in our training model may be required to address the changes in surgical education and skill acquisition, so that well tolerated, competent and skillful pediatric surgeons continue to enter the workforce.

An Assessment of Fine Surgical KnotTying

OBJECTIVE

Operating room simulation exercises have been well established as an effective means of improving confidence, task engagement, and learning retention among surgical residents. We have established a cost-effective model and scoring system assessing resident skills to tie secure surgical knots with minimal tension.

DESIGN

A circular grid divided into 18 segments was placed underlying an aluminum can. Trainees tie 20 surgical square knots scored for time and total knot length. Movement of the can outside the grid served as a scoring penalty. Recorded were time, length of the 20 knots, and number of segments exposed at exercise end. A score was developed to identify a progression of skills with PGY level. All outcomes were compared between classes using ANOVA.

SETTING

Brown University/Rhode Island Hospital Department of Surgery.

PARTICIPANTS

Surgical residents (PGY1-PGY5) and participating attending surgeons employed by Rhode Island Hospital.

RESULTS

Knot length and exposed segments showed trends of improved scores with ascending PGY level. Only average time attained statistical significance. Overall scores improved with PGY level: Composite scores significantly improved when comparing PGY1 to PGY3, PGY5, and Attending surgeons (p = 0.016, 0.011, and 0.011, respectively). Time significantly improved when comparing PGY1 to PGY3 and Attending surgeons (77vs. 50 and 47 seconds, p = 0.019 and 0.022 respectively). Composite scores were not significantly different above PGY3.

CONCLUSIONS

A low fidelity, high impact knot tying model has been developed to assess the ability to securely tie surgical knots while minimizing tension, with linear increases in scores that appear to plateau at the PGY3 level.

Can Deficiencies in Performance Be Identified Earlier in Surgical Residency? An Initial Report of a Surgical Trainee Assessment of Readiness Exam

OBJECTIVE

Identifying gaps in medical knowledge, patient management, and procedural competence is difficult early in surgical residency. We designed and implemented an end-of-year examination for our postgraduate year 1 residents, entitled Surgical Trainee Assessment of Readiness (STAR). Our objective in this study was to determine whether STAR scores correlated with other available indicators of resident performance, such as the American Board of Surgery in-training exam (ABSITE) and Milestone scores, and if they provided evidence of additional discriminatory value.

STUDY DESIGN

Overall and component scores of the STAR exam were compared to the ABSITE and Milestone assessment scores for the 17 categorical residents that took the exam in 2016 and 2017.

SETTING

Harbor-UCLA Medical Center, a university-affiliated academic medical center.

PARTICIPANTS

Seventeen categorical general surgery residents.

RESULTS

The STAR Total Test Score (β = 2.77, p = 0.006) was an independent predictor of the ABSITE taken the same year, and components of the STAR were independent predictors of ABSITE taken the following year. The STAR Total Test Score was lowest in the 3 residents who had at least 1 low Milestone score assessed in the same year; and 2 of these 3 residents had at least 1 low Milestone score assigned the next year after STAR. Lastly, the Patient Care 1 and 2 Milestones assessed in the same year as STAR were uniformly scored as appropriate for level of training, yet the corresponding STAR component for those milestones demonstrated 3 residents as having deficiencies.

CONCLUSIONS

We have created a multifaceted standardized STAR exam, which correlates with performance on the ABSITE and early milestone scores. It also appears to discriminate resident performance where milestone assessments do not. Further evaluation of the STAR exam with longer term follow-up is needed to confirm these initial findings.

Assessment of Surgical Skills and Competency

Evaluation of surgical skills and competency are important aspects of the medical education process. Measurable and reproducible methods of assessment with objective feedback are essential components of surgical training. Objective Structured Assessment of Technical Skills (OSATS) is widely used across the medical specialties and otolaryngology-specific tools have been developed and validated for sinus and mastoid surgery. Although assessment of surgical skills can be time-consuming and requires human and financial resources, new evaluation methods and emerging technology may alleviate these barriers while also improving data collection practices.

Associate Program Directors in Surgery: A Select Group of Surgical Educators

OBJECTIVE

The role of the Associate Program Director (APD) within surgical education is not clearly defined or regulated by the Accreditation Council for Graduate Medical Education, often leading to variations in the responsibilities among institutions. Required credentials are not specified and compensation and protected time are not regulated resulting in large discrepancies among institutions. APDs are brought into the fold of surgical education to parcel out the escalating responsibilities of program director (PD). The Association of Program Directors in Surgery, Associate Program Directors Committee sent a survey to all APDs to better understand the role of the APDs within the hierarchy of surgical education.

DESIGN

A survey was sent to all 235 general surgery residency programs through the Association of Program Directors in Surgery list serve. The survey collected information on APD demographics, characteristics, and program information, qualifications of the APD, time commitment and compensation, administrative duties, and projected career track.

SETTING

General surgery residency programs within the United States.

PARTICIPANTS

108 Associate Program Directors in general surgery RESULTS: A total of 108 (46%) APDs responded to the survey. Seventy-three (70.2%) of the APD's were males. Most (77.8%) were in practice for more than 5 years, and 69% were at a university-based program. Most of the respondents felt that the administrative and curricular tasks were appropriately distributed between the APD and PD and many shared tasks with the PD. A total of 44.6% were on the path to become a future PD at their institution. An equal number of APDs (42.6%) were compensated above their base salary for being an APD vs no compensation at all; however, 16 (14.8%) had a reduced clinical load as part of their compensation for being an APD.

CONCLUSION

This is the first study to describe the characteristics of APDs within the hierarchy of surgical education. Our data demonstrate that APDs have a substantial role in the function of a residency program and they need to be developed to better define their position in the program leadership.

Thresholds and interpretations: How clinical competency committees identify pediatric residents with performance concerns

BACKGROUND

Clinical competency committee (CCC) identification of residents with performance concerns is critical for early intervention.

METHODS

Program directors and 94 CCC members at 14 pediatric residency programs responded to a written survey prompt asking them to describe how they identify residents with performance concerns. Data was analyzed using thematic analysis.

RESULTS

Six themes emerged from analysis and were grouped into two domains. The first domain included four themes, each describing a path through which residents could meet or exceed a concern threshold:1) written comments from rotation assessments are foundational in identifying residents with performance concerns, 2) concerning performance extremes stand out, 3) isolated data points may accumulate to raise concern, and 4) developmental trajectory matters. The second domain focused on how CCC members and program directors interpret data to make decisions about residents with concerns and contained 2 themes: 1) using norm- and/or criterion-referenced interpretation, and 2) assessing the quality of the data that is reviewed.

CONCLUSIONS

Identifying residents with performance concerns is important for their education and the care they provide. This study delineates strategies used by CCC members across several programs for identifying these residents, which may be helpful for other CCCs to consider in their efforts.

Resident and Program Director's Perceptions of Milestone-Based Feedback in Obstetrics and Gynecology

INTRODUCTION

In July 2014, US residency programs fully implemented the Next Accreditation System including the use of milestone evaluation and reporting. Currently, there has been little investigation into the result of implementation of this new system. Therefore, this study sought to evaluate perceptions of Obstetrics and Gynecology residents and program directors regarding the use of milestone-based feedback and identify areas of deficiency.

METHODS

A Web-based survey was sent to US-based Obstetrics and Gynecology residents and program directors regarding milestone-based assessment implementation.

RESULTS

Out of 245 program directors, 84 responded to our survey (34.3% response rate). Of responding program directors, most reported that milestone-based feedback was useful (74.7%), fair (83.0%), and accurate (76.5%); however, they found it administratively burdensome (78.1%). Residents felt that milestone-based feedback was useful (62.7%) and fair (70.0%). About 64.3% of residents and 74.7% of program directors stated that milestone-based feedback is an effective tool to track resident progression; however, a sizable minority of both groups believe that it does not capture surgical aptitude. Qualitative analysis of free response comments was largely negative and highlighted the administrative burden and lack of accuracy of milestone-based feedback.

CONCLUTION

Overall, both Obstetrics and Gynecology program directors and residents report that milestone-based feedback is useful and fair. Issues of administrative burden, timeliness, evaluation of surgical aptitude, and ability to act on assigned milestone levels were identified. Although this study is limited to one specialty, such issues are likely important to all residents, faculty, and program directors who have implemented the Next Accreditation System requirements.

Aligning In-Service Training Examinations in Plastic Surgery and Orthopaedic Surgery With Competency-Based Education

BACKGROUND

In-service training examinations (ITEs) are used to assess residents across specialties. However, it is not clear how they are integrated with the Accreditation Council for Graduate Medical Education Milestones and competencies.

OBJECTIVE

This study explored the distribution of specialty-specific milestones and competencies in ITEs for plastic surgery and orthopaedic surgery.

METHODS

In-service training examinations were publicly available for plastic surgery (PSITE) and orthopaedics (OITE). Questions on the PSITE for 2014-2016 and the OITE for 2013-2015 were mapped to the specialty-specific milestones and the 6 competencies.

RESULTS

There was an uneven distribution of milestones and competencies in ITE questions. Nine of the 36 Plastic Surgery Milestones represented 52% (341 of 650) of questions, and 3 were not included in the ITE. Of 41 Orthopaedic Surgery Milestones, 7 represented 51% (201 of 394) of questions, and 5 had no representation on the ITE. Among the competencies, patient care was the most common (PSITE = 62% [403 of 650]; OITE = 59% [233 of 394]), followed by medical knowledge (PSITE = 34% [222 of 650]; OITE = 31% [124 of 394]). Distribution of the remaining competencies differed between the 2 specialties (PSITE = 4% [25 of 650]; OITE = 9% [37 of 394]).

CONCLUSIONS

The ITEs tested slightly more than half of the milestones for the 2 specialties, and focused predominantly on patient care and medical knowledge competencies.

Surgical education: Lessons from parenthood

Although one might think surgery and parenting have little in common, there are clear parallels. Historically there has been little formal education for either role. Educators and parents relied on modelling the behavior of others, or trial and error techniques. Mentorship and role models have played a critical role in professional development and continue to have a profound impact. Over the past two decades there has been a marked increase in the resources that are available. Coaching, debriefing, deliberate practice, and formal training are now incorporated in residency programs. Specific lessons from parenthood that can be applied to surgical education include: providing a framework, learning through graduated responsibility, communicating expectations, creating a culture, setting the example, encouraging resilience, promoting autonomy, providing feedback, and navigating failure. The final lesson from parenthood: trust that you have taught them well. And you have to let them go.