"Problem residents": prevalence, problems and remediation in the era of core competencies

Performance of trauma-trained large language models on surgical assessment questions: A new approach in resource identification

BACKGROUND

Large language models have successfully navigated simulated medical board examination questions. However, whether and how language models can be used in surgical education is less understood. Our study evaluates the efficacy of domain-specific large language models in curating study materials for surgical board style questions.

METHODS

We developed EAST-GPT and ACS-GPT, custom large language models with domain-specific knowledge from published guidelines from the Eastern Association of the Surgery of Trauma and the American College of Surgeons Trauma Quality Programs. EAST-GPT, ACS-GPT, and an untrained GPT-4 performance were assessed trauma-related questions from Surgical Education and Self-Assessment Program (18th edition). Large language models were asked to choose answers and provide answer rationales. Rationales were assessed against an educational framework with 5 domains: accuracy, relevance, comprehensiveness, evidence-base, and clarity.

RESULTS

Ninety guidelines trained EAST-GPT and 10 trained ACS-GPT. All large language models were tested on 62 trauma questions. EAST-GPT correctly answered 76%, whereas ACS-GPT answered 68% correctly. Both models outperformed ChatGPT-4 (P < .05), which answered 45% correctly. For reasoning, EAST-GPT achieved the gratest mean scores across all 5 educational framework metrics. ACS-GPT scored lower than ChatGPT-4 in comprehensiveness and evidence-base; however, these differences were not statistically significant.

CONCLUSION

Our study presents a novel methodology in identifying test-preparation resources by training a large language model to answer board-style multiple choice questions. Both trained models outperformed ChatGPT-4, demonstrating its answers were accurate, relevant, and evidence-based. Potential implications of such AI integration into surgical education must be explored.

'The program director's word … it's stronger than the word of God': Epistemic injustice revealed through narratives of remediated graduate medical education residents

BACKGROUND

Though graduate medical education (GME) residency training provides positive experiences for many trainees, it may also result in major stressors and negative experiences, particularly for those requiring remediation. Residents requiring remediation may experience feelings of dismay, shame and guilt that can negatively affect their training, self-efficacy and their medical careers. Power differentials between educators and residents may set the stage for epistemic injustice, which is injustice resulting from the silencing or dismissing a speaker based on identity prejudice. This can lead to decreased willingness of trainees to engage with learning. There is a paucity of literature that explores GME experiences of remediation from the resident perspective.

OBJECTIVE

To synthesise the narratives of physician experiences of remediation during residency through the lens of epistemic injustice.

METHODS

Between January and July 2022, we interviewed US physicians who self-identified as having experienced remediation during residency. They shared events that led to remediation, personal perspectives and emotions about the process and resulting outcomes. Interviews were analysed using narrative analysis with attention to instances of epistemic injustice.

RESULTS

We interviewed 10 participants from diverse backgrounds, specialties and institutions. All participants described contextual factors that likely contributed to their remediation: (1) previous academic difficulty/nontraditional path into medicine, (2) medical disability or (3) minoritised race, gender or sexual identity. Participants felt that these backgrounds made them more vulnerable in their programmes despite attempts to express their needs. Participants reported instances of deflated credibility and epistemic injustices with important effects.

CONCLUSIONS

Participant narratives highlighted that deep power and epistemic imbalances between learners and educators can imperil GME trainees' psychological safety, resulting in instances of professional and personal harm. Our study suggests applying an existing framework to help programme directors (PDs) approach remediation with epistemic humility.

Identifying the exceptional learner in medical education: A doing vs. being framework

PURPOSE

This study aims to understand what is known about the high performing or exceptional learner in medical education. There is a rich literature about learners in difficulty, yet little is known about those performing at the high end, also known as exceptional learners.

METHODS

A qualitative study was undertaken whereby 15 faculty members at the University of Toronto were interviewed to explore their experiences with these learners.

RESULTS

Based on the findings, we developed a framework to categorize characteristics of exceptional learners by differentiating them as either 'Being' (a pre-existing attribute or set of values that the learner possesses from the start of training) or 'Doing' (demonstrable characteristics that can be observed or measured). Using this framework, we identified five characteristics in the category of 'Being', five in the category of 'Doing', and two that could be situated in either or both.

CONCLUSION

Utilizing this framework to describe exceptional learners will aid teachers in identifying them early in their training so that their training experience can be enhanced. This novel approach contributes to our knowledge of the exceptional medical learner. The optimization of the training experience will maximize the opportunity to ensure that these learners reach their full potential to contribute to the healthcare system.

Hiding in Plain Sight: Resident Remediation in Obstetrics and Gynecology

OBJECTIVE

Residents who are in need of remediation are prevalent across residency programs and often tend to be deficient in multiple competencies that the American Council for Graduate Medical Education (ACGME) has established. The purpose of this study was to determine the prevalence of residents requiring remediation, understand the scope of the challenges in resident remediation, and assess what resources were used to aid in remediation in obstetrics and gynecology programs.

DESIGN

An anonymous survey was emailed to obstetrics and gynecology program directors. Survey responses were summarized through descriptive statistics.

SETTING

Obstetrics and gynecology residency program directors were invited to respond to this survey.

PARTICIPANTS

Thirty-nine respondents out of 241 residency training programs responded (16%).

RESULTS

The majority (84.6%) of programs had placed a resident on remediation. The most common area requiring remediation was professionalism (75.8%), followed by medical knowledge (72.7%), interpersonal communication (60.6%), laparoscopic technical skills (54.6%), and inpatient care (42.4%). Residents who required remediation were identified in a number of ways, most commonly through feedback from the Clinical Competency Committee (87.8%) and faculty feedback (84.8%). Program directors utilized a variety of resources, most commonly prior remediation plans from the program, to create remediation plans. Sixty percent of programs had residents who failed remediation.

CONCLUSION

This study highlighted the prevalence of resident remediation in obstetrics and gynecology training programs and the importance of faculty in identifying residents in need of remediation, evaluating residents, and mentoring residents.

Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review update: BEME Guide No. 85

BACKGROUND

Clinical teachers often struggle to record trainee underperformance due to lacking evidence-based remediation options.

OBJECTIVES

To provide updated evidence-based recommendations for addressing academic difficulties among undergraduate and postgraduate medical learners.

METHODS

A systematic review searched databases including MEDLINE, CINAHL, EMBASE, ERIC, Education Source, and PsycINFO (2016-2021), replicating the original Best Evidence Medical Education 56 review strategy. Original research/innovation reports describing intervention(s) for medical learners with academic difficulties were included. Data extraction used Michie's Behaviour Change Techniques (BCT) Taxonomy and program evaluation models from Stufflebeam and Kirkpatrick. Quality appraised used the Mixed Methods Appraisal Tool (MMAT). Authors synthesized extracted evidence by adapting GRADE approach to formulate recommendations.

RESULTS

Eighteen articles met the inclusion criteria, primarily addressing knowledge (66.7%), skills (66.7%), attitudinal problems (50%) and learner's personal challenges (27.8%). Feedback and monitoring was the most frequently employed BCT. Study quality varied (MMAT 0-100%). We identified nineteen interventions (UG: n = 9, PG: n = 12), introducing twelve new thematic content. Newly thematic content addressed contemporary learning challenges such as academic procrastination, and use of technology-enhanced learning resources. Combined with previous interventions, the review offers a total dataset of 121 interventions.

CONCLUSION

This review offers additional evidence-based interventions for learners with academic difficulties, supporting teaching, learning, faculty development, and research efforts.

Is insufficient introspection a reason to terminate residency training? - Scrutinising introspection among residents who disputed dismissal

INTRODUCTION

Insufficient introspection as part of the 4I's model of medical professionalism (introspection, integrity, interaction, and involvement) is considered an important impediment in trainees. How insufficient introspection relates to decisions to terminate residency training remains unclear. Insights into this subject provide opportunities to improve the training of medical professionals.

METHODS

We analysed the Dutch Conciliation Board decisions regarding residents dismissed from training between 2011 and 2020. We selected the decisions on residents deemed 'insufficient' regarding introspection as part of the CanMEDS professional domain and compared their characteristics with the decisions about residents without reported insufficiencies on introspection.

RESULTS

Of the 120 decisions, 86 dismissed residents were unable to fulfil the requirements of the CanMEDS professional domain. Insufficient introspection was the most prominent insufficiency (73/86). These 73 decisions described more residents' insufficiencies in CanMEDS competency domains compared to the rest of the decisions (3.8 vs. 2.7 p < 0.001), without significant differences regarding gender or years of training.

CONCLUSIONS

Insufficient introspection in residents correlates with competency shortcomings programme directors reported in dismissal disputes. The 4I's model facilitates recognition and description of unprofessional behaviours, opening avenues for assessing and developing residents' introspection, but further research is needed for effective implementation in medical education.

Unprofessional behaviour of GP residents leading to a dismissal dispute: characteristics and outcomes of those who appeal

BACKGROUND

Recognition of poor performance in General Practice trainees is important because underperformance compromises patients' health and safety. However, in General Practice, research on persistent underperformance while in training and its ultimate consequences is almost completely lacking. We aim to explore the unprofessional behaviours of residents in General Practice who were dismissed from training and who litigated against dismissal.

METHODS

We performed a structured analysis using open-source data from all General Practice cases before the Conciliation Board of the Royal Dutch Medical Association between 2011 and 2020. Anonymised law cases about residents from all Dutch GP training programmes were analysed in terms of the quantitative and qualitative aspects related to performance.

RESULTS

Between 2011 and 2020, 24 residents who were dismissed from training challenged their programme director's decision. Dismissed residents performed poorly in several competencies, including communication, medical expertise and most prominently, professionalism. Over 90% of dismissed residents failed on professionalism. Most lacked self-awareness and/or failed to profit from feedback. Approximately 80% failed on communication, and about 60% on medical expertise as well. A large majority (more than 80%) of dismissed residents had previously participated in some form of remediation.

CONCLUSIONS

Deficiencies in both professionalism and communication were the most prevalent findings among the dismissed General Practice residents. These two deficiencies overlapped considerably. Dismissed residents who challenged their programme director's decision were considered to lack self-awareness, which requires introspection and the appreciation of feedback from others.

Dutch dismissal practices: characteristics, consequences, and contrasts in residents' case law in community-based practice versus hospital-based specialties

BACKGROUND

In the Netherlands, 2 to 10% of the residents terminate training prematurely. Infrequently, termination of training is by dismissal. Incidentally, residents may disagree, dispute and challenge these decisions from the programme directors. Resident dismissal is always a difficult decision, most commonly made after, repeated assessments, and triangulation of the resulting assessment data and one or more remediation attempts. Nevertheless, the underlying reasons for dismissal and the policies for remediation and dismissal may differ between training programmes. Such differences may however impact the chance of remediation success, the chance of dismissal and subsequent residents' appeals.

METHOD

We included a total of 70 residents from two groups (community-based and hospital-based specialties) during 10 years of appeals. Subsequently, we compared these groups on factors potentially associated with the outcome of the conciliation board decision regarding the residents' dismissal. We focused herein on remediation strategies applied, and reasons reported to dismiss residents.

RESULTS

In both groups, the most alleged reason to dismiss residents was lack of trainability, > 97%. This was related to deficiencies in professionalism in community-based practice and medical expertise in hospital-based specialties respectively. A reason less frequently mentioned was endangerment of patient care, < 26%. However, none of these residents accused of endangerment, actually jeopardized the patients' health, probably due to the vigilance of their supervisors. Remediation strategies varied between the two groups, whereas hospital-based specialties preferred formal remediation plans in contrast to community-based practice. A multitude of remediation strategies per competency (medical expertise, professionalism, communication, management) were applied and described in these law cases.

DISCUSSION

Residents' appeals in community-based practice were significantly less likely to succeed compared to hospital-based specialties. Hypothesised explanatory factors underlying these differences include community-based practices' more prominent attention to the longitudinal assessment of professionalism, the presence of regular quarterly progress meetings, precise documentation of deficiencies, and discretion over the timing of dismissal in contrast to dismissal in the hospital-based specialties which is only formally possible during scheduled formal summative assessment meetings.

Do deficiencies in CanMEDS competencies of dismissed residents differ according to specialty?

INTRODUCTION

Program directors dismiss a small percentage of residents from residency training programs, presumably due to underperformance or lack of progress. Whether underperformance in competency domains differs by residents' specialty is unknown.

METHODS

In 2021, we analysed the case law of Dutch residents who were dismissed from training by the program director, and who challenged this dismissal before the national conciliation board between 2011 and 2020. Across medical specialties we compared which of the CanMEDS competency domains these residents failed to meet.

RESULTS

We found 116 cases of residents dismissed from their training programmes who challenged the decision of the program director before the board. In general, most residents were unable to meet the requirements of several CanMEDS competency domains (usually: medical expert, communicator, and professional). In surgery, all dismissed residents failed to meet the competency domain of the medical expert, while most of the dismissed psychiatry residents met this domain. In specialties with a primarily diagnostic task, more dismissed residents failed to meet the competency domain of the scholar, while dismissed general medicine residents (for example family medicine and nursing homecare) were less likely to do so. Residents in general medicine, more often than other specialties, however, failed to meet the competency domain of the professional.

CONCLUSION

Residents dismissed from training, who challenged their dismissal, failed to meet the requirements of multiple CanMEDS competency domains. Competency domain failures differ by specialty.

Changing the Landscape of Remediation: The Creation and Implementation of an Institution-Wide Graduate Medical Education Performance Enhancement Program

Purpose Remediation is a daunting process for both residency leadership and trainees due to several factors including limited time and resources, variable processes, and negative stigma. Our objective was to transform the remediation process by creating a transparent institution-wide program that collates tools/resources, interdepartmental faculty mentors, and positive rebranding. Methods Education leadership across seven specialties created a process for trainees with professionalism and interpersonal-communication skills deficiencies. Formalized departmental program-based improvement plan (PIP) and an institutional house staff performance enhancement plan (HPEP) were developed by consensus of triggers/behaviors. Utilizing published literature, a toolkit was created and implemented. Trainees were enrolled in HPEP if PIP was unsuccessful or exhibited ≥1 major trigger. Wellness evaluations were incorporated into the process to screen for external contributing factors. Surveys were sent to the program director (PD), faculty mentor, and trainee one month and six months after participation. Results Between 2018 and 2021, 12 trainees were enrolled. Overall feedback from PDs and the trainees was positive. The main challenge was finding mutual time for the faculty mentor and trainee to meet. Six-month surveys reported no relapses in unprofessionalism. One-year follow-up of the trainees was limited. Conclusions Utilizing an institution-wide standardized process of performance improvement with the removal of negative stereotyping is a unique approach to remediation. Initial feedback is promising, and future outcome data are necessary to assess the utility. The HPEP may be adopted by other academic institutions and may shift the attitudes about remediation and allow trainees to see the process as an opportunity for professional growth.

Impact of Satisfactory/Unsatisfactory Grading on Student Motivation to Learn, Academic Performance, and Well-Being

Satisfactory/unsatisfactory (S/U) grading is often proposed to ameliorate stress by reducing the competitive nature of letter grading. Though explored considerably in human medical programs, minimal literature focuses on the veterinary school setting. The purpose of this study was to evaluate the impact of S/U grading on veterinary students' motivation to learn, academic performance, and well-being. Cornell University's COVID-19 pandemic response provided a unique opportunity to compare S/U and letter grading on the same population of students during a single pre-clinical foundation course, with the first half being graded S/U (spring semester 2020), returning to letter grades in the second half (fall semester 2020). Students were retroactively surveyed on the effect of S/U vs. letter grading on their overall educational experience and well-being, with 67.8% class participation. The majority of respondents (71.3%) stated that S/U grading had a positive impact on their overall learning experience. More than half (53.8%) perceived that they learned the same amount of information and had the same level of motivation (58.8%), even though most (61.3%) stated that they spent less time preparing for S/U assessments than letter grade assessments. Positive impact factor effects for S/U grading included decreased stress, more time for self-care, improved learning, and increased learning enjoyment. S/U grading did not negatively impact academic performance. In conclusion, this study demonstrates that, in our particular study population and setting, S/U grading conferred well-being and learning experience advantages to students without any reduction in motivation for learning or academic performance.

REACT: Rapid Evaluation Assessment of Clinical Reasoning Tool

INTRODUCTION

Clinical reasoning encompasses the process of data collection, synthesis, and interpretation to generate a working diagnosis and make management decisions. Situated cognition theory suggests that knowledge is relative to contextual factors, and clinical reasoning in urgent situations is framed by pressure of consequential, time-sensitive decision-making for diagnosis and management. These unique aspects of urgent clinical care may limit the effectiveness of traditional tools to assess, teach, and remediate clinical reasoning.

METHODS

Using two validated frameworks, a multidisciplinary group of clinicians trained to remediate clinical reasoning and with experience in urgent clinical care encounters designed the novel Rapid Evaluation Assessment of Clinical Reasoning Tool (REACT). REACT is a behaviorally anchored assessment tool scoring five domains used to provide formative feedback to learners evaluating patients during urgent clinical situations. A pilot study was performed to assess fourth-year medical students during simulated urgent clinical scenarios. Learners were scored using REACT by a separate, multidisciplinary group of clinician educators with no additional training in the clinical reasoning process. REACT scores were analyzed for internal consistency across raters and observations.

RESULTS

Overall internal consistency for the 41 patient simulations as measured by Cronbach's alpha was 0.86. A weighted kappa statistic was used to assess the overall score inter-rater reliability. Moderate reliability was observed at 0.56.

DISCUSSION

To our knowledge, REACT is the first tool designed specifically for formative assessment of a learner's clinical reasoning performance during simulated urgent clinical situations. With evidence of reliability and content validity, this tool guides feedback to learners during high-risk urgent clinical scenarios, with the goal of reducing diagnostic and management errors to limit patient harm.

Post graduate remediation programs in medicine: a scoping review

BACKGROUND

Recognizing that physicians may struggle to achieve knowledge, skills, attitudes and or conduct at one or more stages during their training has highlighted the importance of the 'deliberate practice of improving performance through practising beyond one's comfort level under guidance'. However, variations in physician, program, contextual and healthcare and educational systems complicate efforts to create a consistent approach to remediation. Balancing the inevitable disparities in approaches and settings with the need for continuity and effective oversight of the remediation process, as well as the context and population specific nature of remediation, this review will scrutinise the remediation of physicians in training to better guide the design, structuring and oversight of new remediation programs.

METHODS

Krishna's Systematic Evidence Based Approach is adopted to guide this Systematic Scoping Review (SSR in SEBA) to enhance the transparency and reproducibility of this review. A structured search for articles on remediation programs for licenced physicians who have completed their pre-registration postings and who are in training positions published between 1st January 1990 and 31st December 2021 in PubMed, Scopus, ERIC, Google Scholar, PsycINFO, ASSIA, HMIC, DARE and Web of Science databases was carried out. The included articles were concurrently thematically and content analysed using SEBA's Split Approach. Similarities in the identified themes and categories were combined in the Jigsaw Perspective and compared with the tabulated summaries of included articles in the Funnelling Process to create the domains that will guide discussions.

RESULTS

The research team retrieved 5512 abstracts, reviewed 304 full-text articles and included 101 articles. The domains identified were characteristics, indications, frameworks, domains, enablers and barriers and unique features of remediation in licenced physicians in training programs.

CONCLUSION

Building upon our findings and guided by Hauer et al. approach to remediation and Taylor and Hamdy's Multi-theories Model, we proffer a theoretically grounded 7-stage evidence-based remediation framework to enhance understanding of remediation in licenced physicians in training programs. We believe this framework can guide program design and reframe remediation's role as an integral part of training programs and a source of support and professional, academic, research, interprofessional and personal development.

Characterization of Clinical Skills Remediation: A National Survey of Medical Schools

OBJECTIVES

Clinical skills instruction is a standard part of medical school curricula, but how institutions address learners who struggle in this area is less clear. Although recommendations for the remediation of clinical skills at an institutional level have been published, how these recommendations are being implemented on a national scale is unknown. In this descriptive study, we characterize current clinical skills remediation practices at US medical schools and US-accredited Caribbean medical schools.

METHODS

We conducted a cross-sectional survey of medical educators who work with struggling students. From March 24, 2020 to April 9, 2020, the Directors of Clinical Skills Remediation Working Group conducted an e-mail survey incorporating four aspects of remediation program design and function: identification, assessment, active remediation, and ongoing evaluation.

RESULTS

In total, 92 individuals representing 45 institutions provided descriptive information about their respective remediation programs. The majority of respondents have a formal process of identifying (75%) and assessing (86%) students who are identified as struggling with clinical skills, but lack a standardized method of categorizing deficits. Fewer institutions have a standardized approach to active remediation and ongoing evaluation of struggling learners. Fifty-two percent of institutions provide training to faculty involved in the remediation process.

CONCLUSIONS

Although most institutions are able to identify struggling students, they lack a standardized approach to intervene. Remediation effectiveness is limited by a lack of student buy-in and institutional time, expertise, and resources. These findings highlight the need for more formalized structure and standardization in remediation program design and implementation.

What Behaviors Define a Good Physician? Assessing and Communicating About Noncognitive Skills

Once medical students attain a certain level of medical knowledge, success in residency often depends on noncognitive attributes, such as conscientiousness, empathy, and grit. These traits are significantly more difficult to assess than cognitive performance, creating a potential gap in measurement. Despite its promise, competency-based medical education (CBME) has yet to bridge this gap, partly due to a lack of well-defined noncognitive observable behaviors that assessors and educators can use in formative and summative assessment. As a result, typical undergraduate to graduate medical education handovers stress standardized test scores, and program directors trust little of the remaining information they receive, sometimes turning to third-party companies to better describe potential residency candidates. The authors have created a list of noncognitive attributes, with associated definitions and noncognitive skills-called observable practice activities (OPAs)-written for learners across the continuum to help educators collect assessment data that can be turned into valuable information. OPAs are discrete work-based assessment elements collected over time and mapped to larger structures, such as milestones, entrustable professional activities, or competencies, to create learning trajectories for formative and summative decisions. Medical schools and graduate medical education programs could adapt these OPAs or determine ways to create new ones specific to their own contexts. Once OPAs are created, programs will have to find effective ways to assess them, interpret the data, determine consequence validity, and communicate information to learners and institutions. The authors discuss the need for culture change surrounding assessment-even for the adoption of behavior-based tools such as OPAs-including grounding the work in a growth mindset and the broad underpinnings of CBME. Ultimately, improving assessment of noncognitive capacity should benefit learners, schools, programs, and most importantly, patients.

Metrics of Resident Achievement for Defining Program Aims

Introduction

Resident achievement data is a powerful but underutilized means of program evaluation, allowing programs to empirically measure whether they are meeting their program aims, facilitate refinement of curricula and improve resident recruitment efforts. The goal was to provide an overview of available metrics of resident achievement and how these metrics can be used to inform program aims.

Methods

A literature search was performed using PubMed and Google Scholar between May and November of 2020. Publications were eligible for inclusion if they discussed or assessed “excellence” or “success” during residency training. A narrative review structure was chosen due to the intention to provide an examination of the literature on available resident achievement metrics.

Results

57 publications met inclusion criteria and were included in the review. Metrics of excellence were grouped into larger categories, including success defined by program factors, academics, national competencies, employer factors, and possible new metrics.

Conclusions

Programs can best evaluate whether they are meeting their program aims by creating a list of important resident-level metrics based on their stated goals and values using one or more of the published definitions as a foundation. Each program must define which metrics align best with their individual program aims and mission.

Where Do International Medical Graduates Matriculate for Internal Medicine Training? A National Longitudinal Study

INTRODUCTION

In 2020, roughly 25% of applicants who matched into internal medicine (IM) residencies were international medical graduates (IMGs). We examine 12-year trends in distribution of IMGs among IM training programs and explore differences in program perceptions towards IMG recruitment.

METHODS

Since 2007, Association of Program Directors in Internal Medicine Annual Surveys have collected data about trainees by medical school graduate type. Sixteen additional questions regarding perceptions of IMGs were included in the 2017 spring survey.

RESULTS

The 2017 survey response rate was 63.3% (236/373) and ranged from 61.9 to 70.2% for the 2007-2019 Annual Surveys. During that 12-year period, 55-70% of community programs' and 22-30% of university programs' PGY1 positions were filled by IMGs. In 2017, 45% of community programs' and 15% of university programs' interview and ranking positions were allocated to IMGs. Departmental pressure (university 45.6% [95% CI 43.7-47.5]; community 28.2% [95% CI 26.6-29.7]; p = 0.007), institutional priority (university 64.0% [95% CI 62.1-66.0]; community 41% [95% CI 36.9-44.6]; p = 0.001), and reputational concerns (university 52.8% [95% CI 50.0-55.6]; community 38.5% [95% CI 36.0-40.9]; p = 0.045) were cited as factors influencing recruitment of IMGs.

CONCLUSION

Our study was limited to exploring program factors in residency recruitment and did not assess applicant preferences. There is a large, longstanding difference in the recruitment of IMGs to US community-based and university residencies, beginning during the interview and ranking process. Further research in disparities in IMG recruitment is needed, including exploring pressures, preferences, and potential biases associated with the recruitment of IMGs.

Unheard Voices: A Qualitative Study of Resident Perspectives on Remediation

BACKGROUND

Remediation is an important component of residency training that ensures residents are progressing toward competency and unsupervised practice. There is literature describing educators' attitudes about remediation; however, little is known about residents' perspectives regarding peers who are struggling and remediation. Understanding this perspective is critical to supporting struggling residents and developing successful remediation programs.

OBJECTIVE

The objective of this study was to describe residents' perspectives on peers who are struggling and remediation processes within graduate medical education programs.

METHODS

In 2015, we conducted focus groups of residents in a multi-institutional exploratory qualitative study designed to investigate resident perspectives on remediation. Focus groups included questions on identification of residents who are struggling, reasons residents face difficulty in training, attitudes toward remediation, and understanding of the remediation process. Using conventional content analysis, we analyzed the focus group data to discover common themes.

RESULTS

Eight focus groups were performed at 3 geographically distinct institutions. A total of 68 residents participated, representing 12 distinct medical specialties. Four major themes emerged from the participants' discussion: lack of transparency, negative stigma, overwhelming emotions, and a need for change.

CONCLUSIONS

Resident perspectives on remediation are affected by communication, culture, and emotions. The resident participants called for change, seeking greater understanding and transparency about what it means to struggle and the process of remediation. The residents also believed that remediation can be embraced and normalized.

Management of residents in difficulty in a Swiss general internal medicine outpatient clinic: Change is necessary!

AIMS OF THE STUDY

Residents in difficulty are a major cause for concern in medical education, with a prevalence of 7-15%. They are often detected late in their training and cannot make use of remediation plans. Nowadays, most training hospitals in Switzerland do not have a specific program to identify and manage residents in difficulty. The aim of the study was to explore the challenges perceived by physicians regarding the process of identifying, diagnosing, and supporting residents in difficulty in a structured and programmatic way. We explored perceptions of physicians at different hierarchical levels (residents (R), Chief residents (CR), attending physicians (A), Chief Physician (CP)) in order to better understand these challenges.

METHODS

We conducted an exploratory qualitative study between December 2015 and July 2016. We asked volunteers from the Primary Care Division of the Geneva University Hospitals to partake to three focus groups (with CR, A, R) and one interview with the division's CP. We transcribed, coded, and qualitatively analyzed the three focus groups and the interview, using a content thematic approach and Fishbein's conceptual framework.

RESULTS

We identified similarities and differences in the challenges of the management of residents in difficulty on a programmatic way amongst physicians of different hierarchical levels. Our main findings: Supervisors (CR, A, CP) have good identification skills of residents in difficulty, but they did not put in place systematic remediation strategies.Supervisors (CR, A) were concerned about managing residents in difficulty. They were aware of the possible adverse effects on patient care, but "feared to harm" resident's career by documenting a poor institutional assessment.Residents "feared to share" their own difficulties with their supervisors. They thought that it would impact their career negatively.The four physician's hierarchical level reported environmental constraints (lack of funding, time constraint, lack of time and resources…).

CONCLUSION

Our results add two perspectives to specialized recommendations regarding the implementation of remediation programs for residents in difficulty. The first revolves around the need to identify and fully understand not only the beliefs but also the implicit norms and the feeling of self-efficacy that are shared by teachers and that are likely to motivate them to engage in the management of residents in difficulty. The second emphasizes the importance of analyzing these elements that constitute the context for a change and of identifying, in close contact with the heads of the institutions, which factors may favor or hinder it. This research action process has fostered awareness and discussions at different levels. Since then, various actions and processes have been put in place at the Faculty of Medicine in Geneva.

Growth mindset in competency-based medical education

The ongoing adoption of competency-based medical education (CBME) across health professions training draws focus to learner-centred educational design and the importance of fostering a growth mindset in learners, teachers, and educational programs. An emerging body of literature addresses the instructional practices and features of learning environments that foster the skills and strategies necessary for trainees to be partners in their own learning and progression to competence and to develop skills for lifelong learning. Aligned with this emerging area is an interest in Dweck's self theory and the concept of the growth mindset. The growth mindset is an implicit belief held by an individual that intelligence and abilities are changeable, rather than fixed and immutable. In this paper, we present an overview of the growth mindset and how it aligns with the goals of CBME. We describe the challenges associated with shifting away from the fixed mindset of most traditional medical education assumptions and practices and discuss potential solutions and strategies at the individual, relational, and systems levels. Finally, we present future directions for research to better understand the growth mindset in the context of CBME.

Managing the ambiguity of the trainee and the trainer

The “trainee in difficulty” (TID) can have multiple causative factors which can impact the delivery of an expected standard of skill sets. The communication and interpersonal skills in Emergency Department (ED) setting are key to any trainee’s performance in an ED environment where team playing is a major factor in achieving safe and holistic care for patients. Trainer or a young faculty member responsible for the training may not have the emotional intelligence or experience to deal with all the issues faced by the TID. This paper talks about the difficulties faced by an experienced trainee in difficulty, who has changed his career from an experienced ED nurse to a trainee registrar in Emergency Medicine. The second case study is about a young emergency medicine residency program director who fails to appropriately address a trainee’s situation and compounds the trainees’ issues. The effect of honest, transparent communication of an educational supervisor and setting clear goals for the TID can have a huge impact on trainees’ performance. A residency program director’s inexperience and poor skills to deal and escalate the trainee’s issues may jeopardize a young physicians’ career.

Student Perceptions of educational handovers

BACKGROUND

Educational handovers can provide competency information about graduating medical students to residency program directors post-residency placement. Little is known about students' comfort with this novel communication.

OBJECTIVE

To examine graduated medical students' perceptions of educational handovers.

METHODS

The authors created and distributed an anonymous survey to 166 medical students at a single institution following graduation in the spring of 2018. Within this cohort, 40 students had an educational handover sent to their future program director. The survey explored comfort level with handovers (1=very uncomfortable; 5=very comfortable) and ideal content (e.g., student strengths, areas for improvement, goals, grades received after residency application). Respondents self-reported their performance in medical school and whether a handover was sent. Correlation analyses examined relationships between performance and other variables. T-tests examined differences between students who did and did not have a handover letter sent.

RESULTS

The survey response rate was 40.4% (67/166) - 47.8% of students felt comfortable with handovers, 19.4% were neutral, and 32.8% were uncomfortable. There was no correlation between self-reported medical school performance and comfort level. Respondents felt most strongly that strengths should be included, followed by goals. Those who had a handover letter sent expressed significantly higher comfort level (3.8 ± 1.0 vs. 2.6 ±1.3, p=0.003) with this communication.

CONCLUSION

Medical students reported varying levels of comfort with educational handovers; however, those who had handovers sent had more positive perceptions. In order to improve the education continuum, it is essential to engage students in the development of this handover communication.

The "Difficult Learner" in anesthesiology: Challenges, pitfalls, and recommendations

Struggling learners often require interventions that are time-consuming and emotionally exhausting for both the trainee and faculty. Numerous barriers, including lack of resources, faculty development, and fear of legal retribution, can impede medical educators from developing and implementing robust remediation plans. Despite the large volume of literature citing professionalism education and the "hidden curriculum" as problem areas in medical education, frontline educators lack practical tools and empowerment to address unprofessionalism in trainees. The paucity of resources in this arena has led to decreased job satisfaction, increased burnout, and an exodus from academic medicine. Department leadership acknowledgment and investment in training faculty on remediation strategies for learners in difficulty and providing tools to meet these challenging job demands will improve faculty's job satisfaction and overall well-being. The authors review salient literature and methodology for diagnosing learners in difficulty, with focus on a high-yield, pragmatic approach that can be taken by medical training programs, including those that lack a robust medical education infrastructure.

Mental Health Conditions Among Struggling GME Learners: Results From a Single Center Remediation Program

BACKGROUND

Graduate medical education (GME) learners may struggle with clinical performance during training. A subset of these trainees has mental health conditions (MHCs).

OBJECTIVE

To characterize the MHCs that underlie poor trainee performance and their relationship to specific clinical performance deficit (CPD).

METHODS

At the University of Virginia (UVA), GME learners not meeting appropriate milestones, or who request help, have the option to self-refer or be referred to COACH (Committee on Achieving Competence Through Help). A physician remediation expert assesses the learner and identifies a primary CPD. If there is concern for an MHC, referral is made to a psychologist with expertise in working with trainees. All learners are offered remediation for the CPD. Using descriptive statistics, we tracked the prevalence of MHC and their correlation with specific CPDs.

RESULTS

Between 2016 and 2019, COACH assessed 7% (61 of 820) of GME learners at UVA. Thirty-eight percent (23 of 61) had an MHC associated with the CPD. Anxiety was the most common MHC (48%), followed by depression (17%), cognitive dysfunction (17%), adjustment disorder (13%), and other (4%). Professionalism was the most identified CPD among learners with MHCs (52%). Of remediated learners, 47% have successfully finished remediation, 21% were terminated or voluntarily left their program, and 32% are still being remediated (83% of whom are in good standing).

CONCLUSIONS

MHCs were identified in nearly 40% of struggling learners referred to a centralized remediation program. Professionalism is the most identified CPD among learners with MHCs.

Seeing but not believing: Insights into the intractability of failure to fail

CONTEXT

Inadequate documentation of observed trainee incompetence persists despite research-informed solutions targeting this failure to fail phenomenon. Documentation could be impeded if assessment language is misaligned with how supervisors conceptualise incompetence. Because frameworks tend to itemise competence as well as being vague about incompetence, assessment design may be improved by better understanding and describing of how supervisors experience being confronted with a potentially incompetent trainee.

METHODS

Following constructivist grounded theory methodology, analysis using a constant comparison approach was iterative and informed data collection. We interviewed 22 physicians about their experiences supervising trainees who demonstrate incompetence; we quickly found that they bristled at the term 'incompetence,' so we began to use 'underperformance' in its place.

RESULTS

Physicians began with a belief and an expectation: all trainees should be capable of learning and progressing by applying what they learn to subsequent clinical experiences. Underperformance was therefore unexpected and evoked disbelief in supervisors, who sought alternate explanations for the surprising evidence. Supervisors conceptualised underperformance as: an inability to engage with learning due to illness, a life event or learning disorders, so that progression was stalled, or an unwillingness to engage with learning due to lack of interest, insight or humility.

CONCLUSION

Physicians conceptualise underperformance as problematic progression due to insufficient engagement with learning that is unresponsive to intensified supervision. Although failure to fail tends to be framed as a reluctance to document underperformance, the prior phase of disbelief prevents confident documentation of performance and delays identification of underperformance. The findings offer further insight and possible new solutions to address under-documentation of underperformance.

Amant L, Glover Takahashi S. Managing residents in difficulty within CBME residency educational systems: a scoping review

BACKGROUND

Best practices in managing residents in difficulty (RID) in the era of competency-based medical education (CBME) are not well described. This scoping review aimed to inventory the current literature and identify major themes in the articles that address or employ CBME as part of the identification and remediation of residents in difficulty.

METHODS

Articles published between 2011 to 2017 were included if they were about postgraduate medical education, RID, and offered information to inform the structure and/or processes of CBME. All three reviewers performed a primary screening, followed by a secondary screening of abstracts of the chosen articles, and then a final comprehensive sub-analysis of the 11 articles identified as using a CBME framework.

RESULTS

Of 165 articles initially identified, 92 qualified for secondary screening; the 63 remaining articles underwent full-text abstracting. Ten themes were identified from the content analysis with "identification of RID" (41%) and "defining and classifying deficiencies" (30%) being the most frequent. In the CBME article sub-analysis, the most frequent themes were: need to identify RID (64%), improving assessment tools (45%), and roles and responsibilities of players involved in remediation (27%). Almost half of the CBME articles were published in 2016-2017.

CONCLUSIONS

Although CBME programs have been implemented for many years, articles have only recently begun specifically addressing RID within a competency framework. Much work is needed to describe the sequenced progression, tailored learning experiences, and competency-focused instruction. Finally, future research should focus on the outcomes of remediation in CBME programs.

What Does It Mean for a Case to be 'Local'?: the Importance of Local Relevance and Resonance for Bioethics Education in the Asia-Pacific Region

Contemporary bioethics education has been developed predominately within Euro-American contexts, and now, other global regions are increasingly joining the field, leading to a richer global understanding. Nevertheless, many standard bioethics curriculum materials retain a narrow geographic focus. The purpose of this article is to use local cases from the Asia-Pacific region as examples for exploring questions such as 'what makes a case or example truly local, and why?', 'what topics have we found to be best explained through local cases or examples?', and 'how does one identify a relevant local case?' Furthermore, we consider the global application of local cases to help extend the possible scope of the discussion, opening new avenues for the development of practical bioethics educational materials. We begin with a background description and discussion of why local cases enhance bioethics education, move to an overview of what is currently available and what is not for the region, and then outline a discussion of what it means to be local using example cases drawn from Hong Kong, Australia, Pakistan, and Malaysia. We are not creating a casebook but rather constructing by example a toolbox for designing active and dynamic learning cases using regional diversity as contextualised cases with generalised principles.

The AAMC Standardized Video Interview and the Electronic Standardized Letter of Evaluation in Emergency Medicine: A Comparison of Performance Characteristics

PURPOSE

To compare the performance characteristics of the electronic Standardized Letter of Evaluation (eSLOE), a widely used structured assessment of emergency medicine (EM) residency applicants, and the Association of American Medical Colleges (AAMC) Standardized Video Interview (SVI), a new tool designed by the AAMC to assess interpersonal and communication skills and professionalism knowledge.

METHOD

The authors matched EM residency applicants with valid SVI total scores and completed eSLOEs in the 2018 Match application cycle. They examined correlations and group differences for both tools, United States Medical Licensing Examination (USMLE) Step exam scores, and honor society memberships.

RESULTS

The matched sample included 2,884 applicants. SVI score and eSLOE global assessment ratings demonstrated small positive correlations approaching r = 0.20. eSLOE ratings had higher correlations with measures of academic ability (USMLE scores, academic honor society membership) than did SVI scores. Group differences were minimal for the SVI, with scores slightly favoring women (d = -0.21) and US-MD applicants (d = 0.23-0.42). Group differences in eSLOE ratings were small, favoring women over men (approaching d = -0.20) and white applicants over black applicants (approaching d = 0.40).

CONCLUSIONS

Small positive correlations between SVI score and eSLOE global assessment ratings, alongside varying correlations with academic ability indicators, suggest these are complementary tools. Findings suggest the eSLOE is subject to similar sources and degrees of bias as other common assessments; these group differences were not observed with the SVI. Further examination of both tools is necessary to understand their ability to predict clinical performance.

Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review: BEME Guide No. 56

Background: Clinical teachers often struggle to report unsatisfactory trainee performance, partly because of a lack of evidence-based remediation options. Objectives: To identify interventions for undergraduate (UG) and postgraduate (PG) medical learners experiencing academic difficulties, link them to a theory-based framework and provide literature-based recommendations around their use. Methods: This systematic review searched MEDLINE, CINAHL, EMBASE, ERIC, Education Source and PsycINFO (1990-2016) combining these concepts: medical education, professional competence/difficulty and educational support. Original research/innovation reports describing intervention(s) for UG/PG medical learners with academic difficulties were included. Data extraction employed Michie's Behavior Change Techniques (BCT) Taxonomy and program evaluation models from Stufflebeam and Kirkpatrick. Quality appraisal used the Mixed Methods Appraisal Tool (MMAT). The authors synthesized extracted evidence by adapting the GRADE approach to formulate recommendations. Results: Sixty-eight articles met the inclusion criteria, most commonly addressing knowledge (66.2%), skills (53.9%) and attitudinal problems (26.2%), or learner personal issues (41.5%). The most common BCTs were Shaping knowledge, Feedback/monitoring, and Repetition/substitution. Quality appraisal was variable (MMAT 0-100%). A thematic content analysis identified 109 interventions (UG: n = 84, PG: n = 58), providing 24 strong, 48 moderate, 26 weak and 11 very weak recommendations. Conclusion: This review provides a repertoire of literature-based interventions for teaching/learning, faculty development, and research purposes.

Comparison of the Standardized Video Interview and Interview Assessments of Professionalism and Interpersonal Communication Skills in Emergency Medicine

OBJECTIVES

The Association of American Medical Colleges Standardized Video Interview (SVI) was recently added as a component of emergency medicine (EM) residency applications to provide additional information about interpersonal communication skills (ICS) and knowledge of professionalism (PROF) behaviors. Our objective was to ascertain the correlation between the SVI and residency interviewer assessments of PROF and ICS. Secondary objectives included examination of 1) inter- and intrainstitutional assessments of ICS and PROF, 2) correlation of SVI scores with rank order list (ROL) positions, and 3) the potential influence of gender on interview day assessments.

METHODS

We conducted an observational study using prospectively collected data from seven EM residency programs during 2017 and 2018 using a standardized instrument. Correlations between interview day PROF/ICS scores and the SVI were tested. A one-way analysis of variance was used to analyze the association of SVI and ROL position. Gender differences were assessed with independent-groups t-tests.

RESULTS

A total of 1,264 interview-day encounters from 773 unique applicants resulted in 4,854 interviews conducted by 151 interviewers. Both PROF and ICS demonstrated a small positive correlation with the SVI score (r = 0.16 and r = 0.17, respectively). ROL position was associated with SVI score (p < 0.001), with mean SVI scores for top-, middle-, and bottom-third applicants being 20.9, 20.5, and 19.8, respectively. No group differences with gender were identified on assessments of PROF or ICS.

CONCLUSIONS

Interview assessments of PROF and ICS have a small, positive correlation with SVI scores. These residency selection tools may be measuring related, but not redundant, applicant characteristics. We did not identify gender differences in interview assessments.

An Emergency Medicine Remediation Consult Service: Access to Expert Remediation Advice and Resources

Resident remediation is a complex and common issue in emergency medicine programs and requires a specific knowledge base. The Remediation Task Force (RTF) of the Council of Residency Directors in Emergency Medicine (CORD-EM) was created to identify remediation best practices and to develop tools for program directors. Initially housed on a Wiki page, and now located within the CORD-EM website, the RTF provides resources including accepted universal language for documentation and sample remediation plans. The RTF also created a remediation consult service composed of experienced educators to provide real-time structured feedback and advice to submitted remediation scenarios with consultation outcomes and conclusions uploaded to the website. CORD-EM members now have easy access to online resources and expert advice for remediation queries through the consult service. The combination of online resources and access to real-time expert advice is an innovative approach to improving resident remediation and recognizing best practices.

Development of Evaluations to Assess the Quality of Residency Program Directors

OBJECTIVE

The objective was to develop residency program director specific evaluation tools with face validity for identifying areas of excellence and opportunities for improvement.

METHODS

Investigators conducted standardized interviews with graduate medical education (GME) leadership, department chairs, program directors, and senior residents assessing their perspectives on the desirable qualities of program directors. Responses were categorized by theme to create benchmarked evaluation tools.

RESULTS

Residents and faculty emphasized communication, mentorship, and role modeling. Faculty emphasized maintenance of accreditation and resident discipline, while residents emphasized wellness, approachability, and maintenance of the learning environment. The investigators then developed a 10-item assessment for residents and a similar 12-item assessment for department chairs and GME leadership with face validity.

CONCLUSION

Although the responsibilities of residency training directors are numerous and variable by specialty, the investigators were able to develop valid program director evaluation tool templates that can highlight the successes and opportunities for improvement of these academicians.

Defining the "Problem Resident" and the Implications of the Unfixable Problem: The Rationale for a "Front-door" Solution

Introduction

Problem residents are common in graduate medical education, yet little is known about their characteristics, deficits, and the consequences for emergency medicine (EM) residencies. The American Board of Internal Medicine (ABIM) defines a problem resident as “a trainee who demonstrates a significant enough problem that requires intervention by someone of authority, usually the program director [PD] or chief resident.” Although this is a comprehensive definition, it lacks specificity. Our study seeks to add granularity and nuance to the definition of “problem resident,” which can be used to guide the recruitment, selection, and training of residents.

Methods

We conducted semi-structured interviews with a convenience sample of EM PDs between 2011 and 2012. We performed qualitative analysis of the resulting transcripts with our thematic analysis based on the principles of grounded theory. We reached thematic sufficiency after 17 interviews. Interviews were coded as a team through consensus.

Results

The analysis identified diversity in the type, severity, fixability, and attribution of problems among problem residents. PDs applied a variety of thresholds to define a problem resident with many directly rejecting the ABIM definition. There was consistency in defining academic problems and some medical problems as “fixable.” In contrast, personality problems were consistently defined as “non-fixable.” Despite the diversity of the definition, there was consensus that residents who caused “turbulence” were problem residents.

Conclusion

The ABIM definition of the problem resident captures trainees who many PDs do not consider problem residents. We propose that an alternative definition of the problem resident would be “a resident with a negative sphere of influence beyond their personal struggle.” This combination acknowledges the identified themes of turbulence and the diversity of threshold. Further, the combination of PDs’ unwillingness to terminate trainees and the presence of non-fixable problems implies the need for a “front-door” solution that emphasizes personality issues at the potential expense of academic potential. This “front-door” solution depends on the commitment of all stakeholders including medical schools, the Association of American Medical Colleges, and PDs.

Pediatric Program Leadership's Contribution Toward Resident Wellness

BACKGROUND

Residency program leaders are required to support resident well-being, but often they do not receive training in how to do so.

OBJECTIVE

To determine frequency in which program leadership provides support for resident well-being, comfort in supporting resident well-being, and factors associated with need for additional training in supporting resident well-being.

METHODS

National cross-sectional web-based survey in June 2015 of pediatric program directors, associate program directors, and coordinators about their experiences supporting resident well-being. Univariate and bivariate descriptive statistics compared responses between groups. Generalized linear modeling, adjusting for program region, size, program leadership role, and number of years in role determined factors associated with need for additional training.

RESULTS

The response rate was 39.3% (322/820). Most respondents strongly agreed that supporting resident well-being is an important part of their role, but few reported supporting resident well-being as part of their job description. Most reported supporting residents' clinical, personal, and health issues at least annually, and in some cases weekly, with 72% spending >10%of their time on resident well-being. Most program leaders desired more training. After adjusting for level of comfort in dealing with resident well-being issues, program leaders more frequently exposed to resident well-being issues were more likely to desire additional training (P < .02).

CONCLUSIONS

Program leaders spend a significant amount of time supporting resident well-being. Although they think that supporting resident well-being is an important part of their job, opportunities exist for developing program leaders through including resident wellness on job descriptions and training program leaders how to support resident well-being.

Simple Frameworks for Daily Work: Innovative Strategies to Coach Residents Struggling With Time Management, Organization, and Efficiency

BACKGROUND

Organization and efficiency are central to success on busy inpatient services and may be relevant to demonstrating certain milestones. Most residents adopt these skills by observing supervisors and peers. For some, this method of emulation and adaptation does not occur, with the potential for a negative effect on patient care and team morale. Information on effective strategies for remediating organization and efficiency deficits is lacking.

OBJECTIVE

We explored the major themes of organization and efficiency referred to the University of Pennsylvania Department of Medicine Early Intervention and Remediation Committee (EIRC), and developed tools for their remediation.

METHODS

Assessments of residents and fellows referred to the EIRC between July 2014 and October 2016 were reviewed for organization and efficiency deficits. Common areas were identified, and an iterative process of learner observations and expert input was used to develop remediation tools.

RESULTS

Over a 2-year period, the EIRC developed remediation plans for 4% of residents (13 of 342 total residents), and for 1 internal medicine subspecialty fellow. Organization and efficiency was the primary or secondary deficit in more than half of those assessed. Most common deficiencies involved admitting a patient efficiently, performing effective prerounding, and composing daily progress notes/presentations. Remediation tools that provided deconstruction of tasks to their most granular and reproducible components were effective in improving performance.

CONCLUSIONS

Deficits in organization and efficiency can disproportionately affect resident performance and delay milestone achievement. Many residents would benefit from detailed frameworks and assistance with new approaches to basic elements of daily work.

Criterion-Based Assessment in a Norm-Based World: How Can We Move Past Grades?

In the United States, the medical education community has begun a shift from the Flexnerian time-based model to a competency-based medical education model. The graduate medical education (GME) community is substantially farther along in this transition than is the undergraduate medical education (UME) community.GME has largely adopted the use of competencies and their attendant milestones and increasingly is employing the framework of entrustable professional activities (EPAs) to assess trainee competence. The UME community faces several challenges to successfully navigating a similar transition. First is the reliance on norm-based reference standards in the UME-GME transition, comparing students' performance versus their peers' with grades, United States Medical Licensing Examination Step 1 and Step 2 score interpretation, and the structured Medical School Performance Evaluation, or dean's letter. Second is the reliance on proxy assessments rather than direct observation of learners. Third is the emphasis on summative rather than formative assessments.Educators have overcome a major barrier to change by establishing UME outcomes assessment criteria with the advent and general acceptance of the physician competency reference set and the Core EPAs for Entering Residency in UME. Now is the time for the hard work of developing assessments steeped in direct observation that can be accepted by learners and faculty across the educational continuum and can be shown to predict clinical performance in a much more meaningful way than the current measures of grades and examinations. The acceptance of such assessments will facilitate the UME transition toward competency-based medical education.

Comprehensive Assessment of Struggling Learners Referred to a Graduate Medical Education Remediation Program

BACKGROUND

Implementation of the Next Accreditation System has provided a standardized framework for identifying learners not meeting milestones, but there is as yet no corresponding framework for remediation.

OBJECTIVE

We developed a comprehensive assessment process that allows correct diagnosis of a struggling learner's deficit(s) to promote successful remediation.

METHODS

At the University of Pennsylvania, resident learners within the Department of Medicine who are not meeting milestones are referred to the Early Intervention Remediation Committee (EIRC). The EIRC, composed of 14 faculty members with expertise in remediation, uses a standardized process to assess learners' deficits. These faculty members categorize primary deficits as follows: medical knowledge, clinical reasoning, organization and efficiency, professionalism, and communication skills. The standardized process of assessment includes an analysis of the learner's file, direct communication with evaluators, an interview focused on learner perception of the problem, screening for underlying medical or psychosocial issues, and a review of systems for deficits in the 6 core competencies. Participants were surveyed after participating in this process.

RESULTS

Over a 2-year period, the EIRC assessed and developed remediation plans for 4% of learners (14 of a total 342). Following remediation and reassessment, the identified problems were satisfactorily resolved in all cases with no disciplinary action. While the process was time intensive, an average of 45 hours per learner, the majority of faculty and residents rated it as positive and beneficial.

CONCLUSIONS

This structured assessment process identifies targeted areas for remediation and adds to the tools available to Clinical Competency Committees.

Effect of Performance Deficiencies on Graduation and Board Certification Rates: A 10-yr Multicenter Study of Anesthesiology Residents

BACKGROUND

This multicenter, retrospective study was conducted to determine how resident performance deficiencies affect graduation and board certification.

METHODS

Primary documents pertaining to resident performance were examined over a 10-yr period at four academic anesthesiology residencies. Residents entering training between 2000 and 2009 were included, with follow-up through February 2016. Residents receiving actions by the programs' Clinical Competency Committee were categorized by the area of deficiency and compared to peers without deficiencies.

RESULTS

A total of 865 residents were studied (range: 127 to 275 per program). Of these, 215 residents received a total of 405 actions from their respective Clinical Competency Committee. Among those who received an action compared to those who did not, the proportion graduating differed (93 vs. 99%, respectively, P < 0.001), as did the proportion achieving board certification (89 vs. 99%, respectively, P < 0.001). When a single deficiency in an Essential Attribute (e.g., ethical, honest, respectful behavior; absence of impairment) was identified, the proportion graduating dropped to 55%. When more than three Accreditation Council for Graduate Medical Education Core Competencies were deficient, the proportion graduating also dropped significantly.

CONCLUSIONS

Overall graduation and board certification rates were consistently high in residents with no, or isolated, deficiencies. Residents deficient in an Essential Attribute, or multiple competencies, are at high risk of not graduating or achieving board certification. More research is needed on the effectiveness and selective deployment of remediation efforts, particularly for high-risk groups.

Defining Uniform Processes for Remediation, Probation and Termination in Residency Training

It is important that residency programs identify trainees who progress appropriately, as well as identify residents who fail to achieve educational milestones as expected so they may be remediated. The process of remediation varies greatly across training programs, due in part to the lack of standardized definitions for good standing, remediation, probation, and termination. The purpose of this educational advancement is to propose a clear remediation framework including definitions, management processes, documentation expectations and appropriate notifications. Informal remediation is initiated when a resident's performance is deficient in one or more of the outcomes-based milestones established by the Accreditation Council for Graduate Medical Education, but not significant enough to trigger formal remediation. Formal remediation occurs when deficiencies are significant enough to warrant formal documentation because informal remediation failed or because issues are substantial. The process includes documentation in the resident's file and notification of the graduate medical education office; however, the documentation is not disclosed if the resident successfully remediates. Probation is initiated when a resident is unsuccessful in meeting the terms of formal remediation or if initial problems are significant enough to warrant immediate probation. The process is similar to formal remediation but also includes documentation extending to the final verification of training and employment letters. Termination involves other stakeholders and occurs when a resident is unsuccessful in meeting the terms of probation or if initial problems are significant enough to warrant immediate termination.

Describing Failure in a Clinical Clerkship: Implications for Identification, Assessment and Remediation for Struggling Learners

BACKGROUND

In addition to training future members of the profession, medical schools perform the critical role of identifying students who are failing to meet minimum standards in core competencies.

OBJECTIVE

To better understand reasons for failure in an internal medicine clerkship.

DESIGN

A qualitative content analysis of letters describing reasons for students' failure.

PARTICIPANTS

Forty-three students (31 men) who failed the internal medicine clerkship at the University of Minnesota Medical School, 2002-2013.

APPROACH

We conducted a qualitative content analysis of the 43 letters describing reasons for students' failure. We coded critical deficiencies and mapped them to the Physician Competency Reference Set (PCRS) competency domains and classified them into two categories: conduct (unprofessional behaviors) and knowledge and skills specific to the practice of medicine. We then calculated the frequency of each critical deficiency. We statistically tested for relationships between gender and critical deficiencies in each of the competency domains.

KEY RESULTS

We coded 50 critical deficiencies with all codes mapping to a PCRS competency domain. The most frequently cited deficiencies were "insufficient knowledge" (79 % of students) and "inadequate patient presentation skills" (74 %). Students exhibited critical deficiencies in all eight competency domains, with the highest concentrations in Knowledge for Practice (98 %) and Interpersonal and Communication Skills (91 %). All students demonstrated deficiencies in multiple competencies, with 98 % having deficiencies in three or more. All 43 students demonstrated deficits in the knowledge and skills category, and 81 % had concurrent conduct issues. There were no statistically significant relationships between gender and critical deficiencies in any competency domain.

CONCLUSIONS

This study highlights both the diversity and commonality of reasons that students fail a clinical clerkship. Knowing the range of areas where students struggle, as well as the most likely areas of difficulty, may aid faculty in identifying students who are failing and in developing remediation strategies.