Resident evaluation and remediation: a comprehensive approach

Design, Implementation, and Outcomes of an Interprofessional Mobile Web Application for Preceptors for Challenging Issues

OBJECTIVE

The purpose of this study was to develop and assess an easily accessible interprofessional mobile web application to assist preceptors with challenging teaching and learning situations.

METHODS

Phase 1 was a modified Delphi process of 48 advanced practice nursing, dentistry, medicine, and pharmacy preceptors to determine the content of the application. Phase 2 consisted of 12 preceptors from the 4 disciplines piloting a prototype to refine the tool using design-thinking principles. Feedback was analyzed using inductive coding and thematic analysis. Phase 3 evaluated the impact of the final tool on 80 preceptors' satisfaction, knowledge, self-efficacy, and perception of behavior change.

RESULTS

Consensus on 10 topics was reached in the following 3 themes: feedback and communication, clinical and professional development of learners, and precepting efficiency. Preceptors rated the tool as efficient and applicable. Features perceived as useful included concise and applicable content that was easy to navigate with practical video examples. Features to improve included academic jargon, length of content, and lack of connectivity with other preceptors. Knowledge and self-efficacy improved after the use of the refined tool. Change in perceptions of behavior after 1 month was mixed, with a significant change in accessing resources to address challenging situations and regularly reflecting on challenging situations and no significant change in awareness, frequency, or success in managing challenging situations.

CONCLUSION

An interprofessional mobile web application for challenging teaching and learning situations developed through a modified Delphi process was deemed efficient and relevant and demonstrated positive knowledge and self-efficacy change.

Post graduate remediation programs in medicine: a scoping review

BACKGROUND

Recognizing that physicians may struggle to achieve knowledge, skills, attitudes and or conduct at one or more stages during their training has highlighted the importance of the 'deliberate practice of improving performance through practising beyond one's comfort level under guidance'. However, variations in physician, program, contextual and healthcare and educational systems complicate efforts to create a consistent approach to remediation. Balancing the inevitable disparities in approaches and settings with the need for continuity and effective oversight of the remediation process, as well as the context and population specific nature of remediation, this review will scrutinise the remediation of physicians in training to better guide the design, structuring and oversight of new remediation programs.

METHODS

Krishna's Systematic Evidence Based Approach is adopted to guide this Systematic Scoping Review (SSR in SEBA) to enhance the transparency and reproducibility of this review. A structured search for articles on remediation programs for licenced physicians who have completed their pre-registration postings and who are in training positions published between 1st January 1990 and 31st December 2021 in PubMed, Scopus, ERIC, Google Scholar, PsycINFO, ASSIA, HMIC, DARE and Web of Science databases was carried out. The included articles were concurrently thematically and content analysed using SEBA's Split Approach. Similarities in the identified themes and categories were combined in the Jigsaw Perspective and compared with the tabulated summaries of included articles in the Funnelling Process to create the domains that will guide discussions.

RESULTS

The research team retrieved 5512 abstracts, reviewed 304 full-text articles and included 101 articles. The domains identified were characteristics, indications, frameworks, domains, enablers and barriers and unique features of remediation in licenced physicians in training programs.

CONCLUSION

Building upon our findings and guided by Hauer et al. approach to remediation and Taylor and Hamdy's Multi-theories Model, we proffer a theoretically grounded 7-stage evidence-based remediation framework to enhance understanding of remediation in licenced physicians in training programs. We believe this framework can guide program design and reframe remediation's role as an integral part of training programs and a source of support and professional, academic, research, interprofessional and personal development.

Interactive Narrative Simulation as a Method for Preceptor Development

(1) Background: This proof-of-concept study assessed an interactive web-based tool simulating three challenging non-academic learning situations—student professionalism, cross-cultural interactions, and student well-being—as a means of preceptor development. (2) Methods: Three scripts focused on professionalism, cross-cultural interactions, and student well-being were developed and implemented using a commercial narrative tool with branching dialog. Delivered online, this tool presented each challenge to participants. Participants had up to four response options at each turn of the conversation; the choice of response influenced the subsequent conversation, including coaching provided at the resolution of the situation. Participants were invited to complete pre-activity, immediate post-activity, and one-month follow-up questionnaires to assess satisfaction, self-efficacy, engagement, and knowledge change with the tool. Knowledge was assessed through situational judgment tests (SJTs). (3) Results: Thirty-two pharmacist preceptors participated. The frequency of participants reflecting on challenging learning situations increased significantly one-month post-simulation. Participants affirmatively responded that the tool was time-efficient, represented similar challenges they encountered in precepting, was easily navigable, and resulted in learning. Self-efficacy with skills in managing challenging learning situations increased significantly immediately post-simulation and at a one-month follow-up. Knowledge as measured through SJTs was not significantly changed. (4) Conclusions: Preceptors found an interactive narrative simulation a relevant, time-efficient approach for preceptor development for challenging non-academic learning situations. Post-simulation, preceptors more frequently reflected on challenging learning situations, implying behavior change. Self-efficacy and self-report of knowledge increased. Future research is needed regarding knowledge assessments.

Emotion in remediation: A scoping review of the medical education literature

OBJECTIVES

Remediation can be crucial and high stakes for medical learners, and experts agree it is often not optimally conducted. Research from other fields indicates that explicit incorporation of emotion improves education because of emotion's documented impacts on learning. Because this could present an important opportunity for improving remediation, we aimed to investigate how the literature on remediation interventions in medical education discusses emotion.

METHODS

The authors used Arksey and O'Malley's framework to conduct a scoping literature review of records describing remediation interventions in medical education, using PubMed, CINAHL Complete, ERIC, Web of Science and APA PsycInfo databases, including all English-language publications through 1 May 2020 meeting search criteria. They included publications discussing remediation interventions either empirically or theoretically, pertaining to physicians or physician trainees of any level. Two independent reviewers used a standardised data extraction form to report descriptive information; they reviewed included records for the presence of mentions of emotion, described the mentions and analysed results thematically.

RESULTS

Of 1644 records, 199 met inclusion criteria and were reviewed in full. Of those, 112 (56%) mentioned emotion in some way; others focused solely on cognitive aspects of remediation. The mentions of emotion fell into three themes based on when the emotion was cited as present: during regular coursework or practice, upon referral for remediation and during remediation. One-quarter of records (50) indicated potential intentional incorporation of emotion into remediation programme design, but they were non-specific as to how emotions related to the learning process itself.

CONCLUSION

Even though emotion is omnipresent in remediation, medical educators frequently do not factor emotion into the design of remediation approaches and rarely explicitly utilise emotion to improve the learning process. Applications from other fields may help medical educators leverage emotion to improve learning in remediation, including strategies to frame and design remediation.

Optimising the delivery of remediation programmes for doctors: A realist review

CONTEXT

Medical underperformance puts patient safety at risk. Remediation, the process that seeks to 'remedy' underperformance and return a doctor to safe practice, is therefore a crucially important area of medical education. However, although remediation is used in health care systems globally, there is limited evidence for the particular models or strategies employed. The purpose of this study was to conduct a realist review to ascertain why, how, in what contexts, for whom and to what extent remediation programmes for practising doctors work to restore patient safety.

METHOD

We conducted a realist literature review consistent with RAMESES standards. We developed a programme theory of remediation by carrying out a systematic search of the literature and through regular engagement with a stakeholder group. We searched bibliographic databases (MEDLINE, EMBASE, PsycINFO, HMIC, CINAHL, ERIC, ASSIA and DARE) and conducted purposive supplementary searches. Relevant sections of text relating to the programme theory were extracted and synthesised using a realist logic of analysis to identify context-mechanism-outcome configurations (CMOcs).

RESULTS

A 141 records were included. The majority of the studies were from North America (64%). 29 CMOcs were identified. Remediation programmes are effective when a doctor's insight and motivation are developed and behaviour change reinforced. Insight can be developed by providing safe spaces, using advocacy to promote trust and framing feedback sensitively. Motivation can be enhanced by involving the doctor in remediation planning, correcting causal attribution, goal setting and destigmatising remediation. Sustained change can be achieved by practising new behaviours and skills, and through guided reflection.

CONCLUSION

Remediation can work when it creates environments that trigger behaviour change mechanisms. Our evidence synthesis provides detailed recommendations on tailoring implementation and design strategies to improve remediation interventions for doctors.

Remediation programmes for practising doctors to restore patient safety: the RESTORE realist review

Background

An underperforming doctor puts patient safety at risk. Remediation is an intervention intended to address underperformance and return a doctor to safe practice. Used in health-care systems all over the world, it has clear implications for both patient safety and doctor retention in the workforce. However, there is limited evidence underpinning remediation programmes, particularly a lack of knowledge as to why and how a remedial intervention may work to change a doctor’s practice.

Objectives

To (1) conduct a realist review of the literature to ascertain why, how, in what contexts, for whom and to what extent remediation programmes for practising doctors work to restore patient safety; and (2) provide recommendations on tailoring, implementation and design strategies to improve remediation interventions for doctors.

Design

A realist review of the literature underpinned by the Realist And MEta-narrative Evidence Syntheses: Evolving Standards quality and reporting standards.

Data sources

Searches of bibliographic databases were conducted in June 2018 using the following databases: EMBASE, MEDLINE, Cumulative Index to Nursing and Allied Health Literature, PsycINFO, Education Resources Information Center, Database of Abstracts of Reviews of Effects, Applied Social Sciences Index and Abstracts, and Health Management Information Consortium. Grey literature searches were conducted in June 2019 using the following: Google Scholar (Google Inc., Mountain View, CA, USA), OpenGrey, NHS England, North Grey Literature Collection, National Institute for Health and Care Excellence Evidence, Electronic Theses Online Service, Health Systems Evidence and Turning Research into Practice. Further relevant studies were identified via backward citation searching, searching the libraries of the core research team and through a stakeholder group.

Review methods

Realist review is a theory-orientated and explanatory approach to the synthesis of evidence that seeks to develop programme theories about how an intervention produces its effects. We developed a programme theory of remediation by convening a stakeholder group and undertaking a systematic search of the literature. We included all studies in the English language on the remediation of practising doctors, all study designs, all health-care settings and all outcome measures. We extracted relevant sections of text relating to the programme theory. Extracted data were then synthesised using a realist logic of analysis to identify context–mechanism–outcome configurations.

Results

A total of 141 records were included. Of the 141 studies included in the review, 64% related to North America and 14% were from the UK. The majority of studies (72%) were published between 2008 and 2018. A total of 33% of articles were commentaries, 30% were research papers, 25% were case studies and 12% were other types of articles. Among the research papers, 64% were quantitative, 19% were literature reviews, 14% were qualitative and 3% were mixed methods. A total of 40% of the articles were about junior doctors/residents, 31% were about practicing physicians, 17% were about a mixture of both (with some including medical students) and 12% were not applicable. A total of 40% of studies focused on remediating all areas of clinical practice, including medical knowledge, clinical skills and professionalism. A total of 27% of studies focused on professionalism only, 19% focused on knowledge and/or clinical skills and 14% did not specify. A total of 32% of studies described a remediation intervention, 16% outlined strategies for designing remediation programmes, 11% outlined remediation models and 41% were not applicable. Twenty-nine context–mechanism–outcome configurations were identified. Remediation programmes work when they develop doctors’ insight and motivation, and reinforce behaviour change. Strategies such as providing safe spaces, using advocacy to develop trust in the remediation process and carefully framing feedback create contexts in which psychological safety and professional dissonance lead to the development of insight. Involving the remediating doctor in remediation planning can provide a perceived sense of control in the process and this, alongside correcting causal attribution, goal-setting, destigmatising remediation and clarity of consequences, helps motivate doctors to change. Sustained change may be facilitated by practising new behaviours and skills and through guided reflection.

Limitations

Limitations were the low quality of included literature and limited number of UK-based studies.

Future work

Future work should use the recommendations to optimise the delivery of existing remediation programmes for doctors in the NHS.

Study registration

This study is registered as PROSPERO CRD42018088779.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 9, No. 11. See the NIHR Journals Library website for further project information.

Developing a process to manage challenges encountered by experiential education administrators

INTRODUCTION

This study explored preceptor and student related issues reported in pharmacy experiential education settings and solutions that experiential education administrators (EEAs) applied to inform a process to manage these challenges.

METHODS

This mixed-methods study was conducted in two phases. In phase one, five EEAs from three schools of pharmacy collected quantitative and qualitative data over a two-year period on issues reported by students and preceptors and the solutions EEAs employed. Descriptive statistics and content analysis were used to analyze quantitative and qualitative data, respectively. Interviews with six EEAs from four schools were completed in phase two. An additional school was added in phase two. Open coding, pattern coding, and summative analysis were completed on interview data.

RESULTS

Preceptors and students reported 156 and 65 issues, respectively. The most common issues reported were student reliability/responsibility (22, 14%) and preceptor communication (13, 20%). The most frequently applied solutions were coach preceptor (51, 24%) and meet with student (20, 24%). Interviews revealed more information about the strategies EEAs used. Approaches included reactive responses such as talking to and coaching preceptors or students. Preventative strategies were desired, such as tracking data and training.

CONCLUSIONS

Results informed two approaches that EEAs can employ to resolve student and preceptor issues. In the proactive process, issues are tracked to inform student and preceptor development. The reactive approach involves meeting with the preceptor or student, gathering information from both parties, coaching the preceptor or student, and involving other administrators as needed for serious concerns.

Remediation for surgical trainees: recommendations from a narrative review

BACKGROUND

Remediation involves formalized support for surgical trainees with significant underperformance to return to expected standards. There is a need to understand current evidence of remediation for surgical trainees to inform practice and justify investment of resources.

METHODS

Following the principles of a systematic review, we conducted a narrative analysis to make recommendations for remediation of underperforming surgical trainees.

RESULTS

From a review of 55 articles on remediation of trainees in medical and surgical sub-specialities, we have identified system and process level recommendations. Remediation is reported as long-term, complex and resource-intensive. Establishing a defined and standardized remediation framework enables co-ordination of multi-modal interventions. System level recommendations aim to consolidate protocols via developing better assessment, intervention and re-evaluation modalities whilst also strengthening support to supervisors conducting the remediation. Process level recommendations should be tailored for the specific needs of each trainee, aiming to be proactive with interventions within a programmatic framework. Regular reassessment is required, and long-term follow-up shows that remediation efforts are often successful.

CONCLUSION

While remediation within a programmatic framework is complex, it is often a successful approach to return surgical trainees to their expected standard. Future directions involve applying learning theories, encouraging research methods and to develop integrated collaborate protocols and support to synergize efforts.

The Value of Numerical USMLE Step 1 Scores in Radiology Resident Selection

RATIONALE AND OBJECTIVES

In response to a recent proposal to change scoring on the United States Medical Licensing (USMLE) Step 1 exam to pass/fail, we sought to determine the value of numerical Step 1 scores in predicting success in our radiology residency program.

MATERIALS AND METHODS

Residency applications for 157 residents entering the program between 2005 and 2017 were retrospectively reviewed. Biographical (gender, sports participation, advanced degree), undergraduate (school, major), and medical school (grades, USMLE Step 1 score, Alpha Omega Alpha membership, letters of recommendation, publications) data were recorded. Multivariate regression analysis was used to examine the relationship between these application factors and subsequent performance as a radiology resident, as determined by completion of the program without requiring corrective action, select Accreditation Council for Graduate Medical Education milestones, and selection as chief resident.

RESULTS

Corrective action was required for 7% (n = 12) of residents. Of the predictor variables, only Step 1 score was associated with the need for corrective action (p < 0.001). The interpretation of exams milestone was associated with higher Step 1 scores (p = 0.001) and number of medical school clerkship honors (p = 0.008). Selection as chief resident was associated with sports participation (p = 0.04), and clerkship honors (p = 0.02).

CONCLUSION

Numerical USMLE Step 1 scores are predictive of successful completion of radiology residency training without the need for corrective action, and of accelerated competence in the interpretation of exams milestone. Continued reporting of numerical Step 1 scores would be valuable in selection of radiology residents.

Remediating professionalism lapses in medical students and doctors: A systematic review

CONTEXT

A remediation intervention aims to facilitate the improvement of an individual whose competence in a particular skill has dropped below the level expected. Little is known regarding the effectiveness of remediation, especially in the area of professionalism. This review sought to identify and assess the effectiveness of interventions to remediate professionalism lapses in medical students and doctors.

METHODS

Databases Embase, MEDLINE, Education Resources Information Center and the British Education Index were searched in September 2017 and October 2018. Studies reporting interventions to remediate professionalism lapses in medical students and doctors were included. A standardised data extraction form incorporating a previously described behaviour change technique taxonomy was utilised. A narrative synthesis approach was adopted. Quality was assessed using the Critical Appraisal Skills Programme checklist.

RESULTS

A total of 19 studies on remediation interventions reported in 23 articles were identified. Of these, 13 were case studies, five were cohort studies and one was a qualitative study; 37% targeted doctors, 26% medical students, 16% residents and 21% involved mixed populations. Most interventions were multifaceted and addressed professionalism issues concomitantly with clinical skills, but some focused on specific areas (eg sexual boundaries and disruptive behaviours). Most used three or more behaviour change techniques. The included studies were predominantly of low quality as 13 of the 19 were case studies. It was difficult to assess the effectiveness of the interventions as the majority of studies did not carry out any evaluation.

CONCLUSIONS

The review identifies a paucity of evidence to guide best practice in the remediation of professionalism lapses in medical students and doctors. The literature tentatively suggests that remediating lapses in professionalism, as part of a wider programme of remediation, can facilitate participants to graduate from a programme of study, and pass medical licensing and mock oral board examinations. However, it is not clear from this literature whether these interventions are successful in remediating lapses in professionalism specifically. Further research is required to improve the design and evaluation of interventions to remediate professionalism lapses.

Standardized Letters of Concern and Remediation Contracts: Templates for Program Directors

BACKGROUND

Remediation of the struggling resident is a universal phenomenon, and the majority of program directors will remediate at least 1 resident during their tenure.

OBJECTIVE

The goal of this project was to create a standardized template for program directors to use at all stages of remediation.

METHODS

Between 2017 and 2018, the Council of Residency Directors in Emergency Medicine (CORD-EM) Remediation Committee searched for best practices in the medical literature and compiled a survey that was e-mailed to the CORD-EM listserv. After reviewing all information, a standardized remediation contract was created, reviewed by legal counsel, and distributed to members.

RESULTS

Forty-two percent (110 of 263) of program directors or assistant program directors on the CORD-EM listserv answered the initial survey and provided guidance on current remediation practices. The committee created formal and informal standard remediation contracts as both fillable templates and alterable documents. These were reviewed by CORD-EM general legal counsel and approved by the CORD-EM Board of Directors for distribution. The project took approximately 20 hours to complete over 8 months and involved a cost of $480 for legal fees.

CONCLUSIONS

With program director input and legal counsel review, the CORD-EM Remediation Committee produced standardized remediation contracts, which can be used by all emergency medicine programs after comparison to local institutional policy and local legal review. This process was feasible and can be replicated by other specialties.

An Emergency Medicine Remediation Consult Service: Access to Expert Remediation Advice and Resources

Resident remediation is a complex and common issue in emergency medicine programs and requires a specific knowledge base. The Remediation Task Force (RTF) of the Council of Residency Directors in Emergency Medicine (CORD-EM) was created to identify remediation best practices and to develop tools for program directors. Initially housed on a Wiki page, and now located within the CORD-EM website, the RTF provides resources including accepted universal language for documentation and sample remediation plans. The RTF also created a remediation consult service composed of experienced educators to provide real-time structured feedback and advice to submitted remediation scenarios with consultation outcomes and conclusions uploaded to the website. CORD-EM members now have easy access to online resources and expert advice for remediation queries through the consult service. The combination of online resources and access to real-time expert advice is an innovative approach to improving resident remediation and recognizing best practices.

Defining the "Problem Resident" and the Implications of the Unfixable Problem: The Rationale for a "Front-door" Solution

Introduction

Problem residents are common in graduate medical education, yet little is known about their characteristics, deficits, and the consequences for emergency medicine (EM) residencies. The American Board of Internal Medicine (ABIM) defines a problem resident as “a trainee who demonstrates a significant enough problem that requires intervention by someone of authority, usually the program director [PD] or chief resident.” Although this is a comprehensive definition, it lacks specificity. Our study seeks to add granularity and nuance to the definition of “problem resident,” which can be used to guide the recruitment, selection, and training of residents.

Methods

We conducted semi-structured interviews with a convenience sample of EM PDs between 2011 and 2012. We performed qualitative analysis of the resulting transcripts with our thematic analysis based on the principles of grounded theory. We reached thematic sufficiency after 17 interviews. Interviews were coded as a team through consensus.

Results

The analysis identified diversity in the type, severity, fixability, and attribution of problems among problem residents. PDs applied a variety of thresholds to define a problem resident with many directly rejecting the ABIM definition. There was consistency in defining academic problems and some medical problems as “fixable.” In contrast, personality problems were consistently defined as “non-fixable.” Despite the diversity of the definition, there was consensus that residents who caused “turbulence” were problem residents.

Conclusion

The ABIM definition of the problem resident captures trainees who many PDs do not consider problem residents. We propose that an alternative definition of the problem resident would be “a resident with a negative sphere of influence beyond their personal struggle.” This combination acknowledges the identified themes of turbulence and the diversity of threshold. Further, the combination of PDs’ unwillingness to terminate trainees and the presence of non-fixable problems implies the need for a “front-door” solution that emphasizes personality issues at the potential expense of academic potential. This “front-door” solution depends on the commitment of all stakeholders including medical schools, the Association of American Medical Colleges, and PDs.

Programmatic Assessment in Emergency Medicine: Implementation of Best Practices

BACKGROUND

Programmatic assessment is the intentional collection of key data from multiple sources for both assessment of learning and assessment for learning.

OBJECTIVE

We developed a system of programmatic assessment (PA) to identify competency progression (summative) and assessment for learning to assist residents in their formative development.

METHODS

The programmatic assessment was designed iteratively from 2014 through 2016. All assessments were first categorized by competency domain and source of assessment. The number of assessment modalities for each competency domain was collected. These multisource assessments were then mapped by program leadership to the milestones to develop a master PA blueprint. A resident learning management system provided the platform for aggregating formative and summative data, allowing residents and faculty ongoing access to guide learning and assessment. A key component of programmatic assessment was to support resident integration of assessment information through feedback by faculty after shifts and during monthly formal assessments, semiannual resident reviews, and summative judgments by the Clinical Competency Committee.

RESULTS

Through the PA, the 6 competency domains are assessed through multiple modalities: patient care (22 different assessments), professionalism (18), systems-based practice (17), interprofessional and communication skills (16), medical knowledge (11), and practice-based learning and improvement (6). Each assessment provides feedback to the resident in various formats. Our programmatic assessment has been utilized for more than 2 years with iterative improvements.

CONCLUSIONS

The implementation of programmatic assessment allowed our program to organize diverse, multisourced feedback to drive both formative and summative assessments.

Critical Deficiency Ratings in Milestone Assessment: A Review and Case Study

PURPOSE

The Accreditation Council for Graduate Medical Education (ACGME) requires programs to report learner progress using specialty-specific milestones. It is unclear how milestones can best identify critical deficiencies (CDs) in trainee performance. Specialties developed milestones independently of one another; not every specialty included CDs within milestones ratings. This study examined the proportion of ACGME milestone sets that include CD ratings, and describes one residency program's experiences using CD ratings in assessment.

METHOD

The authors reviewed ACGME milestones for all 99 specialties in November 2015, determining which rating scales contained CDs. The authors also reviewed three years of data (July 2012-June 2015) from the University of Cincinnati Medical Center (UCMC) internal medicine residency assessment system based on observable practice activities mapped to ACGME milestones. Data were analyzed by postgraduate year, assessor type, rotation, academic year, and core competency. The Mantel-Haenszel chi-square test was used to test for changes over time.

RESULTS

Specialties demonstrated heterogeneity in accounting for CDs in ACGME milestones, with 22% (22/99) of specialties having no language describing CDs in milestones assessment. Thirty-three percent (63/189) of UCMC internal medicine residents received at least one CD rating, with CDs accounting for 0.18% (668/364,728) of all assessment ratings. The authors identified CDs across multiple core competencies and rotations.

CONCLUSIONS

Despite some specialties not accounting for CDs in milestone assessment, UCMC's experience demonstrates that a significant proportion of residents may be rated as having a CD during training. Identification of CDs may allow programs to develop remediation and improvement plans.

Consensus-based training and assessment model for general surgery

Background

Surgical education is becoming competency-based with the implementation of in-training milestones. Training guidelines should reflect these changes and determine the specific procedures for such milestone assessments. This study aimed to develop a consensus view regarding operative procedures and tasks considered appropriate for junior and senior trainees, and the procedures that can be used as technical milestone assessments for trainee progression in general surgery.

Methods

A Delphi process was followed where questionnaires were distributed to all 17 Canadian general surgery programme directors. Items were ranked on a 5-point Likert scale, with consensus defined as Cronbach's α of at least 0·70. Items rated 4 or above on the 5-point Likert scale by 80 per cent of the programme directors were included in the models.

Results

Two Delphi rounds were completed, with 14 programme directors taking part in round one and 11 in round two. The overall consensus was high (Cronbach's α = 0·98). The training model included 101 unique procedures and tasks, 24 specific to junior trainees, 68 specific to senior trainees, and nine appropriate to all. The assessment model included four procedures.

Conclusion

A system of operative procedures and tasks for junior- and senior-level trainees has been developed along with an assessment model for trainee progression. These can be used as milestones in competency-based assessments.

International assessment practices along the continuum of surgical training

BACKGROUND

The objectives of this study were to assemble an international perspective on (1) current, and (2) ideal technical performance assessment methods, and (3) barriers to their adoption during: selection, in-training, and certification.

METHODS

A questionnaire was distributed to international educational directorates.

RESULTS

Eight of 10 jurisdictions responded. Currently, aptitude tests or simulated tasks are used during selection, observational rating scales during training and nothing is used at certification. Ideally, innate ability should be determined during selection, in-training evaluation reports, and global rating scales used during training, whereas global and procedure-specific rating scales used at the time of certification. Barriers include lack of predictive evidence for use in selection, financial limitations during training, and a combination with respect to certification.

CONCLUSIONS

Identifying current and ideal evaluation methods will prove beneficial to ensure the best assessments of technical performance are chosen for each training time point.

Promoting Success: A Professional Development Coaching Program for Interns in Medicine

BACKGROUND

Residency is an intense period. Challenges, including burnout, arise as new physicians develop their professional identities. Residency programs provide remediation, but emotional support for interns is often limited. Professional development coaching of interns, regardless of their performance, has not been reported.

OBJECTIVE

Design, implement, and evaluate a program to support intern professional development through positive psychology coaching.

METHODS

We implemented a professional development coaching program in a large residency program. The program included curriculum development, coach-intern interactions, and evaluative metrics. A total of 72 internal medicine interns and 26 internal medicine faculty participated in the first year. Interns and coaches were expected to meet quarterly; expected time commitments per year were 9 hours (per individual coached) for coaches, 5 1/2 hours for each individual coachee, and 70 hours for the director of the coaching program. Coaches and interns were asked to complete 2 surveys in the first year and to participate in qualitative interviews.

RESULTS

Eighty-two percent of interns met with their coaches 3 or more times. Coaches and their interns assessed the program in multiple dimensions (participation, program and professional activities, burnout, coping, and coach-intern communication). Most of the interns (94%) rated the coaching program as good or excellent, and 96% would recommend this program to other residency programs. The experience of burnout was lower in this cohort compared with a prior cohort.

CONCLUSIONS

There is early evidence that a coaching program of interactions with faculty trained in positive psychology may advance intern development and partially address burnout.

What is the Prevalence and Success of Remediation of Emergency Medicine Residents?

INTRODUCTION

The primary objective of this study was to determine the prevalence of remediation, competency domains for remediation, the length, and success rates of remediation in emergency medicine (EM).

METHODS

We developed the survey in Surveymonkey™ with attention to content and response process validity. EM program directors responded how many residents had been placed on remediation in the last three years. Details regarding the remediation were collected including indication, length and success. We reported descriptive data and estimated a multinomial logistic regression model.

RESULTS

We obtained 126/158 responses (79.7%). Ninety percent of programs had at least one resident on remediation in the last three years. The prevalence of remediation was 4.4%. Indications for remediation ranged from difficulties with one core competency to all six competencies (mean 1.9). The most common were medical knowledge (MK) (63.1% of residents), patient care (46.6%) and professionalism (31.5%). Mean length of remediation was eight months (range 1-36 months). Successful remediation was 59.9% of remediated residents; 31.3% reported ongoing remediation. In 8.7%, remediation was deemed "unsuccessful." Training year at time of identification for remediation (post-graduate year [PGY] 1), longer time spent in remediation, and concerns with practice-based learning (PBLI) and professionalism were found to have statistically significant association with unsuccessful remediation.

CONCLUSION

Remediation in EM residencies is common, with the most common areas being MK and patient care. The majority of residents are successfully remediated. PGY level, length of time spent in remediation, and the remediation of the competencies of PBLI and professionalism were associated with unsuccessful remediation.

Assessment and Management of Professionalism Issues in Pathology Residency Training: Results From Surveys and a Workshop by the Graduate Medical Education Committee of the College of American Pathologists

Professionalism issues are common in residency training and can be very difficult to recognize and manage. Almost one-third of the milestones for pathology recently instituted by the Accreditation Council for Graduate Medical Education encompass aspects of professionalism. Program directors are often unsure of how and when to remediate residents for unprofessional behavior. We used a case-based educational approach in a workshop setting to assist program directors in the management of unprofessional behavior in residents. Eight case scenarios highlighting various aspects of unprofessional behavior by pathology residents were developed and presented in an open workshop forum at the annual pathology program director’s meeting. Prior to the workshop, 2 surveys were conducted: (1) to collect data on program directors’ experience with identifying, assessing, and managing unprofessional behavior in their residents and (2) to get feedback from workshop registrants on how they would manage each of the 8 case scenarios. A wide range of unprofessional behaviors have been observed by pathology program directors. Although there is occasionally general agreement on how to manage specific behaviors, there remains wide variation in how to manage many of the presented unprofessional behaviors. Remediation for unprofessional behavior in pathology residents remains a difficult and challenging process. Additional education and research in this area are warranted.

Resident remediation, probation, and dismissal basic considerations for program directors

OBJECTIVES

To suggest a basic new approach for pathology training programs to consider when a resident requires remediation, probation, or dismissal.

METHODS

Remediation, probation, or dismissal of the poorly performing pathology resident is one of the most difficult and challenging aspects of any pathology training program. The poorly performing resident requires extra time and resources from the faculty and the program and can be disruptive for the entire program. Effective remediation requires faculty development, a well-constructed remediation or probation plan, and documentation.

RESULTS

Despite best efforts, not all remediation plans are successful and dismissal of the resident will need to be seriously considered.

CONCLUSIONS

Approaches to dealing with resident performance issues can be variable and need to be tailored to the issue being addressed.

Early detection of factual knowledge deficiency and remediation in otolaryngology residency education

OBJECTIVES/HYPOTHESIS

Within otolaryngology residency training, the annual Otolaryngology Training Examination (OTE) is the primary method used to assess, quantify, and compare the factual knowledge acquired by each resident. The objective of this study was to develop a more frequent method for tracking of factual knowledge to prevent educational delay.

STUDY DESIGN

Retrospective analysis of educational scores.

METHODS

For each didactic lecture within a single otolaryngology residency training program, multiple choice questions were provided before and after each lecture. Questions were based on lecture objectives that were derived from the American Board of Otolaryngology curriculum. Scores were tracked over the course of 1 academic year and compared to the scores of residents on the OTE administered in that year to determine correlation with a validated measure of factual knowledge. The effect of remedial measures on improvement in OTE scores was determined.

RESULTS

Over the course of 1 academic year, there were 328 questions presented to 12 residents before and after 32 lectures in the didactic program. Ten residents completed an average of 244 questions. Overall OTE scores demonstrated a significant and very strong correlation to lecture question scores (Pearson r = 0.86, P = .002). Remedial measures for residents during the previous 5 years who had inadequate OTE scores were successful in improving scores (P = .002).

CONCLUSIONS

A structured didactic program that uses review questions to assess knowledge can be used to track acquisition of factual knowledge. Early identification of residents with deficiencies facilitates the development of individualized learning plans that result in successful remediation.