Early detection of factual knowledge deficiency and remediation in otolaryngology residency education

Post graduate remediation programs in medicine: a scoping review

BACKGROUND

Recognizing that physicians may struggle to achieve knowledge, skills, attitudes and or conduct at one or more stages during their training has highlighted the importance of the 'deliberate practice of improving performance through practising beyond one's comfort level under guidance'. However, variations in physician, program, contextual and healthcare and educational systems complicate efforts to create a consistent approach to remediation. Balancing the inevitable disparities in approaches and settings with the need for continuity and effective oversight of the remediation process, as well as the context and population specific nature of remediation, this review will scrutinise the remediation of physicians in training to better guide the design, structuring and oversight of new remediation programs.

METHODS

Krishna's Systematic Evidence Based Approach is adopted to guide this Systematic Scoping Review (SSR in SEBA) to enhance the transparency and reproducibility of this review. A structured search for articles on remediation programs for licenced physicians who have completed their pre-registration postings and who are in training positions published between 1st January 1990 and 31st December 2021 in PubMed, Scopus, ERIC, Google Scholar, PsycINFO, ASSIA, HMIC, DARE and Web of Science databases was carried out. The included articles were concurrently thematically and content analysed using SEBA's Split Approach. Similarities in the identified themes and categories were combined in the Jigsaw Perspective and compared with the tabulated summaries of included articles in the Funnelling Process to create the domains that will guide discussions.

RESULTS

The research team retrieved 5512 abstracts, reviewed 304 full-text articles and included 101 articles. The domains identified were characteristics, indications, frameworks, domains, enablers and barriers and unique features of remediation in licenced physicians in training programs.

CONCLUSION

Building upon our findings and guided by Hauer et al. approach to remediation and Taylor and Hamdy's Multi-theories Model, we proffer a theoretically grounded 7-stage evidence-based remediation framework to enhance understanding of remediation in licenced physicians in training programs. We believe this framework can guide program design and reframe remediation's role as an integral part of training programs and a source of support and professional, academic, research, interprofessional and personal development.

Early Detection and Remediation of Problem Learners

There are myriad types of problem learners in surgical residency and most have difficulty in more than 1 competency. Programs that use a standard curriculum of study and assessment are most successful in identifying struggling learners early. Many problem learners lack appropriate systems for study; a multidisciplinary educational team that is separate from the team that evaluates the success of remediation is critical. Struggling residents who require formal remediation benefit from performance improvement plans that clearly outline the issues of concern, describe the steps required for remediation, define success of remediation, and outline consequences for failure to remediate appropriately.

Amant L, Glover Takahashi S. Managing residents in difficulty within CBME residency educational systems: a scoping review

BACKGROUND

Best practices in managing residents in difficulty (RID) in the era of competency-based medical education (CBME) are not well described. This scoping review aimed to inventory the current literature and identify major themes in the articles that address or employ CBME as part of the identification and remediation of residents in difficulty.

METHODS

Articles published between 2011 to 2017 were included if they were about postgraduate medical education, RID, and offered information to inform the structure and/or processes of CBME. All three reviewers performed a primary screening, followed by a secondary screening of abstracts of the chosen articles, and then a final comprehensive sub-analysis of the 11 articles identified as using a CBME framework.

RESULTS

Of 165 articles initially identified, 92 qualified for secondary screening; the 63 remaining articles underwent full-text abstracting. Ten themes were identified from the content analysis with "identification of RID" (41%) and "defining and classifying deficiencies" (30%) being the most frequent. In the CBME article sub-analysis, the most frequent themes were: need to identify RID (64%), improving assessment tools (45%), and roles and responsibilities of players involved in remediation (27%). Almost half of the CBME articles were published in 2016-2017.

CONCLUSIONS

Although CBME programs have been implemented for many years, articles have only recently begun specifically addressing RID within a competency framework. Much work is needed to describe the sequenced progression, tailored learning experiences, and competency-focused instruction. Finally, future research should focus on the outcomes of remediation in CBME programs.

Guidelines: The dos, don'ts and don't knows of remediation in medical education

INTRODUCTION

Two developing forces have achieved prominence in medical education: the advent of competency-based assessments and a growing commitment to expand access to medicine for a broader range of learners with a wider array of preparation. Remediation is intended to support all learners to achieve sufficient competence. Therefore, it is timely to provide practical guidelines for remediation in medical education that clarify best practices, practices to avoid, and areas requiring further research, in order to guide work with both individual struggling learners and development of training program policies.

METHODS

Collectively, we generated an initial list of Do's, Don'ts, and Don't Knows for remediation in medical education, which was then iteratively refined through discussions and additional evidence-gathering. The final guidelines were then graded for the strength of the evidence by consensus.

RESULTS

We present 26 guidelines: two groupings of Do's (systems-level interventions and recommendations for individual learners), along with short lists of Don'ts and Don't Knows, and our interpretation of the strength of current evidence for each guideline.

CONCLUSIONS

Remediation is a high-stakes, highly complex process involving learners, faculty, systems, and societal factors. Our synthesis resulted in a list of guidelines that summarize the current state of educational theory and empirical evidence that can improve remediation processes at individual and institutional levels. Important unanswered questions remain; ongoing research can further improve remediation practices to ensure the appropriate support for learners, institutions, and society.

Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review: BEME Guide No. 56

Background: Clinical teachers often struggle to report unsatisfactory trainee performance, partly because of a lack of evidence-based remediation options. Objectives: To identify interventions for undergraduate (UG) and postgraduate (PG) medical learners experiencing academic difficulties, link them to a theory-based framework and provide literature-based recommendations around their use. Methods: This systematic review searched MEDLINE, CINAHL, EMBASE, ERIC, Education Source and PsycINFO (1990-2016) combining these concepts: medical education, professional competence/difficulty and educational support. Original research/innovation reports describing intervention(s) for UG/PG medical learners with academic difficulties were included. Data extraction employed Michie's Behavior Change Techniques (BCT) Taxonomy and program evaluation models from Stufflebeam and Kirkpatrick. Quality appraisal used the Mixed Methods Appraisal Tool (MMAT). The authors synthesized extracted evidence by adapting the GRADE approach to formulate recommendations. Results: Sixty-eight articles met the inclusion criteria, most commonly addressing knowledge (66.2%), skills (53.9%) and attitudinal problems (26.2%), or learner personal issues (41.5%). The most common BCTs were Shaping knowledge, Feedback/monitoring, and Repetition/substitution. Quality appraisal was variable (MMAT 0-100%). A thematic content analysis identified 109 interventions (UG: n = 84, PG: n = 58), providing 24 strong, 48 moderate, 26 weak and 11 very weak recommendations. Conclusion: This review provides a repertoire of literature-based interventions for teaching/learning, faculty development, and research purposes.

Are Otolaryngology Milestones Predictive of Otolaryngology Training Examination Scores?

IMPORTANCE:

Otolaryngology residents take the otolaryngology training examination (OTE) yearly to assess their fund of knowledge. The Accreditation Council for Graduate Medical Education (ACGME) milestone evaluations are also conducted semiannually. Accurate prediction of training examination performance allows identification of residents who are performing well and those who need targeted remediation. Prior studies in other specialties have attempted to use milestone evaluations to help predict in-training examination scores.

OBJECTIVE:

In this study, we aim to identify whether ACGME milestone evaluation scores predict OTE performance.

DESIGN:

Milestone ratings and OTE scores for residents at 2 US otolaryngology residency programs were collected. Multivariate analysis was achieved using linear mixed modeling. We considered a 2-tailed P value of ≤ .05 as statistically significant.

SETTING:

Two US otolaryngology residency programs.

PARTICIPANTS:

Forty-eight otolaryngology residents postgraduate years 2 to 5.

RESULTS:

Otolaryngology training examination scores and ACGME milestone evaluations were collected from 48 residents from postgraduate year 2 to 5 between the years 2014 and 2017. One hundred eight OTE scores were available. Linear mixed-effect models were constructed, and after adjusting for level of training and OTE year, the total milestone rating made a negligible impact in estimating OTE percentage correct (β = -.01, P = .9). Similarly, total milestone rating demonstrated a minimal contribution in approximating OTE national stanine score after adjusting for the level of training (β = -.003, P = .9).

CONCLUSIONS AND RELEVANCE:

In our study, ACGME milestone evaluations are not predictive of residents' OTE performance. What these milestone evaluation data mean and how they should be used continues to be an unanswered question. We should aim to identify the most effective applications of the milestone data collected yearly by otolaryngology programs.

The impact of a resident-run review curriculum and USMLE scores on the Otolaryngology in-service exam

OBJECTIVE

Describe the association of USMLE Step 1 scores and the institution of a dedicated board review curriculum with resident performance on the Otolaryngology training examination.

STUDY DESIGN

Retrospective cross sectional study.

METHODS

We reviewed American Board of Otolaryngology Training Examination (OTE) scores for an otolaryngology residency program between 2005 and 2016. USMLE Step 1 scores were collected. In 2011 a resident-run OTE review curriculum was instituted with the goal of improving test preparation. Scores were compared before and after curriculum institution. Linear regression was performed to identify predictors of OTE scores.

RESULTS

47 residents were evaluated, 24 before and 23 after instituting the curriculum. There was a moderate correlation between USMLE step 1 scores and OTE scores for all years. For PGY-2 residents, mean OTE scores improved from 25th percentile to 41st percentile after institution of the review curriculum (p = 0.05). PGY 3-5 residents demonstrated no significant improvement. On multivariate linear regression, after controlling for USMLE step 1 scores, a dedicated board review curriculum predicted a 23-point percentile improvement in OTE scores for PGY-2 residents (p = 0.003). For other post-graduate years, the review curriculum did not predict score improvement.

CONCLUSION

USMLE step 1 scores are moderately correlated with OTE performance. A dedicated OTE review curriculum may improve OTE scores for PGY-2 residents, but such a curriculum may have less benefit for intermediate- and senior-level residents.

LEVEL OF EVIDENCE

4.

What is the Prevalence and Success of Remediation of Emergency Medicine Residents?

INTRODUCTION

The primary objective of this study was to determine the prevalence of remediation, competency domains for remediation, the length, and success rates of remediation in emergency medicine (EM).

METHODS

We developed the survey in Surveymonkey™ with attention to content and response process validity. EM program directors responded how many residents had been placed on remediation in the last three years. Details regarding the remediation were collected including indication, length and success. We reported descriptive data and estimated a multinomial logistic regression model.

RESULTS

We obtained 126/158 responses (79.7%). Ninety percent of programs had at least one resident on remediation in the last three years. The prevalence of remediation was 4.4%. Indications for remediation ranged from difficulties with one core competency to all six competencies (mean 1.9). The most common were medical knowledge (MK) (63.1% of residents), patient care (46.6%) and professionalism (31.5%). Mean length of remediation was eight months (range 1-36 months). Successful remediation was 59.9% of remediated residents; 31.3% reported ongoing remediation. In 8.7%, remediation was deemed "unsuccessful." Training year at time of identification for remediation (post-graduate year [PGY] 1), longer time spent in remediation, and concerns with practice-based learning (PBLI) and professionalism were found to have statistically significant association with unsuccessful remediation.

CONCLUSION

Remediation in EM residencies is common, with the most common areas being MK and patient care. The majority of residents are successfully remediated. PGY level, length of time spent in remediation, and the remediation of the competencies of PBLI and professionalism were associated with unsuccessful remediation.