Examining the Most Impactful Strategies for In-service Preparation: A Systemic Review

INTRODUCTION

Resident performance on the Plastic Surgery In-Service Examination (PSITE) is used as a predictor of success on the American Board of Plastic Surgery Written Examination, as well as resident progression and fellowship applications. However, information specifically addressing strategies on optimal PSITE preparation is lacking in the plastic surgery literature when compared to general surgery. For this reason, we aim to understand if the topic is well-studied and denote effective study strategies and curricular interventions in both fields that can help residents and programs optimize PSITE performance.

METHODS

A literature search including studies from 2012 to 2022 was conducted following PRISMA guidelines in PubMed and EMBASE to identify articles on strategies to improve in-service exam scores for general surgery and plastic surgery. Only studies that reported measurable outcomes in raw score, percentile score, or percent correct were included.

RESULTS

Qualitative analysis of 30 articles revealed 2 categories of interventions: individual study habits and institutional curricular interventions. In general surgery literature, 27 articles examined interventions positively impacting resident ABSITE scores, with 21 studies classified as institutional curricular interventions and 6 articles addressing individual study habits. Themes associated with improved ABSITE performance included mandatory remediation programs, dedicated study time, and problem-based learning interventions. In contrast, only 3 articles in plastic surgery literature discussed interventions associated with improved PSITE scores, all falling under curricular interventions.

CONCLUSION

Unfortunately, the plastic surgery literature lacks concrete evidence on how residents can improve performance. Future research in plastic surgery should replicate successful strategies from general surgery and further investigate optimal preparation strategies for the PSITE. Such endeavors can contribute to improving resident performance and advancing plastic surgery education and patient care.

Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review update: BEME Guide No. 85

BACKGROUND

Clinical teachers often struggle to record trainee underperformance due to lacking evidence-based remediation options.

OBJECTIVES

To provide updated evidence-based recommendations for addressing academic difficulties among undergraduate and postgraduate medical learners.

METHODS

A systematic review searched databases including MEDLINE, CINAHL, EMBASE, ERIC, Education Source, and PsycINFO (2016-2021), replicating the original Best Evidence Medical Education 56 review strategy. Original research/innovation reports describing intervention(s) for medical learners with academic difficulties were included. Data extraction used Michie's Behaviour Change Techniques (BCT) Taxonomy and program evaluation models from Stufflebeam and Kirkpatrick. Quality appraised used the Mixed Methods Appraisal Tool (MMAT). Authors synthesized extracted evidence by adapting GRADE approach to formulate recommendations.

RESULTS

Eighteen articles met the inclusion criteria, primarily addressing knowledge (66.7%), skills (66.7%), attitudinal problems (50%) and learner's personal challenges (27.8%). Feedback and monitoring was the most frequently employed BCT. Study quality varied (MMAT 0-100%). We identified nineteen interventions (UG: n = 9, PG: n = 12), introducing twelve new thematic content. Newly thematic content addressed contemporary learning challenges such as academic procrastination, and use of technology-enhanced learning resources. Combined with previous interventions, the review offers a total dataset of 121 interventions.

CONCLUSION

This review offers additional evidence-based interventions for learners with academic difficulties, supporting teaching, learning, faculty development, and research efforts.

Predictive factors of first-time failure on the American Board of Colorectal Surgery certifying and qualifying examinations

Objective

To discover if first-attempt failure of the American Board of Colon and Rectal Surgery (ABCRS) board examination is associated with surgical training or personal demographic characteristics.

Methods

Current colon and rectal surgery program directors in the United States were contacted via email. Deidentified records of trainees from 2011 to 2019 were requested. Analysis was performed to identify associations between individual risk factors and failure on the ABCRS board examination on the first attempt.

Results

Seven programs contributed data, totaling 67 trainees. The overall first-time pass rate was 88% (n = 59). Several variables demonstrated potential for association, including Colon and Rectal Surgery In-Training Examination (CARSITE) percentile (74.5 vs 68.0, P = 0.09), number of major cases in colorectal residency (245.0 vs 219.2, P = 0.16), >5 publications during colorectal residency (75.0% vs 25.0%, P = 0.19), and first-time passage of the American Board of Surgery certifying examination (92.5% vs 7.5%, P = 0.18).

Conclusion

The ABCRS board examination is a high-stakes test, and training program factors may be predictive of failure. Although several factors showed potential for association, none reached statistical significance. Our hope is that by increasing our data set, we will identify statistically significant associations that can potentially benefit future trainees in colon and rectal surgery.

Just Do It: Participation in Structured Online Curricula Reliably Improves Low ABSITE Scores

OBJECTIVE

We hypothesized residents enrolled in an Accelerated Clinical Education in Surgery (ACES) program would improve their scores to above the 30^th percentile. We analyzed which components of ACES correlated with improvement.

DESIGN AND SETTING

A retrospective review of three academic cycles (2018-2021) at an academic general surgery residency.

PARTICIPANTS

Residents scoring ≤30^th percentile on the ABSITE were enrolled in ACES. Baseline demographics including STEP scores were collected. ACES included: (1) SCORE and DeckerMed assignments (2) Weekly faculty review sessions and (3) Monthly meeting with assigned mentor. Data were analyzed by Student's t-test, one-way ANOVA and Fisher's exact test.

RESULTS

Twenty-six surgical residents enrolled in ACES. Compared to residents not in ACES, there was no significant difference females (15 vs. 15; p = 0.19) and STEP 2 scores (241 vs. 246; p = 0.06). Residents in ACES had significantly lower STEP 1 (225 vs. 237; p < 0.001) and STEP 3 (212 vs. 223; p < 0.001) scores. Demographics of ACES residents who subsequently scored >30^th percentile were similar to those who didn't, except for STEP 3 scores (216 vs. 204; p = 0.008). For residents in ACES, the completion of assignments between July and January was significantly higher for those who subsequently achieved an ABSITE score >30^th percentile: TWIS, 77% vs. 53% (p = 0.022), Decker WC, 80% vs. 49% (p = 0.009) and Decker MR, 53% vs. 29% (p = 0.016). Completion of an online practice exam prior to ABSITE also correlated with score >30^th percentile (57% vs. 13%, p = 0.007). There was also no correlation between the number of faculty review sessions and ABSITE (11.5 vs.11.9, p = 0.931).

CONCLUSIONS

Participation in a structured online program of reading and quizzes was durably effective in improving ABSITE scores >30^th percentile. Completion of online assignments, rather than scores on practice tests or review sessions, appeared to be the most important factor for success.

Emotion in remediation: A scoping review of the medical education literature

OBJECTIVES

Remediation can be crucial and high stakes for medical learners, and experts agree it is often not optimally conducted. Research from other fields indicates that explicit incorporation of emotion improves education because of emotion's documented impacts on learning. Because this could present an important opportunity for improving remediation, we aimed to investigate how the literature on remediation interventions in medical education discusses emotion.

METHODS

The authors used Arksey and O'Malley's framework to conduct a scoping literature review of records describing remediation interventions in medical education, using PubMed, CINAHL Complete, ERIC, Web of Science and APA PsycInfo databases, including all English-language publications through 1 May 2020 meeting search criteria. They included publications discussing remediation interventions either empirically or theoretically, pertaining to physicians or physician trainees of any level. Two independent reviewers used a standardised data extraction form to report descriptive information; they reviewed included records for the presence of mentions of emotion, described the mentions and analysed results thematically.

RESULTS

Of 1644 records, 199 met inclusion criteria and were reviewed in full. Of those, 112 (56%) mentioned emotion in some way; others focused solely on cognitive aspects of remediation. The mentions of emotion fell into three themes based on when the emotion was cited as present: during regular coursework or practice, upon referral for remediation and during remediation. One-quarter of records (50) indicated potential intentional incorporation of emotion into remediation programme design, but they were non-specific as to how emotions related to the learning process itself.

CONCLUSION

Even though emotion is omnipresent in remediation, medical educators frequently do not factor emotion into the design of remediation approaches and rarely explicitly utilise emotion to improve the learning process. Applications from other fields may help medical educators leverage emotion to improve learning in remediation, including strategies to frame and design remediation.

Trainee Utilization of the SCORE Curriculum is Associated With Improved ABSITE Performance: A Multi-institutional Study

BACKGROUND

The Surgical Council on Resident Education's (SCORE) structured educational curriculum for general surgery residents uses a 2-year repeating cycle of modules and quizzes called "This Week in SCORE" (TWIS) to organize and sequence the curricular content on the SCORE portal. The purpose of this study was to evaluate the impact of using the SCORE content and TWIS curriculum on American Board of Surgery In-Training Examination (ABSITE) performance. To date, no multi-institutional studies have examined this effect.

METHODS

Eight residency programs participated, including university and community-based programs. SCORE usage overall, TWIS usage, and ABSITE percentile scores (adjusted for post-graduate year [PGY]) were analyzed for the academic years 2017 to 2020. SCORE usage was defined as number of SCORE logins annually per resident with "low usage" ≤10 times and "high usage" >10 times. TWIS usage was defined as "low usage" (no TWIS quizzes done) or "high usage" (≥1 TWIS quiz per year).

RESULTS

Four hundred and twenty-eight trainees were evaluated. Trainees with high SCORE and TWIS usage consistently achieved a higher ABSITE percentile score. This difference was maintained in subgroup analysis by PGY with the greatest impact in PGY-1 and PGY-2 levels.

CONCLUSIONS

Utilization of multiple aspects of SCORE content appears to have a significant positive impact on ABSITE performance across all levels of postgraduate training. This multi-institutional study of a large number of users is the first to demonstrate that increased usage of SCORE content appears to be a predictor of ABSITE performance success.

Remediation for surgical trainees: recommendations from a narrative review

BACKGROUND

Remediation involves formalized support for surgical trainees with significant underperformance to return to expected standards. There is a need to understand current evidence of remediation for surgical trainees to inform practice and justify investment of resources.

METHODS

Following the principles of a systematic review, we conducted a narrative analysis to make recommendations for remediation of underperforming surgical trainees.

RESULTS

From a review of 55 articles on remediation of trainees in medical and surgical sub-specialities, we have identified system and process level recommendations. Remediation is reported as long-term, complex and resource-intensive. Establishing a defined and standardized remediation framework enables co-ordination of multi-modal interventions. System level recommendations aim to consolidate protocols via developing better assessment, intervention and re-evaluation modalities whilst also strengthening support to supervisors conducting the remediation. Process level recommendations should be tailored for the specific needs of each trainee, aiming to be proactive with interventions within a programmatic framework. Regular reassessment is required, and long-term follow-up shows that remediation efforts are often successful.

CONCLUSION

While remediation within a programmatic framework is complex, it is often a successful approach to return surgical trainees to their expected standard. Future directions involve applying learning theories, encouraging research methods and to develop integrated collaborate protocols and support to synergize efforts.

The Surgical Knowledge "Growth Curve": Predicting ABSITE Scores and Identifying "At-Risk" Residents

OBJECTIVE

Resident performance on the American Board of Surgery In-Training Examination (ABSITE) is used for evaluation of surgical knowledge and guides resident selection for institutional remediation programs. Remediation thresholds have historically been based on ABSITE percentile scores; however, this does not account for predictors that can impact a resident's exam performance. We sought to identify predictors of yearly ABSITE performance to help identify residents "at-risk" for performing below their expected growth trajectory.

DESIGN

The knowledge of the residents, as measured by standardized ABSITE scores, was modeled as a function of the corresponding postgraduate year via a linear mixed effects regression model. Additional model covariates included written USMLE-1-3 examination scores, gender, number of practice questions completed, and percentage correct of practice questions. For each resident, the predicted ABSITE standard score along with a 95% bootstrap prediction interval was obtained. Both resident-specific and population-level predictions for ABSITE standard scores were also estimated.

SETTING

The study was conducted at a single, large academic medical center (Massachusetts General Hospital, Boston, MA).

PARTICIPANTS

Six years of general surgery resident score reports at a single institution between 2014 and 2019 were deidentified and analyzed.

RESULTS

A total of 376 score reports from 130 residents were analyzed. Covariates that had a significant effect on the model included USMLE-1 score (PGY1: p = 0.013; PGY2: p = 0.007; PGY3: p = 0.011), USMLE-2 score (PGY1: p < 0.001; PGY2: p < 0.001; PGY3: p < 0.001; PGY4: p < 0.001; PGY5: p = 0.032), male gender (PGY1: p = 0.003; PGY2: p < 0.001; PGY3: p < 0.001; PGY4: p = 0.008), and number of practice questions completed (p=0.003). Five residents were identified as having "fallen off" their predicted knowledge curve, including a single resident on 2 occasions. Population prediction curves were obtained at 7 different covariate percentile levels (5%, 10%, 25%, 50%, 75%, 90%, and 95%) that could be used to plot predicted resident knowledge progress.

CONCLUSION

Performance on USMLE-1 and -2 examinations, male gender, and number of practice questions completed were positive predictors of ABSITE performance. Creating residency-wide knowledge growth curves as well as individualized predictive ABSITE performance models allows for more efficient identification of residents potentially at risk for poor ABSITE performance and structured monitoring of surgical knowledge progression.