Improving resident performance on standardized assessments of medical knowledge: a retrospective analysis of interventions correlated to American Board of Surgery In-Service Training Examination performance

American Board of Surgery In-Training Exam Performance Predicted by Question Bank Use While Unassociated With Other Learning Strategies

INTRODUCTION

There is no consensus regarding optimal curricula to teach cognitive elements of general surgery. The American Board of Surgery In-Training Exam (ABSITE) aims to measure trainees' progress in attaining this knowledge. Resources like question banks (QBs), Surgical Council on Resident Education (SCORE) curriculum, and didactic conferences have mixed findings related to ABSITE performance and are often evaluated in isolation. This study characterized relationships between multiple learning methods and ABSITE performance to elucidate the relative educational value of learning strategies.

METHODS

Use and score of QB, SCORE use, didactic conference attendance, and ABSITE percentile score were collected at an academic general surgery residency program from 2017 to 2022. QB data were available in the years 2017-2018 and 2021-2022 during institutional subscription to the same platform. Given differences in risk of qualifying exam failure, groups of ≤30th and >30th percentile were analyzed. Linear quantile mixed regressions and generalized linear mixed models determined factors associated with ABSITE performance.

RESULTS

Linear quantile mixed regressions revealed a relationship between ABSITE performance and QB questions completed (1.5 percentile per 100 questions, P < 0.001) and QB score (1.2 percentile per 1% score, P < 0.001), but not with SCORE use and didactic attendance. Performers >30th percentile had a significantly higher QB score.

CONCLUSIONS

Use and score of QB had a significant relationship with ABSITE performance, while SCORE use and didactic attendance did not. Performers >30th percentile completed a median 1094 QB questions annually with a score of 65%. Results emphasize success of QB use as an active learning strategy, while passive learning methods warrant further evaluation.

Modern Didactic Formats in Surgery: A Systematic Review

Introduction: "Traditional teaching" models often fail to engage millennial residents. Multiple modern didactic methods have been employed. The most frequently used objective measure to assess the effectiveness of didactic formats has been American Board of Surgery In-Training Examination performance.Methods: A literature search was conducted searching PubMed, EMBASE, and JAMA Network from June 2011 to June 2021, in accordance with the PRISMA guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination." Only studies discussing didactic structures were included.Results: A final 16 studies were included. Modern methods such as a "flipped classroom," Team Based Learning (TBL), and "gamification" have all shown increased engagement and significantly improved ABSITE performance. Structured biostatistics reviews may be used to supplement research and statistics which are often missed by other resources.Discussion: Programs have a duty to promote excellent resident education. In addition to fostering individual study habits, didactics and program structures should be optimized for resident development. As opposed to focusing on the pure amount of scheduled protected time, programs may instead consider focusing on the quality of the didactic format used and modern didactic methods may be beneficial.

United States medical licensing examination Step 1 scores are a quantifiably better predictor of American Board of Surgery In-Training Examination success compared to Step 2 scores

BACKGROUND

United States Medical Licensing Examination Step 1 (STEP 1) and Step 2 (STEP 2) scores are used in the selection of surgery residents. The American Board of Surgery In-Training Examination (ABSITE) is employed to assess resident knowledge. We sought to determine whether both STEP 1 and/or STEP 2 were predictive of ABSITE performance.

METHODS

We performed a 10-year retrospective analysis of all PGY levels comparing STEP 1 and 2 scores with raw ABSITE scores and percentile rank. Additionally, we analyzed STEP 1 and 2 scores for the likelihood of scoring above the 20th percentile on the ABSITE.

RESULTS

When STEP 1 and STEP 2 scores were evaluated separately, both were independent predictors of ABSITE performance. However, when STEP 1 and STEP 2 scores were evaluated together, only STEP 1 scores were predictive of ABSITE performance. Finally, only STEP 1 scores were predictive of scoring above the 20th percentile.

CONCLUSION

STEP 1 scores are a better predictor of ABSITE performance than STEP 2.

The Plastic Surgery In-service Training Examination: An In-depth Reference Analysis

The goal of this study was to characterize the references provided as supporting evidence of the Plastic Surgery In-service Training Examination (PSITE) syllabi, including those on the novel “core surgical principles” section.

Practice Makes Perfect: Introducing a Question Bank for ABSITE Preparation Improves Program Performance

BACKGROUND

The American Board of Surgery In-Training Examination (ABSITE) is an important predictor of passing the Qualifying Examination and a determinant of fellowship competitiveness.

OBJECTION

Study the impact of providing program-wide access to a commercially available question bank for ABSITE preparation.

STUDY DESIGN

The surgery residency program purchased access to the TrueLearn question bank in 2018 A paired sample t test analysis compared the 2018 ABSITE percentage and percentile scores, prior to practice question bank access to 2019 ABSITE percentile scores. A simple linear regression analysis was calculated to predict improvement in percentage scores from 2018 to 2019 based on total number of practice questions as well as number of correct practice questions completed. Data were analyzed using SPSS.

RESULTS

Among the residents utilizing practice questions with serial exam scores, the individual resident ABSITE percentage of correct questions showed a statistically significant improvement after introduction of the question bank from 2018 (mean = 68.7, standard deviation = 7.3) to 2019 (mean = 72.2, standard deviation = 7.2; t(35) = -4.529, p < 0.001). A statistically significant regression equation both linear (F(1,33) = 6.274, p = 0.017) and logarithmic (F(1,33) = 7.405, p =0.01) was found with an R2 of 0.160 and 0.183, respectively, for total number of practice questions completed, signifying that more completed practice questions correlated with a higher improvement in ABSITE percentage score. The improvement in residents' ABSITE percentage score increased by 3 ± 1 percentage point for each 100 practice questions completed from 2018 to 2019 (Figure 1). A significant regression equation was also found for improvement in percentage score among all residents (F (1,33) = 8.211, p = 0.007) with an R2 of 0.199 for the number of correct practice questions completed.

CONCLUSION

Use of a commercial question bank improved overall ABSITE scores. More questions answered translated into improved performance. Percent correct on the practice questions also correlated strongly with performance. Programs seeking to improve scores may wish to provide access to a question bank.

The Impact of a Comprehensive Resident Curriculum and Required Participation in "This Week in SCORE" on General Surgery ABSITE Performance and Well-Being

OBJECTIVE

This Week in SCORE (TWIS) is a biennial, general surgery curriculum comprised of weekly online modules, readings, and multiple-choice quizzes. In this study, we examined the impact of required TWIS on American Board of Surgery In-Training Examination (ABSITE) scores and well-being among categorical general surgery residents.

DESIGN, SETTING, AND PARTICIPANTS

TWIS quiz completion became required in 2017. Residents attended weekly lectures, and ABSITE performance was incentivized with educational stipends. Surveys were distributed to assess study preferences, learning styles, burnout, and grit. Thirty-six categorical general surgery residents who took ABSITE in both 2017 and 2018 were evaluated in a paired-sample, retrospective analysis.

RESULTS

After requiring TWIS, median ABSITE percentile increased by 12% (65%-77%, p = 0.001). Weekly TWIS completion (59% vs 89%, p < 0.001) and quiz results (62% vs 69%, p = 0.005) also improved. During this time, emotional exhaustion and depersonalization declined significantly, yet overall burnout scores did not change. Of 21 survey respondents, 66.7% (n = 14) increased weekly study time by a median of 2.5 hours. However, less than half used Surgical Council on Resident Education as their primary study tool. Only 23.8% (n = 5) reported that mandatory TWIS modified their study behavior, while 90.4% (n = 19) felt the culture of education had improved.

CONCLUSIONS

After TWIS participation became required, ABSITE performances improved. Formalized curriculum with frequent assessment may foster accountability among residents, enhancing educational climate, well-being, and test performance.

Factors that Predict an Intern's First ABSITE Score are Known by September

BACKGROUND

Previous studies offer conflicting relevance of a variety of factors to predict resident performance on the The American Board of Surgery In-Training Exam (ABSITE). With numerous stellar applicants scoring poorly on their first ABSITE, we sought to identify key factors that might allow us to tailor pre-emptive study efforts in the fall and early winter to enhance scores.

METHODS

General Surgery residents in our program from 2009 through 2016 were included in our cohort study. Specific trainee data (sex, prelim vs categorical, United States Medical Licensing Examination (USMLE) Step 1 and 2 scores, ABSITE scores, clinical rotations, biannual objective structured clinical examination OSCE-type scores, and in-house prep test [IHPT], etc.) were collected retrospectively. The data were analyzed using JMP pro 10 and MedCalc.

RESULTS

ABSITE scores of our 110 trainees did not vary by sex or by categorical vs preliminary residents. USMLE step 1 and 2, IHPT and one objective structured clinical examination (OSCE) station (Sim-Based Trauma Exam) scores were positively correlated with ABSITE scores (p < 0.05; correlation coefficient (CC) = 0.6 [strong] for IHPT, 0.5 [moderate] for step 1 and 2 and [weak] 0.3 for Sim-Based Trauma Exam). The mean (standard deviation) ABSITE %tile score for residents scoring above 230 (USMLE 1 or 2) vs <230 were different: 81 (3) vs 56 (4) (p < 000.1). Of residents scoring lower than 230 in USMLE 1 or 2, their ABSITE performance had a strong positive linear correlation with their performance in the IHPT (CC = 0.7) and SBTA (CC = 0.5). Residents rotating twice on the Acute Care Surgery services before the ABSITE scored higher than those with 1 or zero rotations (p < 0.05).

CONCLUSIONS

USMLE steps 1 and 2 are useful parameters in our program to predict subsequent resident ABSITE performance. An in-house 60-minute preparation test in September, a 6-minute simulation-based trauma assessment in July, and rotation schedule with 2 stints on Acute Care Surgery were surprisingly useful early intern year tools to predict ABSITE scores.

Success in Implementation of a Resident In-Service Examination Review Series

OBJECTIVES

Primary pathology board certification has been correlated with senior resident in-service examination (RISE) performance. We describe our success with an annual, month-long review series.

METHODS

Aggregate program RISE performance data were gathered for 3 years prior to and 3 years following initiation of the review series. In addition, mean United States Medical Licensing Examination Step 1 and 2 Clinical Knowledge scores for residents participating in each RISE examination were obtained to control for incoming knowledge and test-taking ability. Linear models were used to evaluate differences in average RISE performance prior to and following the initiation of the review series in addition to controlling for relevant covariates.

RESULTS

Significant improvement was noted in the grand total, anatomic pathology section average, clinical pathology section average, and transfusion medicine section. Although not statistically significant, improvement was noted on the cytopathology and clinical chemistry sections. There was no significant difference in scores in hematopathology, molecular pathology, and the special topics section average. In addition, improvement in primary pathology board certification rates was also noted.

CONCLUSIONS

Institution of a month-long RISE review series demonstrated improved overall performance within our training program. The success could easily be replicated in any training program without significant disruption to an annual didactic series.

Active Learning in Medical Education: Application to the Training of Surgeons

Our article defines active learning in the context of surgical education and reviews the growing body of research on new approaches to teaching. We then discuss future perspectives and the challenges faced by the trainee and surgeon in applying active learning to surgical training. As modern surgical education faces numerous challenges, we hope our article will help surgical educators in the evaluation of curriculum development, methods of instruction, and assessment.