The American Board Style Practice In-Training Examination as a Predictor of Performance on the American Board of Surgery In-Training Examination

Prospective Factors that Predict American Board of Surgery In-Training Examination Performance: A Systematic Review

INTRODUCTION

American Board of Surgery In-Training Examination (ABSITE) performance has become an important factor when monitoring resident progress. Understanding which prospective factors predict performance can help identify residents at risk.

METHODS

A literature search was conducted searching PubMed, EMBASE, and JAMA Network from June 2011 to June 2021, in accordance with the PRISMA guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination." Prospective factors such as prior examination performance, clinical evaluations, and demographics were evaluated.

RESULTS

A final 35 studies were included. The prospective factor most consistently found to predict ABSITE performance is performance on prior knowledge-based examinations such as the USMLE step exams. The ACGME Medical Knowledge 1 milestone evaluation also appears to correlate to ABSITE performance, although clinical evaluations, in general, do not. Demographics have no significant correlation to ABSITE performance.

DISCUSSION

Using performance on prior knowledge-based examinations programs may be able to identify residents at risk for failing ABSITE. It may be possible to initiate early intervention before rather than only remediation after poor performance.

Mid-Year Medical Knowledge Milestones and ABSITE Scores in First-Year Surgery Residents

OBJECTIVE

Accreditation Council for Graduate Medical Education (ACGME) Surgery milestone ratings in the "Knowledge of Diseases and Conditions" (MK1) sub competency have been shown to correlate with American Board of Surgery In Training Examination (ABSITE) scores, and hypothesized to predict them. To better assess the predictive value of the MK1 milestone and avoid the potential bias caused by previous years' ABSITE scores, we designed a study including only first-year (PGY-1) residents and analyzed the correlation between their mid-year MK1 ratings and their scores in the ABSITE they took approximately a month later.

METHODS

De-identified United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores, mid-year MK1 milestone ratings and the subsequent ABSITE standard scores for the five academic years from 2014-2015 to 2018-2019 were collected and tabulated for 247 PGY-1 preliminary- and categorical-track residents from ten ACGME-accredited surgery residency programs.

RESULTS

The mid-year rating of PGY-1 residents' MK1 was predictive of their subsequent first ABSITE score for the entire cohort and for the categorical residents' subset. Notably, controlling for all other independent predictors, each half-point increase in MK1 rating was associated with a 25-point increase in ABSITE score. Preliminary residents performed significantly worse on the ABSITE, and their scores did not correlate significantly with their MK1 ratings.

CONCLUSIONS

The mid-year rating of PGY-1 residents' MK1 was predictive of their subsequent first ABSITE score for the entire cohort and for the categorical but not the preliminary residents. This finding suggests that evaluators correctly rated MK1 higher in the categorical residents who did perform better on the subsequent ABSITE.

General Surgery Resident Use of Electronic Resources: 15 Minutes a Day

BACKGROUND

General surgery resident performance on the American Board of Surgery In-Service Training Exam (ABSITE) has been used to predict American Board of Surgery (ABS) passage rates, selection for remediation programs, and ranking of fellowship applicants. We sought to identify electronic resource study habits of general surgery residents associated with successful test scores.

STUDY DESIGN

A single-institution, retrospective review of general surgery resident use of 2 electronic study resources, Surgical Council on Resident Education (SCORE) and TrueLearn (TL), were evaluated for the 7 months before the 2019 ABSITE. Metrics included TL question performance, SCORE use, and a survey about other reading sources. These metrics were evaluated in 3 ABSITE percentile groupings: ≥80^th, 31^st to 79^th, and ≤30^th.

RESULTS

The ≥80^th and 31^st to 79^th percentile groups scored higher on TL questions, at 69% and 67.7%, respectively, compared with 61.4% for the ≤30^th percentile group (p < 0.03). The ≥80^th percentile group spent on average 14.6 minutes/day on SCORE compared with 5.0 minutes/day and 4.7 minutes/day for the 31^st to 79^th and ≤30^th percentile groups, respectively (p < 0.04). The ≥80^th percentile group spent 34.8 minutes/session (77 sessions) compared with 19.2 minutes/session (49 sessions) and 20.7 minutes/session (43 sessions) in the 31^st to 79^th and ≤30^th percentile groups, respectively (p = 0.009).

CONCLUSIONS

Our nomogram incorporates time spent accessing an electronic content-based resource, SCORE, and performance on an electronic question-based resource as a novel method to provide individualized feedback and predict future ABSITE performance.

Practice Makes Perfect: Introducing a Question Bank for ABSITE Preparation Improves Program Performance

BACKGROUND

The American Board of Surgery In-Training Examination (ABSITE) is an important predictor of passing the Qualifying Examination and a determinant of fellowship competitiveness.

OBJECTION

Study the impact of providing program-wide access to a commercially available question bank for ABSITE preparation.

STUDY DESIGN

The surgery residency program purchased access to the TrueLearn question bank in 2018 A paired sample t test analysis compared the 2018 ABSITE percentage and percentile scores, prior to practice question bank access to 2019 ABSITE percentile scores. A simple linear regression analysis was calculated to predict improvement in percentage scores from 2018 to 2019 based on total number of practice questions as well as number of correct practice questions completed. Data were analyzed using SPSS.

RESULTS

Among the residents utilizing practice questions with serial exam scores, the individual resident ABSITE percentage of correct questions showed a statistically significant improvement after introduction of the question bank from 2018 (mean = 68.7, standard deviation = 7.3) to 2019 (mean = 72.2, standard deviation = 7.2; t(35) = -4.529, p < 0.001). A statistically significant regression equation both linear (F(1,33) = 6.274, p = 0.017) and logarithmic (F(1,33) = 7.405, p =0.01) was found with an R2 of 0.160 and 0.183, respectively, for total number of practice questions completed, signifying that more completed practice questions correlated with a higher improvement in ABSITE percentage score. The improvement in residents' ABSITE percentage score increased by 3 ± 1 percentage point for each 100 practice questions completed from 2018 to 2019 (Figure 1). A significant regression equation was also found for improvement in percentage score among all residents (F (1,33) = 8.211, p = 0.007) with an R2 of 0.199 for the number of correct practice questions completed.

CONCLUSION

Use of a commercial question bank improved overall ABSITE scores. More questions answered translated into improved performance. Percent correct on the practice questions also correlated strongly with performance. Programs seeking to improve scores may wish to provide access to a question bank.

A Structured Remediation Program Results in Durable Improvement of American Board of Surgery In-Training Examination (ABSITE®) Performance

The objective of this study was to assess the results of a structured remediation program for surgical residents scoring poorly on the American Board of Surgery In-Training Examination®. Residents scoring below the threshold (<20th percentile) were placed in a remediation program consisting of textbook-based review and mandatory weekly quizzes overseen by the program director. The effects of the program were assessed over a five-year period (2014-2018). Of 168 tests taken, 23 (13.7%) instances of scoring below the threshold occurred (average percentile 11, range 1–19). Postremediation scores in the subsequent year improved with a median (IQR) percentile increase of 29 [23.5, 46] ( P < 0.001). Two residents scored below the threshold twice; all others continued to score above the threshold. Participation in a structured remediation program results in dramatic improvement in American Board of Surgery In-Training Examination® performance, and a single occurrence of poor performance does not necessarily lead to a permanent handicap in academic performance.

Best Practice for Implementation of the SCORE Portal in General Surgery Residency Training Programs

INTRODUCTION

The Surgical Council on Resident Education (SCORE) has presented a workshop annually at the annual meeting of the Association of Program Directors (APDS) to discuss the evolution of the SCORE portal and best practices for implementation within residency training programs.

METHODS/RESULTS

A review of the literature was undertaken, along with a summation of discussion at these several workshops. A history of the SCORE project and a summary of its organizational framework and content are presented. In addition, best practices for use of SCORE within programs are described.

CONCLUSIONS

The SCORE portal is now a decade old, and is used ubiquitously in US surgical training programs. With this experience, there is data to show the utility of SCORE to support trainee learning and programmatic didactics.