Improving ABSITE scores - A meta-analysis of reported remediation models

American Board of Surgery In-Training Exam Performance Predicted by Question Bank Use While Unassociated With Other Learning Strategies

INTRODUCTION

There is no consensus regarding optimal curricula to teach cognitive elements of general surgery. The American Board of Surgery In-Training Exam (ABSITE) aims to measure trainees' progress in attaining this knowledge. Resources like question banks (QBs), Surgical Council on Resident Education (SCORE) curriculum, and didactic conferences have mixed findings related to ABSITE performance and are often evaluated in isolation. This study characterized relationships between multiple learning methods and ABSITE performance to elucidate the relative educational value of learning strategies.

METHODS

Use and score of QB, SCORE use, didactic conference attendance, and ABSITE percentile score were collected at an academic general surgery residency program from 2017 to 2022. QB data were available in the years 2017-2018 and 2021-2022 during institutional subscription to the same platform. Given differences in risk of qualifying exam failure, groups of ≤30th and >30th percentile were analyzed. Linear quantile mixed regressions and generalized linear mixed models determined factors associated with ABSITE performance.

RESULTS

Linear quantile mixed regressions revealed a relationship between ABSITE performance and QB questions completed (1.5 percentile per 100 questions, P < 0.001) and QB score (1.2 percentile per 1% score, P < 0.001), but not with SCORE use and didactic attendance. Performers >30th percentile had a significantly higher QB score.

CONCLUSIONS

Use and score of QB had a significant relationship with ABSITE performance, while SCORE use and didactic attendance did not. Performers >30th percentile completed a median 1094 QB questions annually with a score of 65%. Results emphasize success of QB use as an active learning strategy, while passive learning methods warrant further evaluation.

The Plan-Do-Study-Act (PDSA): An Iterative Approach to Optimize Residents Performance in the American Board of Surgery in-Training Exam (ABSITE)

INTRODUCTION

American Board of Surgery (ABS) In-Training Examination (ITE), or ABSITE, preparation requires an effective study approach. In 2014, the ABS announced the alignment of ABSITE to the SCORE® Curriculum. We hypothesized that implementing a Plan-Do-Study-Act (PDSA) approach would help surgery residents improve their performance on the ABSITE.

METHOD

Over 20 years, in a single institution, residents' ABSITE performance was evaluated over 3 timeframes: Time A (2004-2013), no specific curriculum; Time B (2014-2019), an annual comprehensive ABSITE-simulated SCORE®-based multiple-choice exam (MCQ) was administered; and Time C (2020-2023), like Time B with the addition of the PDSA approach for those with less than 60% correct on the ABSITE-simulated SCORE®-based exam. At the beginning of the academic year, in July, all residents are encouraged to (1) initiate a study plan for the upcoming ABSITE using SCORE® guided by the published ABSITE outlines content topics (Plan), (2) take an ABSITE-simulated SCORE®-based exam in October (Do), (3) assess the results/scores (Study), and (4) identify appropriate next steps (Act). Correlational analysis was performed to evaluate the association between ABSITE scores and ABSITE-simulated SCORE®-based exam scores in Time B and Time C. The primary outcome was the change in the proportions of ABSITE scores <30th percentile.

RESULTS

A total of 294 ABSITE scores of 94 residents (34 females and 60 males) were analyzed. We found stronger correlation between the correct percentage on ABSITE and ABSITE-simulated SCORE®-based exam scores in Time C (r = 0.73, p < 0.0001) compared to Time B (0.62, p < 0.0001). The percentage of residents with ABSITE scores lower than 30th percentile dropped significantly from 14.0% to 3.7% (p = 0.016).

CONCLUSION

Implementing the Plan-Do-Study-Act (PDSA) approach using the SCORE® curriculum significantly enhances residents' performance on the ABSITE exam. Surgery residents are encouraged to use this approach and to utilize the SCORE-contents outlined by the ABS in their study plan.

Does Game-Based Learning Promote Academic Engagement Among Millennial Oral and Maxillofacial Surgery Residents?

BACKGROUND

House staff participating in surgical residency programs are often comprised of adult learners from the millennial generation (Gen Y). With the increasing gap in both age and learning styles between these residents and their educators, their educators are struggling to keep them academically engaged.

PURPOSE

The purpose of this study was to measure the association between implementing a digital online Jeopardy! (Sony Pictures Studios, Culver City, CA) style game (digital game) (Factile Solace Creative, LLC, Melbourne Beach, FL) into oral maxillofacial surgery (OMS) resident didactic training and Oral and Maxillofacial Surgery In-service Training Examination (OMSITE) scores.

STUDY DESIGN, SETTING, SAMPLE

This study was a retrospective cohort study conducted at The New York Presbyterian Hospital/Weill Cornell Medicine. OMS residents who took the OMSITE examination were included in the study. Exclusion criteria included OMS residents who did not take the OMSITE in a given year.

INDEPENDENT VARIABLE

The independent variable was the timing of the implementation of the digital game, divided into before and after implementation.

MAIN OUTCOME VARIABLES

The primary outcome variable was OMSITE scores.

COVARIATES

Covariates included demographics (age, gender, and race), year of training, and number of examination attempts.

ANALYSES

Descriptive and bivariate statistics included Fisher's exact test and Wilcoxon rank sum test. Linear mixed modeling was performed to take into account multiple observations per participants.

RESULTS

The sample composed of 14 resident subjects with a median age of 27.5 (27.0, 28.7) years. Females represented 29% of the cohort. Seventy-one percent of the participants were Post Graduate Year 1. The median (interquartile range) OMSITE score preimplementation: 61 (48, 71) and postimplementation: 79 (68, 87). On average, the OMSITE scores improved significantly following the implementation of this study modality (P < .001).

CONCLUSION AND RELEVANCE

Digital games may be an effective learning tool to prepare residents for OMSITE by promoting academic engagement.

National ABSITE Preparation Practices: A Survey of General Surgery Residency Program Directors

OBJECTIVE

The American Board of Surgery In-Training Examination (ABSITE) was designed as a low-stakes, medical knowledge examination for US general surgery residency programs. However, in practice, this exam has been utilized for higher stakes purposes, such as resident promotion or remediation, and fellowship selection. Several studies have demonstrated the efficacy of ABSITE preparation resources, but best practices for ABSITE preparation and national preparatory habits are currently unknown. The aim of this work was to determine current residency programs' strategies for ABSITE preparation.

DESIGN

We distributed an electronic survey to program directors or program coordinators of US general surgery programs asking them to anonymously report program ABSITE educational practices and ABSITE scores. We analyzed the proportion of responses using descriptive statistics and compared the effect of various strategies using the Mann-Whitney testing for nonparametric data. An average ABSITE percentile score was calculated for each residency based on program self-reported scores.

SETTING

Association of Program Directors (APDS) Listserv PARTICIPANTS: General surgery residency programs participating at the time of distribution (278).

RESULTS

Response rate was 24% (66/278); 41 programs (62.1%) identified as university-affiliated, and 25 (37.9%) were community-based. Median intern class size was 8 (range: 3-14), including preliminary interns. Average ABSITE percentile score was 52.8% (range 36.9%-67.6%). There were no significant differences in ABSITE scores based on affiliation or program size. Educational resources utilized for ABSITE preparation included SCORE (89.3%), Q-banks (50%), and surgical textbooks (25.8%). The majority (56.1%) of programs reported using a year-long curriculum for ABSITE preparation, and 66.6% used a time-limited curriculum completed in the months immediately prior to ABSITE. Most programs reported that ABSITE scores were a low priority (63.6%) or not a priority (13.6%). The existence of a year-long curriculum for ABSITE was positively correlated with score as compared to programs without a year-long curricula (53.9% vs 48.5%, p <0.01). Programs using a time-limited curriculum demonstrated lower scores as compared to programs without time-limited curricula (51.3% v 56.1%, p < 0.01).

CONCLUSION

General surgery programs use a variety of strategies to prepare residents for the ABSITE. Despite reporting that they utilize ABSITE scores for a variety of high stakes purposes including evaluation for promotion and as a predictor of the preparedness for the ABS QE, many programs reported that they consider ABSITE scores as a low priority. A year-long focused curriculum was the only strategy correlated with increased scores, which may reflect the value of encouraging consistent studying and spaced repetition. Additional work is needed to guide programs in optimal utilization of ABSITE scores for remediation and resident evaluation, as well as understanding how ABSITE preparatory strategies correlate with clinical performance.

Just Do It: Participation in Structured Online Curricula Reliably Improves Low ABSITE Scores

OBJECTIVE

We hypothesized residents enrolled in an Accelerated Clinical Education in Surgery (ACES) program would improve their scores to above the 30^th percentile. We analyzed which components of ACES correlated with improvement.

DESIGN AND SETTING

A retrospective review of three academic cycles (2018-2021) at an academic general surgery residency.

PARTICIPANTS

Residents scoring ≤30^th percentile on the ABSITE were enrolled in ACES. Baseline demographics including STEP scores were collected. ACES included: (1) SCORE and DeckerMed assignments (2) Weekly faculty review sessions and (3) Monthly meeting with assigned mentor. Data were analyzed by Student's t-test, one-way ANOVA and Fisher's exact test.

RESULTS

Twenty-six surgical residents enrolled in ACES. Compared to residents not in ACES, there was no significant difference females (15 vs. 15; p = 0.19) and STEP 2 scores (241 vs. 246; p = 0.06). Residents in ACES had significantly lower STEP 1 (225 vs. 237; p < 0.001) and STEP 3 (212 vs. 223; p < 0.001) scores. Demographics of ACES residents who subsequently scored >30^th percentile were similar to those who didn't, except for STEP 3 scores (216 vs. 204; p = 0.008). For residents in ACES, the completion of assignments between July and January was significantly higher for those who subsequently achieved an ABSITE score >30^th percentile: TWIS, 77% vs. 53% (p = 0.022), Decker WC, 80% vs. 49% (p = 0.009) and Decker MR, 53% vs. 29% (p = 0.016). Completion of an online practice exam prior to ABSITE also correlated with score >30^th percentile (57% vs. 13%, p = 0.007). There was also no correlation between the number of faculty review sessions and ABSITE (11.5 vs.11.9, p = 0.931).

CONCLUSIONS

Participation in a structured online program of reading and quizzes was durably effective in improving ABSITE scores >30^th percentile. Completion of online assignments, rather than scores on practice tests or review sessions, appeared to be the most important factor for success.

Prospective Factors that Predict American Board of Surgery In-Training Examination Performance: A Systematic Review

INTRODUCTION

American Board of Surgery In-Training Examination (ABSITE) performance has become an important factor when monitoring resident progress. Understanding which prospective factors predict performance can help identify residents at risk.

METHODS

A literature search was conducted searching PubMed, EMBASE, and JAMA Network from June 2011 to June 2021, in accordance with the PRISMA guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination." Prospective factors such as prior examination performance, clinical evaluations, and demographics were evaluated.

RESULTS

A final 35 studies were included. The prospective factor most consistently found to predict ABSITE performance is performance on prior knowledge-based examinations such as the USMLE step exams. The ACGME Medical Knowledge 1 milestone evaluation also appears to correlate to ABSITE performance, although clinical evaluations, in general, do not. Demographics have no significant correlation to ABSITE performance.

DISCUSSION

Using performance on prior knowledge-based examinations programs may be able to identify residents at risk for failing ABSITE. It may be possible to initiate early intervention before rather than only remediation after poor performance.

Early Detection and Remediation of Problem Learners

There are myriad types of problem learners in surgical residency and most have difficulty in more than 1 competency. Programs that use a standard curriculum of study and assessment are most successful in identifying struggling learners early. Many problem learners lack appropriate systems for study; a multidisciplinary educational team that is separate from the team that evaluates the success of remediation is critical. Struggling residents who require formal remediation benefit from performance improvement plans that clearly outline the issues of concern, describe the steps required for remediation, define success of remediation, and outline consequences for failure to remediate appropriately.