The utility of the ABS in-training examination (ABSITE) score forms: percent correct and percentile score in the assessment of surgical residents

The Plan-Do-Study-Act (PDSA): An Iterative Approach to Optimize Residents Performance in the American Board of Surgery in-Training Exam (ABSITE)

INTRODUCTION

American Board of Surgery (ABS) In-Training Examination (ITE), or ABSITE, preparation requires an effective study approach. In 2014, the ABS announced the alignment of ABSITE to the SCORE® Curriculum. We hypothesized that implementing a Plan-Do-Study-Act (PDSA) approach would help surgery residents improve their performance on the ABSITE.

METHOD

Over 20 years, in a single institution, residents' ABSITE performance was evaluated over 3 timeframes: Time A (2004-2013), no specific curriculum; Time B (2014-2019), an annual comprehensive ABSITE-simulated SCORE®-based multiple-choice exam (MCQ) was administered; and Time C (2020-2023), like Time B with the addition of the PDSA approach for those with less than 60% correct on the ABSITE-simulated SCORE®-based exam. At the beginning of the academic year, in July, all residents are encouraged to (1) initiate a study plan for the upcoming ABSITE using SCORE® guided by the published ABSITE outlines content topics (Plan), (2) take an ABSITE-simulated SCORE®-based exam in October (Do), (3) assess the results/scores (Study), and (4) identify appropriate next steps (Act). Correlational analysis was performed to evaluate the association between ABSITE scores and ABSITE-simulated SCORE®-based exam scores in Time B and Time C. The primary outcome was the change in the proportions of ABSITE scores <30th percentile.

RESULTS

A total of 294 ABSITE scores of 94 residents (34 females and 60 males) were analyzed. We found stronger correlation between the correct percentage on ABSITE and ABSITE-simulated SCORE®-based exam scores in Time C (r = 0.73, p < 0.0001) compared to Time B (0.62, p < 0.0001). The percentage of residents with ABSITE scores lower than 30th percentile dropped significantly from 14.0% to 3.7% (p = 0.016).

CONCLUSION

Implementing the Plan-Do-Study-Act (PDSA) approach using the SCORE® curriculum significantly enhances residents' performance on the ABSITE exam. Surgery residents are encouraged to use this approach and to utilize the SCORE-contents outlined by the ABS in their study plan.

Performance of trauma-trained large language models on surgical assessment questions: A new approach in resource identification

BACKGROUND

Large language models have successfully navigated simulated medical board examination questions. However, whether and how language models can be used in surgical education is less understood. Our study evaluates the efficacy of domain-specific large language models in curating study materials for surgical board style questions.

METHODS

We developed EAST-GPT and ACS-GPT, custom large language models with domain-specific knowledge from published guidelines from the Eastern Association of the Surgery of Trauma and the American College of Surgeons Trauma Quality Programs. EAST-GPT, ACS-GPT, and an untrained GPT-4 performance were assessed trauma-related questions from Surgical Education and Self-Assessment Program (18th edition). Large language models were asked to choose answers and provide answer rationales. Rationales were assessed against an educational framework with 5 domains: accuracy, relevance, comprehensiveness, evidence-base, and clarity.

RESULTS

Ninety guidelines trained EAST-GPT and 10 trained ACS-GPT. All large language models were tested on 62 trauma questions. EAST-GPT correctly answered 76%, whereas ACS-GPT answered 68% correctly. Both models outperformed ChatGPT-4 (P < .05), which answered 45% correctly. For reasoning, EAST-GPT achieved the gratest mean scores across all 5 educational framework metrics. ACS-GPT scored lower than ChatGPT-4 in comprehensiveness and evidence-base; however, these differences were not statistically significant.

CONCLUSION

Our study presents a novel methodology in identifying test-preparation resources by training a large language model to answer board-style multiple choice questions. Both trained models outperformed ChatGPT-4, demonstrating its answers were accurate, relevant, and evidence-based. Potential implications of such AI integration into surgical education must be explored.

American Board of Surgery In-Training Exam Performance Predicted by Question Bank Use While Unassociated With Other Learning Strategies

INTRODUCTION

There is no consensus regarding optimal curricula to teach cognitive elements of general surgery. The American Board of Surgery In-Training Exam (ABSITE) aims to measure trainees' progress in attaining this knowledge. Resources like question banks (QBs), Surgical Council on Resident Education (SCORE) curriculum, and didactic conferences have mixed findings related to ABSITE performance and are often evaluated in isolation. This study characterized relationships between multiple learning methods and ABSITE performance to elucidate the relative educational value of learning strategies.

METHODS

Use and score of QB, SCORE use, didactic conference attendance, and ABSITE percentile score were collected at an academic general surgery residency program from 2017 to 2022. QB data were available in the years 2017-2018 and 2021-2022 during institutional subscription to the same platform. Given differences in risk of qualifying exam failure, groups of ≤30th and >30th percentile were analyzed. Linear quantile mixed regressions and generalized linear mixed models determined factors associated with ABSITE performance.

RESULTS

Linear quantile mixed regressions revealed a relationship between ABSITE performance and QB questions completed (1.5 percentile per 100 questions, P < 0.001) and QB score (1.2 percentile per 1% score, P < 0.001), but not with SCORE use and didactic attendance. Performers >30th percentile had a significantly higher QB score.

CONCLUSIONS

Use and score of QB had a significant relationship with ABSITE performance, while SCORE use and didactic attendance did not. Performers >30th percentile completed a median 1094 QB questions annually with a score of 65%. Results emphasize success of QB use as an active learning strategy, while passive learning methods warrant further evaluation.

Just Do It: Participation in Structured Online Curricula Reliably Improves Low ABSITE Scores

OBJECTIVE

We hypothesized residents enrolled in an Accelerated Clinical Education in Surgery (ACES) program would improve their scores to above the 30^th percentile. We analyzed which components of ACES correlated with improvement.

DESIGN AND SETTING

A retrospective review of three academic cycles (2018-2021) at an academic general surgery residency.

PARTICIPANTS

Residents scoring ≤30^th percentile on the ABSITE were enrolled in ACES. Baseline demographics including STEP scores were collected. ACES included: (1) SCORE and DeckerMed assignments (2) Weekly faculty review sessions and (3) Monthly meeting with assigned mentor. Data were analyzed by Student's t-test, one-way ANOVA and Fisher's exact test.

RESULTS

Twenty-six surgical residents enrolled in ACES. Compared to residents not in ACES, there was no significant difference females (15 vs. 15; p = 0.19) and STEP 2 scores (241 vs. 246; p = 0.06). Residents in ACES had significantly lower STEP 1 (225 vs. 237; p < 0.001) and STEP 3 (212 vs. 223; p < 0.001) scores. Demographics of ACES residents who subsequently scored >30^th percentile were similar to those who didn't, except for STEP 3 scores (216 vs. 204; p = 0.008). For residents in ACES, the completion of assignments between July and January was significantly higher for those who subsequently achieved an ABSITE score >30^th percentile: TWIS, 77% vs. 53% (p = 0.022), Decker WC, 80% vs. 49% (p = 0.009) and Decker MR, 53% vs. 29% (p = 0.016). Completion of an online practice exam prior to ABSITE also correlated with score >30^th percentile (57% vs. 13%, p = 0.007). There was also no correlation between the number of faculty review sessions and ABSITE (11.5 vs.11.9, p = 0.931).

CONCLUSIONS

Participation in a structured online program of reading and quizzes was durably effective in improving ABSITE scores >30^th percentile. Completion of online assignments, rather than scores on practice tests or review sessions, appeared to be the most important factor for success.

Modern Didactic Formats in Surgery: A Systematic Review

Introduction: "Traditional teaching" models often fail to engage millennial residents. Multiple modern didactic methods have been employed. The most frequently used objective measure to assess the effectiveness of didactic formats has been American Board of Surgery In-Training Examination performance.Methods: A literature search was conducted searching PubMed, EMBASE, and JAMA Network from June 2011 to June 2021, in accordance with the PRISMA guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination." Only studies discussing didactic structures were included.Results: A final 16 studies were included. Modern methods such as a "flipped classroom," Team Based Learning (TBL), and "gamification" have all shown increased engagement and significantly improved ABSITE performance. Structured biostatistics reviews may be used to supplement research and statistics which are often missed by other resources.Discussion: Programs have a duty to promote excellent resident education. In addition to fostering individual study habits, didactics and program structures should be optimized for resident development. As opposed to focusing on the pure amount of scheduled protected time, programs may instead consider focusing on the quality of the didactic format used and modern didactic methods may be beneficial.

How to Prepare for the American Board of Surgery In-Training Examination (ABSITE): A Systematic Review

INTRODUCTION

Performance on ABSITE is an important factor when monitoring resident progress. It predicts future performance and has lasting effects. Understanding the highest-yield preparation strategies can help residents in their study efforts and optimize performance.

METHODS

A literature search was conducted searching PubMed, EMBASE and JAMA Network in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination". Only studies discussing individual study habits from May 2011 to May 2021 were included.

RESULTS

19 studies were included in qualitative synthesis. Year-round clinical study failed to show significant correlation to ABSITE performance although year-round ABSITE review was more consistently correlated. During a dedicated study period, increased time and increased total practice questions completed are associated with improved performance. The correlation of individual resources such as ABSITE review books, textbooks, audio podcasts and ABSITE preparatory courses to improved ABSITE performance was not proven.

CONCLUSIONS

Residents should optimize study strategies based on methods that have consistently shown to improve performance. Recommendations for best preparation strategies are provided.

Prospective Factors that Predict American Board of Surgery In-Training Examination Performance: A Systematic Review

INTRODUCTION

American Board of Surgery In-Training Examination (ABSITE) performance has become an important factor when monitoring resident progress. Understanding which prospective factors predict performance can help identify residents at risk.

METHODS

A literature search was conducted searching PubMed, EMBASE, and JAMA Network from June 2011 to June 2021, in accordance with the PRISMA guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination." Prospective factors such as prior examination performance, clinical evaluations, and demographics were evaluated.

RESULTS

A final 35 studies were included. The prospective factor most consistently found to predict ABSITE performance is performance on prior knowledge-based examinations such as the USMLE step exams. The ACGME Medical Knowledge 1 milestone evaluation also appears to correlate to ABSITE performance, although clinical evaluations, in general, do not. Demographics have no significant correlation to ABSITE performance.

DISCUSSION

Using performance on prior knowledge-based examinations programs may be able to identify residents at risk for failing ABSITE. It may be possible to initiate early intervention before rather than only remediation after poor performance.

SCORE-Based Simulated ABSITE Exam Performance as a Predictor of Performance on the ABSITE

INTRODUCTION

The American Board of Surgery In-Training Examination (ABSITE) is a crucial, objective assessment of surgical knowledge during training. In 2014, the American Board of Surgery (ABS) announced the alignment of the ABSITE to the SCORE® (Surgical Council on Resident Education) Curriculum Outline for General Surgery Residency. We hypothesized that implementing a pre-ABSITE SCORE-based exam would help identify underperforming residents and provide early guidance to improve performance on the ABSITE.

METHODS

In October 2014, our university-based surgical residency program began administering a yearly comprehensive pre-ABSITE SCORE-based exam consisting of 225 to 250 multiple-choice questions selected from the SCORE question bank to all our general surgery residents, preliminary and categorical. The 4-hour exam addresses both clinical management (80%) and applied sciences (20%). Residents receive reports with their scores (percentage correct). Residents performing at less than 60% meet with the Program Director for discussion and formulation of a study plan. Correlational analysis was performed between resident ABSITE scores, pre-ABSITE SCORE-based exam scores, gender, resident status (preliminary vs. categorical), postgraduate year (PGY), and the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores.

RESULTS

A total of 244 exam scores (122 pre-ABSITE SCORE-based exams and 122 matched ABSITE) were completed by 51 residents at different PGY levels (32 PGY1, 32 PGY2, 20 PGY3, 19 PGY4, and 18 PGY5). Fifty-seven percent were males, 62% were categorical residents, and 38% were preliminary residents. October pre-ABSITE SCORE-based exam scores were compared to the subsequent January ABSITE scores. Categorical residents completed 101 (83%) of the January exams, while preliminary residents completed 21 (17%) of these paired exams. We found strong correlations between the correct percentage on ABSITE and pre-ABSITE SCORE-based scores (r = 0.637, p < 0.001), between the correct percentage on ABSITE and PGY (r = 0.688, p < 0.001), and between ABSITE and resident status (r = 0.462, p < 0.001). Additionally, there was a weak to negligible correlation between the correct percentage on ABSITE and resident gender (r = 0.274, p = 0.001), USMLE-2 (r = 0.12, p = 0.16), and USMLE-1 (r = 0.04, p = 0.653). Multiple regression analysis, with all predictors, was performed to predict the percentage score on the ABSITE and produced R² 0.58, with an adjusted R² of 0.57, with a large size effect, p < 0.001. After controlling for the other variables, three factors reached statistical significance (p < 0.05): pre-ABSITE SCORE-based exam scores, PGY, and resident gender.

CONCLUSION

We found a strong correlation between performance on the pre-ABSITE SCORE-based exam and performance on the ABSITE exam. Surgery residents are encouraged to start studying earlier and to utilize SCORE contents as outlined by the ABS in their study plan.

Predictive Factors of First Time Pass Rate on the American Board of Surgery Certification in General Surgery Exams: A Systematic Review

OBJECTIVE

General Surgery residency programs are evaluated on their American Board of Surgery (ABS) Qualifying examination (QE) and Certifying examination (CE) pass rates. This systematic review aims to evaluate predictive factors of ABS QE and CE first time pass rates.

DESIGN

Using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines, the following electronic databases were searched: PubMed, Embase, JAMA Network, and Google Scholar. Studies available in the English language published between January 2000 and July 2020 were deemed eligible. Articles that did not assess either of the ABS board examinations performance and pass-rates as outcomes were excluded. The Oxford Centre for Evidence-Based Medicine was used to determine the quality and risk of bias of each study.

RESULTS

A total of 31 publications were included for analysis. Undergraduate medical education variables associated with first-time pass rates on the QE and CE include USMLE score, AOA membership, and class rank. Program factors affecting pass rates include program size, geographic location, and Program Director turnover. There is strong correlation between ABSITE and QE. Evidence supports the utility of mock oral examinations (MOEs) in predicting CE success.

CONCLUSIONS

ABSITE scores demonstrated higher correlation with QE pass rate than CE pass rate. MOEs have a positive association with first-time CE pass rates. Nonmodifiable factors such as race/ethnicity, marital status, and geographic location were also found to be predictors. Delaying board certification examination beyond 1 year after residency graduation significantly reduces first-time pass rate.

Early Detection and Remediation of Problem Learners

There are myriad types of problem learners in surgical residency and most have difficulty in more than 1 competency. Programs that use a standard curriculum of study and assessment are most successful in identifying struggling learners early. Many problem learners lack appropriate systems for study; a multidisciplinary educational team that is separate from the team that evaluates the success of remediation is critical. Struggling residents who require formal remediation benefit from performance improvement plans that clearly outline the issues of concern, describe the steps required for remediation, define success of remediation, and outline consequences for failure to remediate appropriately.

Value of Standardized Testing in Surgical Training

Standardized testing remains a cornerstone of assessment in surgical education. Summative standardized tests make up a bulk of the certification requirements that encompasses demonstration of efficient, safe application of clinically relevant surgical knowledge and skills. Formative standardized tests serve similar role to guide teaching endeavors for the programs and comparison of individual trainees on a national level. Ongoing rigorous psychometric evaluations of the standardized tests ensure reliability and validity; however, standardized tests are not without their limitations and biases.

Learning to Read: Successful Program-Based Remediation Using the Surgical Council on Resident Education (SCORE) Curriculum

BACKGROUND

The Surgical Council on Resident Education (SCORE) curriculum is aligned with the American Board of Surgery (ABS) objectives. Our program adopted the SCORE curriculum in 2015 after poor ABS In-Training Examination (ABSITE) performance and lowest quartile ABS Certifying Exam (CE) and Qualifying Exam (QE) first-time pass rates. We examined the association of SCORE use with ABSITE performance and ABS board exam first-time pass rate.

STUDY DESIGN

At a single institution, a retrospective review of surgery residents' SCORE metrics and ABSITE percentile was conducted for academic years 2015 to 2019. Metrics analyzed on the SCORE web portal were mean total minutes and total visits per resident for all residents using SCORE that year. First-time pass rates of the ABS QE and CE were examined from 2013 to 2019. Chi-square and linear regression analysis were performed, and a 95% level of confidence was assumed (alpha = 0.05).

RESULTS

Yearly data from categorical general surgery residents showed a significant increase in total minutes, total visits, and ABSITE percentile. Combined first time pass rates for the ABS QE and CE significant increased from 70.8% in 2013 to 2015 to 93.9% in 2016 to 2019 (p = 0.018).

CONCLUSIONS

Increased longitudinal use of the SCORE curriculum was associated with programmatic improvements in ABSITE performance and ABS board exam first-time pass rate.

Program Directors’ Perspective of Transplant Training during General Surgery Residency

In 2011, the Accreditation Council for Graduate Medical Education (ACGME) began to mandate “a formal transplant experience” rather than a formal rotation to allow programs more flexibility. What constitutes a “transplant experience” or how to optimize such education remains unclear. The objectives of this study were to identify changes in residents’ transplant exposure and determine the opinions of program directors (PDs) in response to the new requirements. A 15-item questionnaire was sent to all general surgery PDs. Demographics, resident experiences, and PDs’ opinions regarding transplant rotations were collected. The response rate was 50 per cent from university and community programs at transplant and nontransplant centers. Although 55 per cent of PDs felt transplant should not be a required rotation, 92 per cent of programs reported having a formal rotation. Transplant rotations should be four months or less, according to 97 per cent of PDs. Most PDs (93%) think there should not be a minimum number of required transplant operations. Whether residencies were based at transplant centers or size of residencies did not affect PDs’ responses. Transplant requirements were similar between residency programs regardless of PD opinions. Survey participants provided diverse feedback regarding this controversial topic. The new ACGME requirements have made little impact on program requirements. Most programs still mandate a formal transplant rotation. Diverse opinions exist regarding whether such a rotation should be required, highlighting the controversial nature of this issue. To facilitate education in an era of limited work hours, the ACGME changed requirements of general surgery programs to mandate “a formal transplant experience” rather than a transplant rotation. A survey of program directors revealed that although only 45 per cent of support mandatory transplant rotations, 93 per cent of programs still require transplant rotations despite the more lax ACGME requirements. This finding suggests a lack of suitable alternative to provide exposure to the concepts fundamental to transplant surgery, which are valuable in general surgery education.

Mid-Year Medical Knowledge Milestones and ABSITE Scores in First-Year Surgery Residents

OBJECTIVE

Accreditation Council for Graduate Medical Education (ACGME) Surgery milestone ratings in the "Knowledge of Diseases and Conditions" (MK1) sub competency have been shown to correlate with American Board of Surgery In Training Examination (ABSITE) scores, and hypothesized to predict them. To better assess the predictive value of the MK1 milestone and avoid the potential bias caused by previous years' ABSITE scores, we designed a study including only first-year (PGY-1) residents and analyzed the correlation between their mid-year MK1 ratings and their scores in the ABSITE they took approximately a month later.

METHODS

De-identified United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores, mid-year MK1 milestone ratings and the subsequent ABSITE standard scores for the five academic years from 2014-2015 to 2018-2019 were collected and tabulated for 247 PGY-1 preliminary- and categorical-track residents from ten ACGME-accredited surgery residency programs.

RESULTS

The mid-year rating of PGY-1 residents' MK1 was predictive of their subsequent first ABSITE score for the entire cohort and for the categorical residents' subset. Notably, controlling for all other independent predictors, each half-point increase in MK1 rating was associated with a 25-point increase in ABSITE score. Preliminary residents performed significantly worse on the ABSITE, and their scores did not correlate significantly with their MK1 ratings.

CONCLUSIONS

The mid-year rating of PGY-1 residents' MK1 was predictive of their subsequent first ABSITE score for the entire cohort and for the categorical but not the preliminary residents. This finding suggests that evaluators correctly rated MK1 higher in the categorical residents who did perform better on the subsequent ABSITE.

General Surgery Resident Use of Electronic Resources: 15 Minutes a Day

BACKGROUND

General surgery resident performance on the American Board of Surgery In-Service Training Exam (ABSITE) has been used to predict American Board of Surgery (ABS) passage rates, selection for remediation programs, and ranking of fellowship applicants. We sought to identify electronic resource study habits of general surgery residents associated with successful test scores.

STUDY DESIGN

A single-institution, retrospective review of general surgery resident use of 2 electronic study resources, Surgical Council on Resident Education (SCORE) and TrueLearn (TL), were evaluated for the 7 months before the 2019 ABSITE. Metrics included TL question performance, SCORE use, and a survey about other reading sources. These metrics were evaluated in 3 ABSITE percentile groupings: ≥80^th, 31^st to 79^th, and ≤30^th.

RESULTS

The ≥80^th and 31^st to 79^th percentile groups scored higher on TL questions, at 69% and 67.7%, respectively, compared with 61.4% for the ≤30^th percentile group (p < 0.03). The ≥80^th percentile group spent on average 14.6 minutes/day on SCORE compared with 5.0 minutes/day and 4.7 minutes/day for the 31^st to 79^th and ≤30^th percentile groups, respectively (p < 0.04). The ≥80^th percentile group spent 34.8 minutes/session (77 sessions) compared with 19.2 minutes/session (49 sessions) and 20.7 minutes/session (43 sessions) in the 31^st to 79^th and ≤30^th percentile groups, respectively (p = 0.009).

CONCLUSIONS

Our nomogram incorporates time spent accessing an electronic content-based resource, SCORE, and performance on an electronic question-based resource as a novel method to provide individualized feedback and predict future ABSITE performance.

The Impact of a Comprehensive Resident Curriculum and Required Participation in "This Week in SCORE" on General Surgery ABSITE Performance and Well-Being

OBJECTIVE

This Week in SCORE (TWIS) is a biennial, general surgery curriculum comprised of weekly online modules, readings, and multiple-choice quizzes. In this study, we examined the impact of required TWIS on American Board of Surgery In-Training Examination (ABSITE) scores and well-being among categorical general surgery residents.

DESIGN, SETTING, AND PARTICIPANTS

TWIS quiz completion became required in 2017. Residents attended weekly lectures, and ABSITE performance was incentivized with educational stipends. Surveys were distributed to assess study preferences, learning styles, burnout, and grit. Thirty-six categorical general surgery residents who took ABSITE in both 2017 and 2018 were evaluated in a paired-sample, retrospective analysis.

RESULTS

After requiring TWIS, median ABSITE percentile increased by 12% (65%-77%, p = 0.001). Weekly TWIS completion (59% vs 89%, p < 0.001) and quiz results (62% vs 69%, p = 0.005) also improved. During this time, emotional exhaustion and depersonalization declined significantly, yet overall burnout scores did not change. Of 21 survey respondents, 66.7% (n = 14) increased weekly study time by a median of 2.5 hours. However, less than half used Surgical Council on Resident Education as their primary study tool. Only 23.8% (n = 5) reported that mandatory TWIS modified their study behavior, while 90.4% (n = 19) felt the culture of education had improved.

CONCLUSIONS

After TWIS participation became required, ABSITE performances improved. Formalized curriculum with frequent assessment may foster accountability among residents, enhancing educational climate, well-being, and test performance.

A Structured Remediation Program Results in Durable Improvement of American Board of Surgery In-Training Examination (ABSITE®) Performance

The objective of this study was to assess the results of a structured remediation program for surgical residents scoring poorly on the American Board of Surgery In-Training Examination®. Residents scoring below the threshold (<20th percentile) were placed in a remediation program consisting of textbook-based review and mandatory weekly quizzes overseen by the program director. The effects of the program were assessed over a five-year period (2014-2018). Of 168 tests taken, 23 (13.7%) instances of scoring below the threshold occurred (average percentile 11, range 1–19). Postremediation scores in the subsequent year improved with a median (IQR) percentile increase of 29 [23.5, 46] ( P < 0.001). Two residents scored below the threshold twice; all others continued to score above the threshold. Participation in a structured remediation program results in dramatic improvement in American Board of Surgery In-Training Examination® performance, and a single occurrence of poor performance does not necessarily lead to a permanent handicap in academic performance.

Factors that Predict an Intern's First ABSITE Score are Known by September

BACKGROUND

Previous studies offer conflicting relevance of a variety of factors to predict resident performance on the The American Board of Surgery In-Training Exam (ABSITE). With numerous stellar applicants scoring poorly on their first ABSITE, we sought to identify key factors that might allow us to tailor pre-emptive study efforts in the fall and early winter to enhance scores.

METHODS

General Surgery residents in our program from 2009 through 2016 were included in our cohort study. Specific trainee data (sex, prelim vs categorical, United States Medical Licensing Examination (USMLE) Step 1 and 2 scores, ABSITE scores, clinical rotations, biannual objective structured clinical examination OSCE-type scores, and in-house prep test [IHPT], etc.) were collected retrospectively. The data were analyzed using JMP pro 10 and MedCalc.

RESULTS

ABSITE scores of our 110 trainees did not vary by sex or by categorical vs preliminary residents. USMLE step 1 and 2, IHPT and one objective structured clinical examination (OSCE) station (Sim-Based Trauma Exam) scores were positively correlated with ABSITE scores (p < 0.05; correlation coefficient (CC) = 0.6 [strong] for IHPT, 0.5 [moderate] for step 1 and 2 and [weak] 0.3 for Sim-Based Trauma Exam). The mean (standard deviation) ABSITE %tile score for residents scoring above 230 (USMLE 1 or 2) vs <230 were different: 81 (3) vs 56 (4) (p < 000.1). Of residents scoring lower than 230 in USMLE 1 or 2, their ABSITE performance had a strong positive linear correlation with their performance in the IHPT (CC = 0.7) and SBTA (CC = 0.5). Residents rotating twice on the Acute Care Surgery services before the ABSITE scored higher than those with 1 or zero rotations (p < 0.05).

CONCLUSIONS

USMLE steps 1 and 2 are useful parameters in our program to predict subsequent resident ABSITE performance. An in-house 60-minute preparation test in September, a 6-minute simulation-based trauma assessment in July, and rotation schedule with 2 stints on Acute Care Surgery were surprisingly useful early intern year tools to predict ABSITE scores.

Performance on a Surgical In-Training Examination Varies by Training Year and Pathway

BACKGROUND

Few studies in surgery have addressed medical knowledge competency training as defined by the Accreditation Council for Graduate Medical Education. As in-training examinations are ubiquitous educational tools for surgical residents in the United States, insights into examination performance may help fill this void. The purpose of this study was to determine the relationship between In-Service Examination performance and training characteristics in plastic surgery.

METHODS

This retrospective cohort study reviewed performance data for the Plastic Surgery In-Service Training Examination for the years 2012 to 2015. Comparisons were made both within and between training pathways by means of Kruskal-Wallis and Mann-Whitney U tests.

RESULTS

Data were available for 1367 independent (37.9 percent) and 2240 integrated residents (62.1 percent). Among integrated residents, performance increased with additional years of training (p < 0.001), but no difference existed between postgraduate year-5 and postgraduate year-6 residents (p > 0.05). Similarly, independent resident examination performance increased by year of training (p < 0.001), with no difference between postgraduate year-2 and postgraduate year-3 residents (p > 0.05). At each level of training (postgraduate years 4 to 6), integrated residents outperformed their independent resident colleagues (postgraduate years 1 to 3) (p < 0.001).

CONCLUSIONS

Performance on the Plastic Surgery In-Service Training Examination increases during residency, with integrated residents outperforming independent residents. These findings may have implications for medical knowledge competency training as defined by the Accreditation Council for Graduate Medical Education.

Relevancy of an In-Service Examination for Core Knowledge Training in a Surgical Subspecialty

OBJECTIVE

To facilitate knowledge acquisition during plastic surgery residency, we analyzed the breast curriculum on the Plastic Surgery In-Service Training Exam (PSITE).

DESIGN

Breast-related questions on 6 consecutive PSITEs were analyzed (2008-2013). Topics were categorized by the content outline for the American Board of Plastic Surgery written board examination. Question vignettes were classified by taxonomy and clinical setting. References for correct answer choices were categorized by source and publication lag.

RESULTS

A total of 136 breast-related questions were analyzed (136/1174, 12%). Questions tended to appear more in the Breast and Cosmetic (75%) section than the Comprehensive (25%) section (p < 0.001). Most question vignettes were written in a clinical setting (64%, p < 0.001). Question taxonomy was evenly distributed among recall (34%), interpretation (28%), and decision-making (37%, p > 0.05). Only 6% of questions required photographic evaluation. Breast-related topics focused on esthetic problems (35%), traumatic deformities (22%), and tumors (21%). Answer references comprised 293 citations to 63 unique journals published a median of 6 years before PSITE administration. Plastic and Reconstructive Surgery (57%) was the most cited journal (p < 0.001) and Surgery of the Breast: Principles and Art by Spear was the most referenced textbook (22%).

CONCLUSIONS

The PSITE affords a curriculum that reflects breast-related topics on the American Board of Plastic Surgery written board examination. These data may optimize knowledge acquisition in esthetic and reconstructive breast surgery.

Interventions that affect resident performance on the American Board of Surgery In-Training Examination: a systematic review

OBJECTIVE

To systematically review the published literature on the effectiveness of interventions intended to improve residents' American Board of Surgery In-Training Examination (ABSITE) performances.

DESIGN

A systematic review was conducted by 2 independent investigators to identify all publications that examined the effect of specific interventions on residents' ABSITE performances from 1975 to 2013.

RESULTS

Overall, 26 published articles met study criteria. Structured reading programs and setting clear expectations with mandatory remedial programs were consistently effective in improving ABSITE performance, whereas the effect of didactic teaching conferences and problem-based learning groups was mixed. There was marked heterogeneity in the usage of study designs and reporting of results.

CONCLUSIONS

Structured reading programs and mandatory remedial programs appear to be consistently effective measures that can improve residents' ABSITE performances. There is a need for improved study design and reporting in future research conducted in this field.

Using the American Board of Surgery In-Training Examination to predict board certification: a cautionary study

OBJECTIVE

Although designed as a low-stakes formative examination, the American Board of Surgery In-Training Examination (ABSITE) is often used in high-stakes decisions such as promotion, remediation, and retention owing to its perceived ability to predict the outcome of board certification. Because of the discrepancy between intent and use, the ability of ABSITE scores to predict passing the American Board of Surgery certification examinations was analyzed.

METHODS

All first-time American Board of Surgery qualifying examination (QE) examinees between 2006 and 2012 were reviewed. Examinees' postgraduate year (PGY) 1 and PGY5 ABSITE standard scores were linked to QE scores and pass/fail outcomes (n = 6912 and 6846, respectively) as well as first-time certifying examination (CE) pass/fail results (n = 1329). Linear and logistic regression analyses were performed to evaluate the utility of ABSITE scores to predict board certification scores and pass/fail outcomes.

RESULTS

PGY1 ABSITE scores accounted for 22% of the variance in QE scores (p < 0.001). PGY5 scores were a slightly better predictor, accounting for 30% of QE score variance (p < 0.001). Analyses showed that selecting a PGY5 ABSITE score that maximized overall decision accuracy for predicting QE pass/fail outcomes (86% accuracy) resulted in 98% sensitivity, 13% specificity, a positive predictive value of 87%, and a negative predictive value of 57%. ABSITE scores were not predictive of success on the CE.

CONCLUSIONS

ABSITE scores are a useful predictor of QE scores and outcomes but do not predict passing the CE. Although scoring well on the ABSITE is highly predictive of QE success, using low ABSITE scores to predict QE failure results in frequent decision errors. Program directors and other evaluators should use additional sources of information when making high-stakes decisions about resident performance.

Surgery residency curriculum examination scores predict future American Board of Surgery in-training examination performance

IMPORTANCE

A protected block curriculum (PBC) with postcurriculum examinations for all surgical residents has been provided to assure coverage of core curricular topics. Biannual assessment of resident competency will soon be required by the Next Accreditation System.

OBJECTIVE

To identify opportunities for early medical knowledge assessment and interventions, we examined whether performance in postcurriculum multiple-choice examinations (PCEs) is predictive of performance in the American Board of Surgery In-Training Examination (ABSITE) and clinical service competency assessments.

DESIGN

Retrospective single-institutional education research study.

SETTING

Academic general surgery residency program.

PARTICIPANTS

A total of 49 surgical residents.

INTERVENTION

Data for PGY1 and PGY2 residents participating in the 2008 to 2012 PBC are included. Each resident completed 6 PCEs during each year.

MAIN OUTCOME MEASURES

The results of 6 examinations were correlated to percentage-correct ABSITE scores and clinical assessments based on the 6 Accreditation Council for Graduate Medical Education core competencies. Individual ABSITE performance was compared between PGY1 and PGY2. Statistical analysis included multivariate linear regression and bivariate Pearson correlations.

RESULTS

A total of 49 residents completed the PGY1 PBC and 36 completed the PGY2 curriculum. Linear regression analysis of percentage-correct ABSITE and PCE scores demonstrated a statistically significant correlation between the PGY1 PCE 1 score and the subsequent PGY1 ABSITE score (p = 0.037, β = 0.299). Similarly, the PGY2 PCE 1 score predicted performance in the PGY2 ABSITE (p = 0.015, β = 0.383). The ABSITE scores correlated between PGY1 and PGY2 with statistical significance, r = 0.675, p = 0.001. Performance on the 6 Accreditation Council for Graduate Medical Education core competencies correlated between PGY1 and PGY2, r = 0.729, p = 0.001, but did not correlate with PCE scores during either years.

CONCLUSIONS AND RELEVANCE

Within a mature PBC, early performance in a PGY1 and PGY2 PCE is predictive of performance in the respective ABSITE. This information can be used for formative assessment and early remediation of residents who are predicted to be at risk for poor performance in the ABSITE.