The use of a surgery-specific written examination in the selection process of surgical residents

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

PURPOSE

The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.

METHOD

Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.

RESULTS

Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.

CONCLUSIONS

This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Considerations for a Holistic Model in Evaluating Medical Students for Cardiothoracic Surgical Residency

Program directors are tasked with selecting whom they think will be the best fit for residency and the next leaders of the field. While numerical metrics have played a vital role in this process, recent changes to student evaluation are reducing the availability of these metrics. This poses unique challenges for both applicants and program directors. Here we discuss how this will likely shift the focus on other parts of the application and the consequences (good and bad) of doing so.

How to Prepare for the American Board of Surgery In-Training Examination (ABSITE): A Systematic Review

INTRODUCTION

Performance on ABSITE is an important factor when monitoring resident progress. It predicts future performance and has lasting effects. Understanding the highest-yield preparation strategies can help residents in their study efforts and optimize performance.

METHODS

A literature search was conducted searching PubMed, EMBASE and JAMA Network in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination". Only studies discussing individual study habits from May 2011 to May 2021 were included.

RESULTS

19 studies were included in qualitative synthesis. Year-round clinical study failed to show significant correlation to ABSITE performance although year-round ABSITE review was more consistently correlated. During a dedicated study period, increased time and increased total practice questions completed are associated with improved performance. The correlation of individual resources such as ABSITE review books, textbooks, audio podcasts and ABSITE preparatory courses to improved ABSITE performance was not proven.

CONCLUSIONS

Residents should optimize study strategies based on methods that have consistently shown to improve performance. Recommendations for best preparation strategies are provided.

Association Between USMLE Step 1 Scores and In-Training Examination Performance: A Meta-Analysis

PURPOSE

On February 12, 2020, the sponsors of the United States Medical Licensing Examination announced that Step 1 will transition to pass/fail scoring in 2022. Step 1 performance has historically carried substantial weight in the evaluation of residency applicants and as a predictor of subsequent subject-specific medical knowledge. Using a systematic review and meta-analysis, the authors sought to determine the association between Step 1 scores and in-training examination (ITE) performance, which is often used to assess knowledge acquisition during residency.

METHOD

The authors systematically searched Medline, EMBASE, and Web of Science for observational studies published from 1992 through May 10, 2020. Observational studies reporting associations between Step 1 and ITE scores, regardless of medical or surgical specialty, were eligible for inclusion. Pairs of researchers screened all studies, evaluated quality assessment using a modified Newcastle-Ottawa Scale, and extracted data in a standardized fashion. The primary endpoint was the correlation of Step 1 and ITE scores.

RESULTS

Of 1,432 observational studies identified, 49 were systematically reviewed and 37 were included in the meta-analysis. Overall study quality was low to moderate. The pooled estimate of the correlation coefficient was 0.42 (95% confidence interval [CI]: 0.36, 0.48; P < .001), suggesting a weak-to-moderate positive correlation between Step 1 and ITE scores. The random-effects meta-regression found the association between Step 1 and ITE scores was weaker for surgical (versus medical) specialties (beta -0.25 [95% CI: -0.41, -0.09; P = .003]) and fellowship (versus residency) training programs (beta -0.25 [95% CI: -0.47, -0.03; P = .030]).

CONCLUSIONS

The authors identified a weak-to-moderate positive correlation between Step 1 and ITE scores based on a meta-analysis of low-to-moderate quality observational data. With Step 1 scoring transitioning to pass/fail, the undergraduate and graduate medical education communities should continue to develop better tools for evaluating medical students.

Prospective Factors that Predict American Board of Surgery In-Training Examination Performance: A Systematic Review

INTRODUCTION

American Board of Surgery In-Training Examination (ABSITE) performance has become an important factor when monitoring resident progress. Understanding which prospective factors predict performance can help identify residents at risk.

METHODS

A literature search was conducted searching PubMed, EMBASE, and JAMA Network from June 2011 to June 2021, in accordance with the PRISMA guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination." Prospective factors such as prior examination performance, clinical evaluations, and demographics were evaluated.

RESULTS

A final 35 studies were included. The prospective factor most consistently found to predict ABSITE performance is performance on prior knowledge-based examinations such as the USMLE step exams. The ACGME Medical Knowledge 1 milestone evaluation also appears to correlate to ABSITE performance, although clinical evaluations, in general, do not. Demographics have no significant correlation to ABSITE performance.

DISCUSSION

Using performance on prior knowledge-based examinations programs may be able to identify residents at risk for failing ABSITE. It may be possible to initiate early intervention before rather than only remediation after poor performance.

Mastering the Match: Recruiting a Successful Surgery Resident

PURPOSE OF REVIEW

The objective of general surgery residency is to produce competent surgeons. At a minimum this requires being procedurally and clinically capable and able to pass the board exams. Recruitment is designed to select those residents who can successfully do so. But there is more to being a successful resident than that. In this review, we attempt to define a "successful resident" and how to attract them to your program.

RECENT FINDINGS

Resident applicants are still most concerned with matching to a program that will prepare them for a surgery career. Though there is variation of importance for different applicants, resident life, comradery, and relationships with faculty or mentors do factor into residency ranking. The program website remains the most utilized resource for applicants. However, social media (SM) has an increasing role in applicants' evaluation of a program. SM and the preinterview gathering seem to expose the subjective aspects of a program most effectively. Additional assessments evaluating personality, grit or career goals may assist in screening applicants for good "fit."

SUMMARY

In order to recruit successful residents, it is necessary to determine which applicant attributes are important to the program. Additionally, a program must maintain an updated website with clearly delineated resident expectations and program strengths. The screening and interview process must be maximized to target residents with career goals complimentary to available program opportunities. If SM is utilized, post should be frequent with relevant information pertaining to both resident life and educational or clinical opportunities.

Utilization of a Novel Program-Specific Evaluation Tool Results in a Decidedly Different Interview Pool Than Traditional Application Review

BACKGROUND

There are almost twice as many applicants as there are general surgery internships, each utilizing a common application with standard components. These elements are frequently not useful in determining affinity for a program or overall ability, and resultant poor fit may be partially responsible for program attrition. Alternative evaluation instruments would be beneficial to both programs and applicants.

METHODS

An application review committee comprised of resident representatives, faculty representing all program-affiliated institutions, and program leadership completed a written evaluation developed by a third party (SurgWise Consulting) that specializes in industrial and organizational psychology. The responses were compiled to create a standardized assessment tool. This assessment was sent to applicants who were subsequently ranked according to fit with our program. The pool of applicants was separately evaluated using our traditional application review. Two residents independently graded each applicant on a 5-point Likert scale to evaluate common application elements; applicants were subsequently assigned an overall score.

RESULTS

The assessment was completed by 507 (99%) of 512 qualifying applicants. Separately, 378 applications were reviewed by the traditional method for a total of 756 reviews. Of the 96 applicants identified by the assessment tool to invite for interviews, 22 (23%) qualified for interview invitations according to the traditional review method. The assessment produced 74 applicants that otherwise would not have been interviewed.

CONCLUSION

Traditional application review strategies have many shortcomings. A competency-based assessment tool in the residency application selection process identifies a pool of applicants not identified by traditional review methods.

Factors that Predict an Intern's First ABSITE Score are Known by September

BACKGROUND

Previous studies offer conflicting relevance of a variety of factors to predict resident performance on the The American Board of Surgery In-Training Exam (ABSITE). With numerous stellar applicants scoring poorly on their first ABSITE, we sought to identify key factors that might allow us to tailor pre-emptive study efforts in the fall and early winter to enhance scores.

METHODS

General Surgery residents in our program from 2009 through 2016 were included in our cohort study. Specific trainee data (sex, prelim vs categorical, United States Medical Licensing Examination (USMLE) Step 1 and 2 scores, ABSITE scores, clinical rotations, biannual objective structured clinical examination OSCE-type scores, and in-house prep test [IHPT], etc.) were collected retrospectively. The data were analyzed using JMP pro 10 and MedCalc.

RESULTS

ABSITE scores of our 110 trainees did not vary by sex or by categorical vs preliminary residents. USMLE step 1 and 2, IHPT and one objective structured clinical examination (OSCE) station (Sim-Based Trauma Exam) scores were positively correlated with ABSITE scores (p < 0.05; correlation coefficient (CC) = 0.6 [strong] for IHPT, 0.5 [moderate] for step 1 and 2 and [weak] 0.3 for Sim-Based Trauma Exam). The mean (standard deviation) ABSITE %tile score for residents scoring above 230 (USMLE 1 or 2) vs <230 were different: 81 (3) vs 56 (4) (p < 000.1). Of residents scoring lower than 230 in USMLE 1 or 2, their ABSITE performance had a strong positive linear correlation with their performance in the IHPT (CC = 0.7) and SBTA (CC = 0.5). Residents rotating twice on the Acute Care Surgery services before the ABSITE scored higher than those with 1 or zero rotations (p < 0.05).

CONCLUSIONS

USMLE steps 1 and 2 are useful parameters in our program to predict subsequent resident ABSITE performance. An in-house 60-minute preparation test in September, a 6-minute simulation-based trauma assessment in July, and rotation schedule with 2 stints on Acute Care Surgery were surprisingly useful early intern year tools to predict ABSITE scores.

Defining the Applicant Pool for Postgraduate Year-2 Categorical General Surgery Positions

OBJECTIVE

In the spring of 2010, a categorical general surgery postgraduate year (PGY)-2 position became available at our academic medical center secondary to attrition of a PGY-1 resident. We sought to study the unique characteristics of applicants to that position and to describe the selection process with hopes to stimulate additional studies about the unique challenges of recruiting applicants into advanced standing positions.

DESIGN

Applications were received via e-mail and reviewed to characterize the applicant pool. An Excel spreadsheet was used to organize data. Characteristics assessed included United States Medical Licensing Examination (USMLE) scores, Educational Commission for Foreign Medical Graduates status, Alpha Omega Alpha Honor Society status, sex, academic performance, number of case logs, volunteer and job experience, leadership roles, research experience including submissions, and advanced degrees. These characteristics were compared to those of the PGY-1 applicants through the Match that year.

SETTING

Academic medical center.

PARTICIPANTS

Applicants for a categorical general surgery PGY-2 position in 2010.

RESULTS

A total of 129 applicants provided the requested documents. There were 104 males, 25 females, no Alpha Omega Alpha Honor Society candidates, and 82 international candidates. Of all, 46 candidates experienced academic difficulties. Quantitative averages include USMLE 1: 214.17, USMLE 2: 215.74, American Board of Surgery In Training Examination (ABSITE) percentile = 51.96, ABSITE 2 = 46.00, grand total case log: 192.10. Advanced degrees included 2 MBAs, 6 MPHs, and 7 nonphysiology MSs. The selection process to fill the position started on 3/25/2010 when the announcement was published and ended on 5/11/2010 when the offer of acceptance was sent. The selected applicant integrated well with the peers and just graduated from our residency as one of the leaders of the graduating class.

CONCLUSIONS

Although the attrition rate in general surgery remains high, there is a dearth of literature about how best to replace residents. The hardship of replacing residents highlights the importance of studying this group to improve the recruitment process and the quality of replacement residents. The selection process was time consuming and presented its own challenges given the lack of a computerized system for screening. It lasted nearly 7 weeks requiring faculty time commitment to mine through application data/e-mails, correspond with applicants, conduct interviews, and ultimately select an applicant for the position. This is the first study to investigate the applicant pool to advanced standing positions in general surgery and we present it as a pilot study to stimulate further research efforts.

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

BACKGROUND

Selection into specialty training is a high-stakes and resource-intensive process. While substantial literature exists on selection into medical schools, and there are individual studies in postgraduate settings, there seems to be paucity of evidence concerning selection systems and the utility of selection tools in postgraduate training environments.

AIM

To explore, analyze and synthesize the evidence related to selection into postgraduate medical specialty training.

METHOD

Core bibliographic databases including PubMed; Ovid Medline; Embase, CINAHL; ERIC and PsycINFO were searched, and a total of 2640 abstracts were retrieved. After removing duplicates and screening against the inclusion criteria, 202 full papers were coded, of which 116 were included.

RESULTS

Gaps in underlying selection frameworks were illuminated. Frameworks defined by locally derived selection criteria, and heavily weighed on academic parameters seem to be giving way to the evidencing of competency-based selection approaches in some settings. Regarding selection tools, we found favorable psychometric evidence for multiple mini-interviews, situational judgment tests and clinical problem-solving tests, although the bulk of evidence was mostly limited to the United Kingdom. The evidence around the robustness of curriculum vitae, letters of recommendation and personal statements was equivocal. The findings on the predictors of past performance were limited to academic criteria with paucity of long-term evaluations. The evidence around nonacademic criteria was inadequate to make an informed judgment.

CONCLUSIONS

While much has been gained in understanding the utility of individual selection methods, though the evidence around many of them is equivocal, the underlying theoretical and conceptual frameworks for designing holistic and equitable selection systems are yet to be developed.

Selection for Surgical Training: An Evidence-Based Review

PURPOSE

The predictive relationship between candidate selection criteria for surgical training programs and future performance during and at the completion of training has been investigated for several surgical specialties, however there is no interspecialty agreement regarding which selection criteria should be used. Better understanding the predictive reliability between factors at selection and future performance may help to optimize the process and lead to greater standardization of the surgical selection process.

METHODS

PubMed and Ovid MEDLINE databases were searched. Over 560 potentially relevant publications were identified using the search strategy and screened using the Cochrane Collaboration Data Extraction and Assessment Template.

RESULTS

57 studies met the inclusion criteria. Several selection criteria used in the traditional selection demonstrated inconsistent correlation with subsequent performance during and at the end of surgical training. The following selection criteria, however, demonstrated good predictive relationships with subsequent resident performance: USMLE examination scores, Letters of Recommendation (LOR) including the Medical Student Performance Evaluation (MSPE), academic performance during clinical clerkships, the interview process, displaying excellence in extracurricular activities, and the use of unadjusted rank lists.

CONCLUSIONS

This systematic review supports that the current selection process needs to be further evaluated and improved. Multicenter studies using standardized outcome measures of success are now required to improve the reliability of the selection process to select the best trainees.

Surgery residency curriculum examination scores predict future American Board of Surgery in-training examination performance

IMPORTANCE

A protected block curriculum (PBC) with postcurriculum examinations for all surgical residents has been provided to assure coverage of core curricular topics. Biannual assessment of resident competency will soon be required by the Next Accreditation System.

OBJECTIVE

To identify opportunities for early medical knowledge assessment and interventions, we examined whether performance in postcurriculum multiple-choice examinations (PCEs) is predictive of performance in the American Board of Surgery In-Training Examination (ABSITE) and clinical service competency assessments.

DESIGN

Retrospective single-institutional education research study.

SETTING

Academic general surgery residency program.

PARTICIPANTS

A total of 49 surgical residents.

INTERVENTION

Data for PGY1 and PGY2 residents participating in the 2008 to 2012 PBC are included. Each resident completed 6 PCEs during each year.

MAIN OUTCOME MEASURES

The results of 6 examinations were correlated to percentage-correct ABSITE scores and clinical assessments based on the 6 Accreditation Council for Graduate Medical Education core competencies. Individual ABSITE performance was compared between PGY1 and PGY2. Statistical analysis included multivariate linear regression and bivariate Pearson correlations.

RESULTS

A total of 49 residents completed the PGY1 PBC and 36 completed the PGY2 curriculum. Linear regression analysis of percentage-correct ABSITE and PCE scores demonstrated a statistically significant correlation between the PGY1 PCE 1 score and the subsequent PGY1 ABSITE score (p = 0.037, β = 0.299). Similarly, the PGY2 PCE 1 score predicted performance in the PGY2 ABSITE (p = 0.015, β = 0.383). The ABSITE scores correlated between PGY1 and PGY2 with statistical significance, r = 0.675, p = 0.001. Performance on the 6 Accreditation Council for Graduate Medical Education core competencies correlated between PGY1 and PGY2, r = 0.729, p = 0.001, but did not correlate with PCE scores during either years.

CONCLUSIONS AND RELEVANCE

Within a mature PBC, early performance in a PGY1 and PGY2 PCE is predictive of performance in the respective ABSITE. This information can be used for formative assessment and early remediation of residents who are predicted to be at risk for poor performance in the ABSITE.

Correlation between trainee candidate selection criteria and subsequent performance

BACKGROUND

The objective of trainee recruitment is to identify candidates likely to perform well as trainees and subsequent faculty. The effectiveness of this process has not been established. The goal of this study was to identify trainee selection criteria predictive of excellent performance.

STUDY DESIGN

Twenty-nine microsurgery fellows were enrolled from 2008 to 2012. Each candidate was interviewed and rated based on presentation, plastic surgery (PS) training experience, academic potential, personality, social skills, communication skills, and ability to be a team player. An unadjusted rank list was generated based on weighted averages, and an adjusted rank list was then generated at a faculty meeting. At the conclusion of fellowship, each fellow was rated based on the ACGME core competencies. Spearman correlation coefficients (r) were used to measure the correlations between fellow selection criteria and fellow performance.

RESULTS

Plastic surgery training and academic potential had, by far, the strongest correlation to overall performance (r: 0.678, p < 0.001 and r: 0.56, p < 0.002), and to all ACGME competencies. When reformulated to weight PS training and academic potential more heavily than subjective criteria, the scoring system was significantly more predictive of excellent performance (r: 0.49 vs 0.70). The unadjusted rank list was more predictive of excellent performance than the adjusted rank list (r: 0.45 vs 0.65).

CONCLUSIONS

Plastic surgery training experience and academic potential were better predictors of performance than any subjective information ascertained during the interview. Adjustments to the rank list based on faculty discussion resulted in lower performance candidates moving up in ranking. Ranking criteria and interview techniques must be refined to improve predictive power. It may be beneficial for semi-objective criteria to carry more weight than subjective criteria and raw scores to remain unadjusted by extraneous information.