Using United States Medical Licensing Examination(®) (USMLE) examination results to predict later in-training examination performance among general surgery residents

Correlation Between Medical School Performance and Receiving Awards in General Surgery Residency

IMPORTANCE

It is uncertain whether current measures of achievement during medical school predict exceptional performance during surgical residency. One surrogate of excellence during residency may be awards, especially those given for teaching and annual overall accomplishment.

OBJECTIVE

Determine whether markers of superior performance during medical school documented in the Electronic Residency Application Service (ERAS) application and student record correlated with receiving awards during residency.

DESIGN

Data was analyzed from ERAS and student applications of 296 residency graduates. Spearman correlation coefficients were calculated.

PARTICIPANTS

296 residency graduates from 14 US general surgery residency programs.

MAIN OUTCOMES AND MEASUREMENTS

The relationship between each residency and preresidency variable and the outcome of receiving any qualifying award. Secondary analysis included the correlation with winning a teaching or annual overall award.

RESULTS

Although 140 (48%) residents received an award during residency, only 69 (23.3%) received a teaching award, while 50 (17.6%) received one for annual overall performance. Membership in Alpha Omega Alpha (AOA) and honors in pediatrics rotation correlated with receiving any award. USMLE step 1 score and AOA membership reached statistically significant positively correlation with receiving a teaching award, while completing a sub-internship at that institution was negatively associated. For annual overall awards, only completing a sub-internship at the same institution reached statistical significance.

CONCLUSIONS

None of the traditional metrics in the ERAS or global student record consistently correlated with receiving an award during residency. Our findings suggest that preresidency factors available on the global application, including grades, test scores, and honor society membership, poorly correlate with exceptional general surgery residency performance as measured by receiving awards. Residency programs should shift away from using traditional ERAS metrics as the primary criteria for selection, but rather as 1 component of holistic applicant review.

Pediatric residency milestone performance is not predicted by the United States Medical Licensing Examination Step 2 Clinical Knowledge

Objectives

This study aims to show whether correlation exists between pediatric residency applicants' quantitative scores on the United States Medical Licensing Exam Step 2 Clinical Knowledge examination and their subsequent performance in residency training based on the Accreditation Council for Graduate Medical Education Milestones, which are competency-based assessments that aim to determine residents' ability to work unsupervised after postgraduate training. No previous literature has correlated Step 2 Clinical Knowledge scores with pediatric residency performance assessed by Milestones.

Methods

In this retrospective cohort study, the United States Medical Licensing Exam Step 2 Clinical Knowledge Scores and Milestones data were collected from all 188 residents enrolled in a single categorical pediatric residency program from 2012 - 2017. Pearson correlation coefficients were calculated amongst available test and milestone data points to determine correlation between test scores and clinical performance.

Results

Using Pearson correlation coefficients, no significant correlation was found between quantitative scores on the Step 2 Clinical Knowledge exam and average Milestones ratings (r = -0.1 for post-graduate year 1 residents and r = 0.25 for post-graduate year 3 residents).

Conclusions

These results demonstrate that Step 2 scores have no correlation to success in residency training as measured by progression along competency-based Milestones. This information should limit the importance residency programs place on quantitative Step 2 scores in their ranking of residency applicants. Future studies should include multiple residency programs across multiple specialties to help make these findings more generalizable.

Student Perspectives on the Undergraduate Medical Student Research Experience at a Single United States Allopathic Institution

Introduction The purpose of this study was to identify student-reported institutional facilitators and barriers to successful research experiences at a single United States allopathic institution. Residency applications have increasingly become more competitive, and with the United States Medical Licensing Examination (USMLE) Step 1 exam's transition to pass/fail, factors such as research experience and outcomes may become more important to increase residency application competitiveness. This study sought to explore factors that impact successful research experiences leading to tangible outcomes for medical students at our medical school, the Joe R. & Teresa Lozano Long School of Medicine. Methods  A cross-sectional survey was developed and administered via REDCap to 853 students in May 2022. Survey question domains included demographics, past and present research participation, perceived barriers/facilitators to research, tangible outcomes (e.g., publications and posters), and overall satisfaction with research comparing subjectively "best" and "worst" experiences. The Institutional Review Board (IRB) deemed this project as non-regulated research.  Results We had a 24% (n = 204/853) response rate. The responses were distributed equally among the four classes. A big portion of the participants (71%, n = 59/83) identified a tangible outcome as the most important measure of success. Regarding facilitators, students identified having a mentor (89%, n = 165/184) and departmental connections (85%, n = 156/184) as the most important when looking for a project. Barriers included SMART goals (Specific, Measurable, Achievable, Relevant, and Time-Bound) lacking in 31% (n = 24/75) of worst projects, followed by a clear timeline in 29% (n = 22/76) and hours of commitment in 27% (n = 21/78). The best projects were more likely to have resulted in a publication (61% (27/44) vs. 32% (14/44)) or have a poster (64% (28/44) vs. 36% (16/44)). Conclusions Medical students are interested in participating in research, with important facilitators including mentorship and departmental connections. Modifiable variables include lack of clear timelines, well-defined roles and responsibilities, and time commitments. This information may be useful for faculty who mentor medical students or medical schools interested in designing medical student research programs.

The USMLE® STEP 1 Pass or Fail Era of the Vascular Surgery Residency Application Process: Implications for Structural Bias and Recommendations

MOTIVES BEHIND THE SHIFT

USMLE® STEP 1 score reporting has been changed to a binary pass/fail format since January 26, 2022. The motives behind this change were (1) the questionable validity of using USMLE® STEP 1 as a screening tool during the candidate selection process and (2) the negative impact of using standardized examination scores as an initial gatekeeping threshold for the underrepresented in medicine (URiM) candidates applying to GME programs, given their generally lower mean standardized exams scores compared to non-URiM students. The USMLE® administrators justified this change as a tactic to enhance the overall educational experience for all students and to increase the representation of URiM groups. Moreover, they advised the program directors (PDs) to give more attention to other important qualities and components such as the applicant's personality traits, leadership roles and other extracurricular accomplishments, as part of a holistic evaluation strategy. IMPACT OF USMLE® STEP 1 PASS/FAIL: At this early stage, it is unclear how this change will impact Vascular Surgery Integrated residency (VSIR) programs. Several questions are outstanding, most importantly, how VSIR PDs will evaluate applicants absent the variable which heretofore was the primary screening tool. Our previously published survey showed that VSIR PDs will move their attention to other measures such as USMLE® STEP 2CK and letters of recommendation during the VSIR selection process. Furthermore, more emphasis on subjective measures such as the applicant's medical school rank and extracurricular student activities is expected. Given the expected higher weight of USMLE® STEP 2CK in the selection process than ever, many anticipate that medical students will dedicate more of their limited time to its preparation at the expense of both clinical and non-clinical activities. Potentially leaving less time to explore specialty pathways and to determine whether VS is the appropriate career for them.

A FRAMEWORK FOR CANDIDATE EVALUATION

The critical juncture in the VSIR candidate evaluation paradigm presents an opportunity to thoughtfully transform the process via current (Standardized Letter of Recommendation, USMLE® STEP 2CK, and clinical research) and future (Emotional Intelligence, Structure Interview and Personality Assessment) measures which constitute a framework to follow in the USMLE® STEP 1 pass/fail era.

The Impact of Socioeconomic Factors on the 2022 Plastic Surgery Match

BACKGROUND

In 2022, the plastic and reconstructive surgery (PRS) match faced unprecedented system-wide transitions that have redefined conventional measures of applicant success. This challenges the equitable assessment of student competitiveness and diversity in the field.

METHODS

A survey of demography, application content, and 2022 match outcomes was distributed to applicants to a single PRS residency program. Comparative statistics and regression models were performed to assess the predictive value of factors in match success and quality.

RESULTS

A total of 151 respondents (response rate 49.7%) were analyzed. Although step 1 and step 2 CK scores were significantly higher among matched applicants, neither examination predicted match success. Most respondents (52.3%) were women, although gender was also not significantly associated with match success. Underrepresented in medicine applicants made up 19.2% of responses and 16.7% of matches, and the plurality of respondents (22.5%) were raised with a household income ≥$300,000. Both Black race and household income ≤$100,000 were associated with lower odds of scoring above a 240 on either step 1 or step 2 CK (Black: OR, 0.03 and 0.06; P < 0.05 and P < 0.001; income: OR, 0.07-0.47 and 0.1 to 0.8, among income subgroups), receiving interview offers (OR, -9.4; P < 0.05; OR, -11.0 to -5.4), and matching into PRS (OR, 0.2; P < 0.05; OR, 0.2 to 0.5), compared with White and high-income applicants, respectively.

CONCLUSIONS

Systemic inequities in the match process disadvantage underrepresented in medicine candidates and those from lower household incomes. As the residency match continues to evolve, programs must understand and mitigate the impacts of bias in various application components.

USMLE Step 1 scores and PGY-5 ABSITE performance are predictive of American Board of Surgery examination first-time pass rates

INTRO

Standardized exams are utilized throughout a resident's career. With USMLE Step 1 score reports changing to pass/fail only, we sought to evaluate the relationships among USMLE Step 1 and 2, PGY-5 ABSITE, and ABS performance.

METHODS

Retrospective analyses of PGY-5 USMLE Step 1 and Step 2 scores, ABSITE percentile ranks, and ABS QE and CE first-time pass rates of graduates from 2010 to 2019 were performed.

RESULTS

PGY-5 residents scoring ≤ 20th percentile on ABSITE during their PGY-5 year were more likely to have lower USMLE Step 1 scores, but not Step 2 scores, and have a lower ABS first pass rate. Lower Step and ABSITE scores together were associated with lower ABS first pass rate.

CONCLUSION

As USMLE Step 1 is now a pass/fail examination, programs may need to utilize other information to select residency candidates and structure educational activities differently within their programs.

The Association Between USMLE Step 2 Clinical Knowledge Scores and Residency Performance: A Systematic Review and Meta-Analysis

PURPOSE

With the change in Step 1 score reporting, Step 2 Clinical Knowledge (CK) may become a pivotal factor in resident selection. This systematic review and meta-analysis seeks to synthesize existing observational studies that assess the relationship between Step 2 CK scores and measures of resident performance.

METHOD

The authors searched MEDLINE, Web of Science, and Scopus databases using terms related to Step 2 CK in 2021. Two researchers identified studies investigating the association between Step 2 CK and measures of resident performance and included studies if they contained a bivariate analysis examining Step 2 CK scores' association with an outcome of interest: in-training examination (ITE) scores, board certification examination scores, select Accreditation Council for Graduate Medical Education core competency assessments, overall resident performance evaluations, or other subjective measures of performance. For outcomes that were investigated by 3 or more studies, pooled effect sizes were estimated with random-effects models.

RESULTS

Among 1,355 potential studies, 68 met inclusion criteria and 43 were able to be pooled. There was a moderate positive correlation between Step 2 CK and ITE scores (0.52, 95% CI 0.45-0.59, P < .01). There was a moderate positive correlation between Step 2 CK and ITE scores for both nonsurgical (0.59, 95% CI 0.51-0.66, P < .01) and surgical specialties (0.41, 95% CI 0.33-0.48, P < .01). There was a very weak positive correlation between Step 2 CK scores and subjective measures of resident performance (0.19, 95% CI 0.13-0.25, P < .01).

CONCLUSIONS

This study found Step 2 CK scores have a statistically significant moderate positive association with future examination scores and a statistically significant weak positive correlation with subjective measures of resident performance. These findings are increasingly relevant as Step 2 CK scores will likely become more important in resident selection.

The Impact of Holistic Review of Urology Residency Applications on Selection for Interview During the COVID-19 Pandemic

OBJECTIVE

To investigate the impact of a holistic review of urology residency applications on interview selection at our institution during the COVID-19 pandemic.

METHODS

In the 2019-2020 cycle, applicants were filtered by a Step 1 score of 230 and whether they applied from selected east coast medical schools. For the 2020-2021 and 2021-2022 cycles, we implemented a scoring system which focused on desirable attributes based on our program training needs and resources. We compared applicant and interviewee demographics and United States Medical Licensing Examination (USMLE) scores using descriptive statistics and 1-way analysis of variance tests.

RESULTS

A total of 282, 300, and 367 students applied to our residency program with 50, 45, and 52 selected for interviews during the 2019-2020, 2020-2021, and 2021-2022 cycles, respectively. Compared to 2019-2020, the 2020-2021 and 2021-2022 interviewee cohorts comprised of more non-tri-state applicants (36%, 55.6%, and 46.2%, respectively). Underrepresented minority representation increased for the 2020-2021 interviewee cohort; however, this was not observed in 2021-2022 (16%, 24.4%, 15.4%, respectively). Additionally, USMLE Step 1 and 2 scores were similar between interviewee cohorts in 2019-2020, 2020-2021 and 2021-2022, respectively (Step 1: 244.2 ± 8.8, 242 ± 12.1, 242.8 ± 12.4, P = .624) (Step 2: 249.1 ± 11.5, 251.5 ± 10.5, 254.4 ± 10.8, P = .143).

CONCLUSION

Utilizing a comprehensive review resulted in a geographically diverse interview pool and no significant difference in academic performance among interviewees. Holistic review provides an alternative, balanced evaluation of residency applicants which may increase diversity in urology.

The next step: Replacing step 1 as a metric for residency application

Objective

As of January 2022, USMLE Step 1 scores are reported as pass/fail. Historically, Step 1 scores have been a critical component of residency applications, representing one of the few metrics standardized across all applicants independent of the school they attended. In competitive specialties, such as otolaryngology, programs routinely get 100+ applicants for each residency spot and use Step 1 as a screening tool. This study seeks to assess quantifiable metrics in the application that highly competitive residency programs could use for screening in place of Step 1 scores.

Methods

Otolaryngology applications to an academic medical center for the 2019-20 and 2020-21 ERAS cycles were reviewed. Board scores and quantitative research data were extracted. The relationships between Step 1 score and the other metrics were examined by computing Pearson's correlation coefficients and building regression models. Similar analyses were done separately for three different score tiers defined by Step 1 cutoffs at 220 points and 250 points.

Results

Step 2 score was the only variable that had meaningful correlation with Step 1 score (R = .67, p < 2.2e-16). No other objective metric such as journal articles, posters, or oral presentations correlated with Step 1 scores.

Conclusion

Step 1 scores were moderately correlated with Step 2 scores; however, using a Step 2 cutoff instead of a Step 1 cutoff would identify a different cohort of applicants for interview. No other quantifiable application metric had a positive correlation. In future match cycles, highly competitive residency programs will need to adopt new methods to screen candidates.Level of Evidence: Level 3.

A Survey of Vascular Surgery Program Directors: Perspectives Following USMLE Step 1 Conversion to Pass/Fail and Virtual Only Interviews

INTRODUCTION

Step 1 of the United States Medical Licensing Examination (USMLE), a historically important factor in the selection of vascular surgery residents and fellows, transitioned to completely pass/fail on January 26, 2022. Due to the COVID-19 pandemic, residency and fellowship interviews were conducted virtually during the 2020-21 and 2021-22 application cycles. Given these significant changes in the evaluation of candidates for residency training, we sought to understand vascular surgery program directors' perspectives regarding the change in Step 1 scoring and use of virtual interviews as well as determine which factors will assume importance when applying to vascular surgery training programs in the future.

METHODS

A 26-item survey questionnaire was created using Qualtrics survey tools with questions regarding attitudes towards the change in Step 1 scoring and virtual interviews, the importance of additional factors utilized by programs in selecting candidates for interviews and ranking applicants for residency/fellowship selection, and programmatic demographics. This was distributed anonymously to all vascular surgery training programs over a two-week period using a comprehensive list of 249 unique email addresses created by cross-referencing the Accreditation Council for Graduate Medical Education (ACGME) list of certified vascular training programs with email addresses from individual program websites identifying current program leadership. Responses were analyzed with descriptive statistics with values listed as average Likert scale weight ± standard deviation (SD) or percentages.

RESULTS

68 of 249 (27.3%) program and associate program directors responded to the survey. 33.9% of respondents strongly disagreed with Step 1 going completely pass/fail. In the absence of a scored USMLE Step 1, letters of recommendation (average Likert scale weight ± SD, 4.43 ± 0.92), dedication to specialty (4.14 ± 1.03), and USMLE Step 2 CK (4.06 ± 0.92) had the highest average scores for deciding which applicants to interview for integrated vascular surgery residency. For determining which candidates to interview for vascular surgery fellowship, letters of recommendation (4.51 ± 0.84), dedication to specialty (4.12 ± 0.90), and research (4.10 ± 0.80) had the highest average scores. For ranking residency candidates, the interviewee's perceived "fit" (4.61 ± 0.55), letters of recommendation (4.53 ± 0.76), and overall interview experience (4.47 ± 0.62) had the highest average scores. Similarly, the factors with the highest average Likert scores for ranking fellowship candidates included the interviewee's perceived "fit" (4.69 ± 0.51), letters of recommendation (4.65 ± 0.52), and overall interview experience (4.51 ± 0.59). The majority (72.2%) of program directors preferred in-person interviews; however, 50% of respondents were at least "somewhat satisfied" with virtual interviews during the 2021-22 application cycle as they could judge applicants' interview skills at least "moderately well." The minority (18.8%) who preferred virtual interviews most commonly noted a "reduction of the financial burden for applicants" as the reason for this preference.

CONCLUSION

Most vascular surgery program and associate program directors were dissatisfied with USMLE Step 1 going pass/fail with most indicating pre-screening applicants using both Step 1 and Step 2 CK during the residency and fellowship selection processes. In the absence of a scored Step 1, the top factors for interviewing and ranking integrated vascular surgery residency and fellowship candidates included letters of recommendation, dedication to specialty, research, USMLE Step 2 CK, the interviewee's perceived "fit," and overall interview experience. Though most program directors preferred face-to-face interviews, they were overall at least "somewhat satisfied" with the virtual format that took place during the 2021-22 cycle.

Predicting Academic Performance during Plastic Surgery Residency: Can Step 2 Scores Reliably Replace Step 1?

OBJECTIVE

Step 1 will transition to a pass/fail system in 2022. This study aimed to characterize the effects of this change on integrated plastic surgery program directors' selection criteria and assess whether Step 2 Clinical Knowledge (CK) can replace Step 1 as an application selection metric.

DESIGN

Online survey that was administered to a collaborative group of ten plastic surgery program directors collecting USMLE Step 1, Step 2 CK, In-Service, and written board scores for 3 years of graduated integrated residents.

SETTING

Ten academic integrated plastic surgery programs.

PARTICIPANTS

Data from 80 graduated integrated plastic surgery residents.

RESULTS

Across 80 included integrated residents, mean (SD) Step 1 score was 247 (13), Step 2 CK was 249 (13), PGY1-6 In-Service percentiles varied from 45 to 53 percentile, and written board pass rate was 98.3%. Both Step 1 and Step 2 CK correlated highly with In-Service percentiles (both p < 0.001), with Step 2 CK scores correlating similarly with In-Service performance compared to Step 1 (rho 0.359 vs. 0.355, respectively). Across applicant characteristics, program directors reported the highest relative increase in Step 2 CK importance after Step 1 transitions to pass/fail.

CONCLUSIONS

Step 2 CK correlates similarly with plastic surgery In-Service performance compared to Step 1. While Step scores do not necessarily correlate with residency performance, Step 2 CK may also be used as an application screening metric for programs seeking objective data to differentiate plastic surgery applicants.

Characteristics of First-Year Residents in Top-Ranked United States Ophthalmology Residency Programs

Objective  The aim of the study is to investigate the characteristics of first-year residents associated with attending a top-ranked United States (U.S.) ophthalmology residency program over the past decade. Methods  First-year ophthalmology residents in 2009, 2013, 2016, and 2019 were identified from institutional websites, Doximity, LinkedIn and the Wayback Machine. Publications were obtained from Scopus and Google Scholar; research productivity was measured using the h -index, and medical school region based on U.S. Census Bureau designations. Medical school and ophthalmology training program rankings were based on U.S. News & World Report (U.S. News) rankings and National Institutes of Health (NIH) funding. One-way ANOVA, Wilcoxon rank sum, χ 2 , and t -tests were used to analyze trends, and odds ratios (ORs) were calculated using logistic regression. Results  Data were obtained on 81% (1,496/1,850) of the residents; 43% were female; 5% were international medical graduates (IMGs); and 10% had other graduate degrees. Over the decade, the mean h -index increased (0.87-1.26; p <0.05) and the proportion of residents who attended a top 20 medical school decreased (28-18%; p <0.05). In a multivariate logistic regression model, characteristics associated with being a first-year resident in a top 20 program ranked by U.S. News were female gender [OR: 1.32, 95% CI: 1.02-1.72], having a Master's degree [OR: 2.28, 95% CI: 1.29-4.01] or PhD [OR: 2.23, 95% CI: 1.32-3.79], attending a top 20 [OR: 5.26, 95% CI: 3.66-7.55] or a top 40 medical school by NIH funding [OR: 2.45, 95% CI: 1.70-3.54], attending a medical school with a mean USMLE Step 2 score above 243 [OR: 1.64, 95% CI: 1.01-2.67] or located in the Northeast [OR: 2.00, 95% CI: 1.38-2.89] and having an h -index of one or more [OR: 1.92, 95% CI: 1.47-2.51]. Except for gender, these characteristics were also significantly associated with matching to a top 20 ophthalmology program by NIH funding. Conclusion  Female gender, graduate degrees, research productivity, and attending a medical school with high research productivity, high mean USMLE Step 2 score or in the Northeast were key characteristics of first-year residents in top-ranked U.S. ophthalmology residency programs.

United States medical licensing examination Step 1 scores are a quantifiably better predictor of American Board of Surgery In-Training Examination success compared to Step 2 scores

BACKGROUND

United States Medical Licensing Examination Step 1 (STEP 1) and Step 2 (STEP 2) scores are used in the selection of surgery residents. The American Board of Surgery In-Training Examination (ABSITE) is employed to assess resident knowledge. We sought to determine whether both STEP 1 and/or STEP 2 were predictive of ABSITE performance.

METHODS

We performed a 10-year retrospective analysis of all PGY levels comparing STEP 1 and 2 scores with raw ABSITE scores and percentile rank. Additionally, we analyzed STEP 1 and 2 scores for the likelihood of scoring above the 20th percentile on the ABSITE.

RESULTS

When STEP 1 and STEP 2 scores were evaluated separately, both were independent predictors of ABSITE performance. However, when STEP 1 and STEP 2 scores were evaluated together, only STEP 1 scores were predictive of ABSITE performance. Finally, only STEP 1 scores were predictive of scoring above the 20th percentile.

CONCLUSION

STEP 1 scores are a better predictor of ABSITE performance than STEP 2.

Mastering the Match: Recruiting a Successful Surgery Resident

PURPOSE OF REVIEW

The objective of general surgery residency is to produce competent surgeons. At a minimum this requires being procedurally and clinically capable and able to pass the board exams. Recruitment is designed to select those residents who can successfully do so. But there is more to being a successful resident than that. In this review, we attempt to define a "successful resident" and how to attract them to your program.

RECENT FINDINGS

Resident applicants are still most concerned with matching to a program that will prepare them for a surgery career. Though there is variation of importance for different applicants, resident life, comradery, and relationships with faculty or mentors do factor into residency ranking. The program website remains the most utilized resource for applicants. However, social media (SM) has an increasing role in applicants' evaluation of a program. SM and the preinterview gathering seem to expose the subjective aspects of a program most effectively. Additional assessments evaluating personality, grit or career goals may assist in screening applicants for good "fit."

SUMMARY

In order to recruit successful residents, it is necessary to determine which applicant attributes are important to the program. Additionally, a program must maintain an updated website with clearly delineated resident expectations and program strengths. The screening and interview process must be maximized to target residents with career goals complimentary to available program opportunities. If SM is utilized, post should be frequent with relevant information pertaining to both resident life and educational or clinical opportunities.

Resident Selection: The Role of Assessment of Emotional Intelligence and Grit

SUMMARY

The ability of our resident selection process to identify individuals who will ultimately become competent plastic surgeons is crucial to the specialty's future. Current criteria in use are not productive of that outcome. The presence of emotional intelligence and the element of grit have been incorporated in business and the military as factors to be evaluated in potential candidates. Plastic surgery should initiate an investigation of inclusion of a similar assessment of resident applicants.

Medical Students' Reflections on the Recent Changes to the USMLE Step Exams

The United States Medical Licensing Examination (USMLE) consists of Step 1, Step 2 Clinical Knowledge, Step 2 Clinical Skills, and Step 3. To be licensed to practice medicine in the United States, medical students must pass all parts of the USMLE. However, in addition to that pass/fail grade, students are currently given a numerical score for Step 1, Step 2 Clinical Knowledge, and Step 3. Residency program directors have come to use the Step 1 score to efficiently screen a growing number of residency applicants. As a result, a deleterious environment in undergraduate medical education has been created, given the importance of Step 1 to medical students matching to their preferred residency program. It was announced in February 2020 that the score-reporting protocol for Step 1 would be changed from a 3-digit numerical score to pass/fail only, beginning no earlier than January 1, 2022. This decision will undoubtedly impact medical students, medical schools, and residency program directors. Here, the authors discuss the impact that the change to Step 1 scoring will have on these key stakeholder groups, from their perspective as students at MD-granting medical schools in the United States. They also call attention to outstanding issues with the USMLE that must be addressed to improve undergraduate medical education for all stakeholders, and they offer advice for further improvements to the residency application process.

Spaced Education With ABSITE Quest Resulting in Improved American Board of Surgery In-Training Examination Performance

OBJECTIVE

The American Board of Surgery In-Training Examination (ABSITE) is an annual exam taken by general surgery residents as a cognitive assessment of the knowledge gained throughout each year of training. Several question banks are available for ABSITE preparation. However, ABSITE Quest (AQ) utilizes a method called spaced education which has been demonstrated to help with retaining information longer and improve exam performance. This study hypothesizes that using this method will help residents improve their ABSITE performance.

DESIGN

Retrospective survey data was collected from residents who participated in AQ, including postgraduate year (PGY) level, as well as 2019 and 2020 ABSITE percentiles. AQ user data was used to match respondent's total number of questions completed and daily engagement level to the survey data. Paired, single-tailed student's t test was used to evaluate the significance of ABSITE percentile change between 2019 and 2020 among AQ users.

SETTING

ChristianaCare, Newark, DE, United States. Nonclinical.

PARTICIPANTS

All ABSITE Quest users were surveyed (n = 390), of which 104 responded. 21 responses were from PGY1 residents and were excluded, resulting in a total of 83 responses.

RESULTS

The mean percentile difference of AQ users from 2019 to 2020 was +15.8 (p < 0.00001). When categorizing by the total number of questions completed, high users demonstrated a mean percentile difference of +15.3 (p = 0.00002), average users had a difference of +19.1 (p = 0.00029), and low users showed a percentile difference of +1.2 (p = 0.45244). When categorizing by daily engagement level, high users demonstrated a mean percentile difference of +17.9 (p < 0.00001), low users had a mean percentile difference of +15.3 (p = 0.00124), and minimal users showed a mean percentile change of -5.7.

CONCLUSIONS

The use of the spaced education method with ABSITE Quest, especially in users with a greater number of questions completed and high levels of daily engagement, correlated with a significant improvement on ABSITE performance.

Objective predictors of intern performance

BACKGROUND

Residency programs select medical students for interviews and employment using metrics such as the United States Medical Licensing Examination (USMLE) scores, grade-point average (GPA), and class rank/quartile. It is unclear whether these metrics predict performance as an intern. This study tested the hypothesis that performance on these metrics would predict intern performance.

METHODS

This single institution, retrospective cohort analysis included 244 graduates from four classes (2015-2018) who completed an Accreditation Council for Graduate Medical Education (ACGME) certified internship and were evaluated by program directors (PDs) at the end of the year. PDs provided a global assessment rating and ratings addressing ACGME competencies (response rate = 47%) with five response options: excellent = 5, very good = 4, acceptable = 3, marginal = 2, unacceptable = 1. PDs also classified interns as outstanding = 4, above average = 3, average = 2, and below average = 1 relative to other interns from the same residency program. Mean USMLE scores (Step 1 and Step 2CK), third-year GPA, class rank, and core competency ratings were compared using Welch's ANOVA and follow-up pairwise t-tests.

RESULTS

Better performance on PD evaluations at the end of intern year was associated with higher USMLE Step 1 (p = 0.006), Step 2CK (p = 0.030), medical school GPA (p = 0.020) and class rank (p = 0.016). Interns rated as average had lower USMLE scores, GPA, and class rank than those rated as above average or outstanding; there were no significant differences between above average and outstanding interns. Higher rating in each of the ACGME core competencies was associated with better intern performance (p < 0.01).

CONCLUSIONS

Better performance as an intern was associated with higher USMLE scores, medical school GPA and class rank. When USMLE Step 1 reporting changes from numeric scores to pass/fail, residency programs can use other metrics to select medical students for interviews and employment.

The Surgical Knowledge "Growth Curve": Predicting ABSITE Scores and Identifying "At-Risk" Residents

OBJECTIVE

Resident performance on the American Board of Surgery In-Training Examination (ABSITE) is used for evaluation of surgical knowledge and guides resident selection for institutional remediation programs. Remediation thresholds have historically been based on ABSITE percentile scores; however, this does not account for predictors that can impact a resident's exam performance. We sought to identify predictors of yearly ABSITE performance to help identify residents "at-risk" for performing below their expected growth trajectory.

DESIGN

The knowledge of the residents, as measured by standardized ABSITE scores, was modeled as a function of the corresponding postgraduate year via a linear mixed effects regression model. Additional model covariates included written USMLE-1-3 examination scores, gender, number of practice questions completed, and percentage correct of practice questions. For each resident, the predicted ABSITE standard score along with a 95% bootstrap prediction interval was obtained. Both resident-specific and population-level predictions for ABSITE standard scores were also estimated.

SETTING

The study was conducted at a single, large academic medical center (Massachusetts General Hospital, Boston, MA).

PARTICIPANTS

Six years of general surgery resident score reports at a single institution between 2014 and 2019 were deidentified and analyzed.

RESULTS

A total of 376 score reports from 130 residents were analyzed. Covariates that had a significant effect on the model included USMLE-1 score (PGY1: p = 0.013; PGY2: p = 0.007; PGY3: p = 0.011), USMLE-2 score (PGY1: p < 0.001; PGY2: p < 0.001; PGY3: p < 0.001; PGY4: p < 0.001; PGY5: p = 0.032), male gender (PGY1: p = 0.003; PGY2: p < 0.001; PGY3: p < 0.001; PGY4: p = 0.008), and number of practice questions completed (p=0.003). Five residents were identified as having "fallen off" their predicted knowledge curve, including a single resident on 2 occasions. Population prediction curves were obtained at 7 different covariate percentile levels (5%, 10%, 25%, 50%, 75%, 90%, and 95%) that could be used to plot predicted resident knowledge progress.

CONCLUSION

Performance on USMLE-1 and -2 examinations, male gender, and number of practice questions completed were positive predictors of ABSITE performance. Creating residency-wide knowledge growth curves as well as individualized predictive ABSITE performance models allows for more efficient identification of residents potentially at risk for poor ABSITE performance and structured monitoring of surgical knowledge progression.

Best Practices in Letters of Recommendation for General Surgery Residency: Results of Expert Stakeholder Focus Groups

OBJECTIVE

Performance inflation is rampant in applications to general surgery residency. The medical student performance evaluation, transcript, and letters of recommendation (LOR) have all been shown significantly biased in the applicants' favor. This study sought to determine best practices for LOR to improve transparency and alignment of applicant and program characteristics.

DESIGN

Two 1-hour focus groups were conducted using semi-structured interviews. Participants were asked to discuss the value and role of LOR characteristics, including standardized LOR, and provide recommendations for best practices. The transcribed discussions were coded by two educators using grounded theory and an inductive approach utilizing NVivo 12. Codes were then reviewed and revised to achieve consensus and recommendations.

SETTING

Focus groups were held during the annual Surgical Education Week meeting in April 2019.

PARTICIPANTS

General Surgery Program Directors from 10 institutions and Surgery Clerkship Directors from 11 other medical schools participated, with each group meeting independently from the other.

RESULTS

Individually, 18 codes were identified by the authors, with consensus agreed on ten. These were grouped into 4 themes: author factors, letter content, bias, and standardized letters. Overall, a checkbox and short-answer standardized LOR was not recommended, favoring a template of items to include and exclude. Ideal letter writers were felt to be surgeons who best know the applicant, and the Chair's letter, when they have no working knowledge of the applicant, was perceived to add little value. Use of specific examples to demonstrate applicant characteristics were favored, and descriptors for coded language should be included to aid in interpretation.

CONCLUSION

The focus groups identified best practices to guide writing LOR in support of applicants to general surgery residency. A template of content is provided to improve the efficiency, transparency, and accuracy of these letters for the benefit of students, medical schools, and residency programs.

Application Factors Associated With Clinical Performance During Pediatric Internship

OBJECTIVE

Our goal was to identify aspects of residency applications predictive of subsequent performance during pediatric internship.

METHODS

We conducted a retrospective cohort study of graduates of US medical schools who began pediatric internship in a large pediatric residency program in the summers of 2013 to 2017. The primary outcome was the weighted average of subjects' Accreditation Council for Graduate Medical Education pediatric Milestone scores at the end of pediatric internship. To determine factors independently associated with performance, we conducted multivariate linear mixed-effects models controlling for match year and Milestone grading committee as random effects and the following application factors as fixed effects: letter of recommendation strength, clerkship grades, medical school reputation, master's or PhD degrees, gender, US Medical Licensing Examination Step 1 score, Alpha Omega Alpha membership, private medical school, and interview score.

RESULTS

Our study population included 195 interns. In multivariate analyses, the aspects of applications significantly associated with composite Milestone scores at the end of internship were letter of recommendation strength (estimate 0.09, 95% confidence intervals [CI]: 0.04, 0.15), numbers of clerkship honors (est. 0.05, 95% CI: 0.01-0.09), medical school ranking (est. 0.04, 95% CI: 0.08-0.01), having a master's degree (est. 0.19, 95% CI: 0.03-0.36), and not having a PhD (est. 0.14, 95% CI: 0.02-0.26). Overall, the final model explained 18% of the variance in milestone scoring.

CONCLUSIONS

Letter of recommendation strength, clerkship grades, medical school ranking, and having obtained a Master's degree were significantly associated with higher clinical performance during pediatric internship.

Predicting Quality of Clinical Performance From Cardiology Fellowship Applications

Variables in cardiology fellowship applications have not been objectively analyzed against applicants' subsequent clinical performance. We investigated possible correlations in a retrospective cohort study of 65 cardiology fellows at the Mayo Clinic (Rochester, Minn) who began 2 years of clinical training from July 2007 through July 2013. Application variables included the strength of comparative statements in recommendation letters and the authors' academic ranks, membership status in the Alpha Omega Alpha Honor Medical Society, awards earned, volunteer activities, United States Medical Licensing Examination (USMLE) scores, advanced degrees, publications, and completion of a residency program ranked in the top 6 in the United States. The outcome was clinical performance as measured by a mean of faculty evaluation scores during clinical training. The overall mean evaluation score was 4.07 ± 0.18 (scale, 1-5). After multivariable analysis, evaluation scores were associated with Alpha Omega Alpha designation (β=0.13; 95% CI, 0.01-0.25; P=0.03), residency program reputation (β=0.13; 95% CI, 0.05-0.21; P=0.004), and strength of comparative statements in recommendation letters (β=0.08; 95% CI, 0.01-0.15; P=0.02), particularly in letters from residency program directors (β=0.05; 95% CI, 0.01-0.08; P=0.009). Objective factors to consider in the cardiology fellowship application include Alpha Omega Alpha membership, residency program reputation, and comparative statements from residency program directors.

Do USMLE steps, and ITE score predict the American Board of Internal Medicine Certifying Exam results?

BACKGROUND

To evaluate if United States Medical Licensing Examination (USMLE) Step 1, USMLE Step 2 CK, USMLE Step 3, and residency third-year in-service training exam (ITE) scores predict the results of American Board of Internal Medicine Certifying Exam (ABIM-CE).

METHODS

We performed a retrospective review of USMLE Step 1, USMLE Step 2 CK, USMLE Step 3, third-year residency ITE scores and ABIM-CE results of IM residents at our residency program from 2004 through 2017. Statistical analysis was perfrormed using Pearson correlation coefficient, and logistic regression to assess the relationship between USMLE Step 1, USMLE Step 2CK, USMLE Step 3, 3rd year ITE scores and ABIM-CE results. We used Multivariate logistic regression to predict pass or fail results in ABIM-CE based on USMLE and third-year ITE test scores controlling for other covariates.

RESULTS

Among 114 Internal Medicine MD residents included in the study, 92% (n = 105) passed the ABIM-CE. The OR of passing ABIM-CE was 2.70 (95% CI = 1.38-5.29), 2.31 (95% CI = 1.33-4.01), and 1.63 (95% CI = 0.81-3.29) with a ten-point increase in USMLE Step 1, USMLE Step 2 CK and USMLE Step 3 scores respectively. The OR of ABIM-CE passing chance was 2.96 (95% CI = 0.95-9.20), with a ten-point increase in the average score of the above three exams. A 5 % increase in ITE percentage raised the likelihood of passing ABIM-CE (OR 2.92, 95% CI 1.15-7.38). All residents who failed ABIM-CE had Step 1 scores < 220. Among 31 residents with Step 2 CK score < 220, 20% (n = 6) failed ABIM. Similarly, 9% of residents with USMLE Step 3 score < 220 failed ABIM-CE; all residents who failed had scored < 220. The probability curve predicted that the chance of passing ABIM- CE was around 80% with USMLE scores greater than 200 and increased to almost 100% with USMLE scores of 250 or more.

CONCLUSION

USMLE Step 1, USMLE Step 2 CK, and third-year ITE scores can predict the chances of passing ABIM-CE. The third-year ITE score has a higher preditive value compared to USMLE Step 1 and USMLE Step 2 scores. USMLE Step 1 scores more predictive of ABIM-CE results compared to USMLE Step 2CK scores. Thus, residency programs can identify internal medicine residents at risk of failing ABIM-CE and formulate interventions at an early stage during residency training. Measures such as enrolling them in question banks or board review courses can be helpful in improving their chances of passing ABIM-CE.

Mid-Year Medical Knowledge Milestones and ABSITE Scores in First-Year Surgery Residents

OBJECTIVE

Accreditation Council for Graduate Medical Education (ACGME) Surgery milestone ratings in the "Knowledge of Diseases and Conditions" (MK1) sub competency have been shown to correlate with American Board of Surgery In Training Examination (ABSITE) scores, and hypothesized to predict them. To better assess the predictive value of the MK1 milestone and avoid the potential bias caused by previous years' ABSITE scores, we designed a study including only first-year (PGY-1) residents and analyzed the correlation between their mid-year MK1 ratings and their scores in the ABSITE they took approximately a month later.

METHODS

De-identified United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores, mid-year MK1 milestone ratings and the subsequent ABSITE standard scores for the five academic years from 2014-2015 to 2018-2019 were collected and tabulated for 247 PGY-1 preliminary- and categorical-track residents from ten ACGME-accredited surgery residency programs.

RESULTS

The mid-year rating of PGY-1 residents' MK1 was predictive of their subsequent first ABSITE score for the entire cohort and for the categorical residents' subset. Notably, controlling for all other independent predictors, each half-point increase in MK1 rating was associated with a 25-point increase in ABSITE score. Preliminary residents performed significantly worse on the ABSITE, and their scores did not correlate significantly with their MK1 ratings.

CONCLUSIONS

The mid-year rating of PGY-1 residents' MK1 was predictive of their subsequent first ABSITE score for the entire cohort and for the categorical but not the preliminary residents. This finding suggests that evaluators correctly rated MK1 higher in the categorical residents who did perform better on the subsequent ABSITE.

Predictors of USMLE Step 1 Outcomes: Charting Successful Study Habits

BACKGROUND

The United States Medical Licensing Examination Step 1 is a test that affects many aspects of medical students' careers. The aim of this study was to assess the predictive value of various studying habits and academic traits.

ACTIVITY

A survey concerning Step 1 study habits and scores was collected and analyzed.

RESULTS AND DISCUSSION

Study results showed that preclinical curriculum grades, practice test scores, and the number of practice questions completed were positively correlated with Step 1 scores. The strongest predictor of Step 1 scores was preclinical curriculum grades: each unit increase in a letter grade was associated with a 12-point increase in Step 1 scores.

USMLE Scores and Clinical Rotation Role in Predicting ABSITE Performance Among Surgery Interns

BACKGROUND

The American Board of Surgery In-training Examination (ABSITE) is an important marker of medical knowledge. It is unclear what factors predict or improve these scores. We evaluated demographics, United States Medical Licensing Examination (USMLE) step 1 and 2 scores, and surgical rotations during the intern year to determine if there were any correlations with the ABSITE performance.

METHODS

This was a multicenter retrospective review during a 6-y period, investigating the correlation and association of demographics, USMLE scores, and types of rotations on the ABSITE percentile score of interns. Demographics included age, gender, race/ethnicity, U.S. versus international/foreign medical graduate for stratified analyses. Descriptive analysis was performed with ANOVA, correlation was evaluated with 95% confidence interval, and significance was defined as P < 0.05.

RESULTS

Complete records obtained on 89 interns from six different general surgery programs over 6 y revealed that there was a significant correlation between USMLE 1 and 2 with the ABSITE. USMLE 2 correlation was the strongest (r = 0.44, 95% confidence interval = [0.25-0.60], P < 0.05). There was a significant difference in ABSITE performance (mean score difference of 17.3 percentile, P = 0.01) of interns who had an intensive care unit rotation before examination. Other surgical rotations were not associated with an ABSITE difference. Demographic factors such as age, gender, race/ethnicity, or medical graduate background status were not associated with ABSITE scores.

CONCLUSIONS

USMLE step 2 scores had a higher correlation with intern ABSITE performance. An intensive care unit rotation before taking the ABSITE was associated with a significant difference in their percentile scores. Demographic factors were not correlated with ABSITE performance.

Identification of Factors Associated with Hematology-Oncology Fellow Academic Success and Career Choice

Factors affecting hematology-oncology trainees' academic success and career choices have not been well characterized. We performed a retrospective study of 57 hematology-oncology fellows trained at Mayo Clinic between 2008 and 2017 in an attempt to identify factors associated with success during fellowship and with career choice (academic versus private). Sex, age, residency quality, and letters of recommendation indicating a "top" applicant were not associated with hematology or oncology in-training examination (ITE) scores, research productivity (abstracts/publications during fellowship), or career choice (academic versus private). Fellows with higher United States Medical Licensing Examination (USMLE) scores were more likely to perform well on ITE, but examination scores did not predict academic productivity or academic versus private career choice. More academically productive fellows were more likely to choose academic careers. Both ITE scores and productivity were associated with receipt of national and/or institutional awards. Finally, fellows who were non-US citizens and/or international medical graduates (IMG) had higher academic productivity both pre-fellowship and during fellowship and as per the observations above were more likely to choose academic careers. In conclusion, predictors of superior knowledge differ from predictors of academic productivity/career choice, and it is important to take multiple factors into account when selecting candidates most likely to succeed during fellowship.

The USMLE Step 1 Pass/Fail Reporting Proposal: The APDR Position

BACKGROUND

The National Board of Medical Examiners (NBME) and the United States Medical Licensing Examination (USMLE) has convened a conference of "key stakeholders" on March 11-12, 2019 to consider reporting the results of the USMLE Step 1 as pass/fail.

DISCUSSION

While the original purpose of the USMLE Step 1 was to provide an objective basis for medical licensing, the score is increasingly used in residency applicant screening and selection because it is an objective, nationally recognized metric allowing comparison across medical schools in and outside the United States. Excessive reliance on the Step 1 score in the matching process has led to "Step 1 Culture" that drives medical schools to "teach to the test," increases medical student anxiety, and disadvantages minorities that have been shown to score lower on the USMLE Step 1 examination. The outsize role of the USMLE Step 1 score in resident selection is due to lack of standardization in medical school transcripts, grade inflation, and the lack of class standing in many summative assessments. Furthermore, the numeric score allows initial Electronic Residency Application Service filtering, commonly used by programs to limit the number of residency applications to review.

CONCLUSION

The Association of Program Directors in Radiology (APDR) is concerned that pass/fail reporting of the USMLE Step 1 score would take away an objective measure of medical student's knowledge and the incentive to acquire as much of it as possible. Although the APDR is not in favor of the Step 1 exam being used as a screening tool, in the absence of an equal or better metric for applicant comparison the APDR opposes the change in Step 1 reporting from the numeric score to pass/fail.

USMLE Step 2 CK: Best Predictor of Multimodal Performance in an Internal Medicine Residency

BACKGROUND

Internal medicine (IM) residency programs receive information about applicants via academic transcripts, but studies demonstrate wide variability in satisfaction with and usefulness of this information. In addition, many studies compare application materials to only 1 or 2 assessment metrics, usually standardized test scores and work-based observational faculty assessments.

OBJECTIVE

We sought to determine which application materials best predict performance across a broad array of residency assessment outcomes generated by standardized testing and a yearlong IM residency ambulatory long block.

METHODS

In 2019, we analyzed available Electronic Residency Application Service data for 167 categorical IM residents, including advanced degree status, research experience, failures during medical school, undergraduate medical education award status, and United States Medical Licensing Examination (USMLE) scores. We compared these with post-match residency multimodal performance, including standardized test scores and faculty member, peer, allied health professional, and patient-level assessment measures.

RESULTS

In multivariate analyses, USMLE Step 2 Clinical Knowledge (CK) scores were most predictive of performance across all residency performance domains measured. Having an advanced degree was associated with higher patient-level assessments (eg, physician listens, physician explains, etc). USMLE Step 1 scores were associated with in-training examination scores only. None of the other measured application materials predicted performance.

CONCLUSIONS

USMLE Step 2 CK scores were the highest predictors of residency performance across a broad array of performance measurements generated by standardized testing and an IM residency ambulatory long block.

Factors that Predict an Intern's First ABSITE Score are Known by September

BACKGROUND

Previous studies offer conflicting relevance of a variety of factors to predict resident performance on the The American Board of Surgery In-Training Exam (ABSITE). With numerous stellar applicants scoring poorly on their first ABSITE, we sought to identify key factors that might allow us to tailor pre-emptive study efforts in the fall and early winter to enhance scores.

METHODS

General Surgery residents in our program from 2009 through 2016 were included in our cohort study. Specific trainee data (sex, prelim vs categorical, United States Medical Licensing Examination (USMLE) Step 1 and 2 scores, ABSITE scores, clinical rotations, biannual objective structured clinical examination OSCE-type scores, and in-house prep test [IHPT], etc.) were collected retrospectively. The data were analyzed using JMP pro 10 and MedCalc.

RESULTS

ABSITE scores of our 110 trainees did not vary by sex or by categorical vs preliminary residents. USMLE step 1 and 2, IHPT and one objective structured clinical examination (OSCE) station (Sim-Based Trauma Exam) scores were positively correlated with ABSITE scores (p < 0.05; correlation coefficient (CC) = 0.6 [strong] for IHPT, 0.5 [moderate] for step 1 and 2 and [weak] 0.3 for Sim-Based Trauma Exam). The mean (standard deviation) ABSITE %tile score for residents scoring above 230 (USMLE 1 or 2) vs <230 were different: 81 (3) vs 56 (4) (p < 000.1). Of residents scoring lower than 230 in USMLE 1 or 2, their ABSITE performance had a strong positive linear correlation with their performance in the IHPT (CC = 0.7) and SBTA (CC = 0.5). Residents rotating twice on the Acute Care Surgery services before the ABSITE scored higher than those with 1 or zero rotations (p < 0.05).

CONCLUSIONS

USMLE steps 1 and 2 are useful parameters in our program to predict subsequent resident ABSITE performance. An in-house 60-minute preparation test in September, a 6-minute simulation-based trauma assessment in July, and rotation schedule with 2 stints on Acute Care Surgery were surprisingly useful early intern year tools to predict ABSITE scores.

Does the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination predict 'on-the-job' performance during UK higher specialty surgical training?

INTRODUCTION

The Intercollegiate Membership of the Royal College of Surgeons (MRCS) is a mandatory examination to enter higher surgical specialty training in the UK. It is designed to help to ensure that successful candidates are competent to practice as higher surgical trainees. The annual review of competence progression (ARCP) assesses trainees' competence to progress to the next level of training and can be interpreted as a measure of 'on-the-job' performance. We investigated the relationship between MRCS performance and ARCP outcomes.

MATERIALS AND METHODS

All UK medical graduates who passed MRCS (Parts A and B) from 2007 to 2016 were included. MRCS scores, attempts and sociodemographics for each candidate were crosslinked with ARCP outcomes (satisfactory, unsatisfactory and insufficient evidence). Multinomial logistic regression was used to identify potential independent predictors of ARCP outcomes.

RESULTS

A total of 2570 trainees underwent 11,064 ARCPs; 1589 (61.8%) had only satisfactory outcomes recorded throughout training; 510 (19.9%) had at least one unsatisfactory outcome; and 471 (18.3%) supplied insufficient evidence. After adjusting for age, gender, first language and Part A performance, ethnicity (non-white vs white, OR 1.36, 95% CI 1.08 to 1.71), Part B passing score (OR 0.98, 95% CI 0.98 to 1.00) and number of attempts at Part B (two or more attempts vs one attempt, OR 1.50, 95% CI 1.16 to 1.94) were found to be independent predictors of an unsatisfactory ARCP outcome.

CONCLUSIONS

This is the first study to identify predictors of ARCP outcomes during higher surgical specialty training in the UK and provides further evidence of the predictive validity of the MRCS examination.

Do Residency Selection Factors Predict Radiology Resident Performance?

RATIONALE AND OBJECTIVES

The purpose of our study is to determine what information in medical student residency applications predicts radiology residency success as defined by objective clinical performance data.

MATERIALS AND METHODS

We performed a retrospective cohort study of residents who entered our institution's residency program through the National Resident Matching Program as postgraduate year 2 residents and completed the program over the past 2 years. Medical school grades, selection to Alpha Omega Alpha (AOA) Honor Society, United States Medical Licensing Examination (USMLE) scores, publication in peer-reviewed journals, and whether the applicant was from a peer institution were the variables examined. Clinical performance was determined by calculating each resident's cumulative major discordance rate for on-call cases the resident read and gave a preliminary interpretation. A major discordance was defined as a difference between the preliminary resident and the final attending interpretations that could immediately impact the care of the patient. A multivariate logistic regression was performed to determine significant variables.

RESULTS

Twenty-seven residents provided preliminary reports on call for 67,145 studies. The mean major discordance rate was 1.08% (range 0.34%-2.54%). Higher USMLE Step 1 scores, publication before residency, and election to AOA Honor Society were all statistically significant predictors of lower major discordance rates (P values 0.01, 0.01,  and <0.001, respectively).

CONCLUSIONS

Overall resident performance was excellent. There are predictors that help select the better performing residents, namely higher USMLE Step 1 scores, one to two publications during medical school, and election to AOA in the junior year of medical school.

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

BACKGROUND

Selection into specialty training is a high-stakes and resource-intensive process. While substantial literature exists on selection into medical schools, and there are individual studies in postgraduate settings, there seems to be paucity of evidence concerning selection systems and the utility of selection tools in postgraduate training environments.

AIM

To explore, analyze and synthesize the evidence related to selection into postgraduate medical specialty training.

METHOD

Core bibliographic databases including PubMed; Ovid Medline; Embase, CINAHL; ERIC and PsycINFO were searched, and a total of 2640 abstracts were retrieved. After removing duplicates and screening against the inclusion criteria, 202 full papers were coded, of which 116 were included.

RESULTS

Gaps in underlying selection frameworks were illuminated. Frameworks defined by locally derived selection criteria, and heavily weighed on academic parameters seem to be giving way to the evidencing of competency-based selection approaches in some settings. Regarding selection tools, we found favorable psychometric evidence for multiple mini-interviews, situational judgment tests and clinical problem-solving tests, although the bulk of evidence was mostly limited to the United Kingdom. The evidence around the robustness of curriculum vitae, letters of recommendation and personal statements was equivocal. The findings on the predictors of past performance were limited to academic criteria with paucity of long-term evaluations. The evidence around nonacademic criteria was inadequate to make an informed judgment.

CONCLUSIONS

While much has been gained in understanding the utility of individual selection methods, though the evidence around many of them is equivocal, the underlying theoretical and conceptual frameworks for designing holistic and equitable selection systems are yet to be developed.

Impact of performance in a mandatory postgraduate surgical examination on selection into specialty training

Background

The Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination is undertaken by large numbers of trainees in the UK and internationally as a mandatory step within surgical training. Unlike some high-stakes medical examinations, the MRCS is yet to be validated. A quantitative study was undertaken to assess its predictive validity by investigating the relationship between MRCS (Parts A and B) and national selection interview scores for general and vascular surgery in the UK.

Methods

Pearson correlation coefficients were used to examine the linear relationship between each assessment, and linear regression analyses were employed to identify potential independent predictors of the national selection score. All UK medical graduates who attempted the interview in 2011-2015 were included.

Results

Some 84·4 per cent of the candidates (1231 of 1458) were matched with MRCS data. There was a significant positive correlation between the first attempt score at Part B of the MRCS examination and the national selection score (r = 0·38, P < 0·001). In multivariable analysis, 17 per cent of variance in the national selection first attempt score was explained by the Part B MRCS score and number of attempts (change in R ² value of 0·10 and 0·07 respectively; P < 0·001). Candidates who required more than two attempts at Part B were predicted to score 8·1 per cent less than equally matched candidates who passed at their first attempt.

Conclusion

This study supports validity of the MRCS examination, and indicates its predictive value regarding entry into specialist training.

USMLE Scores Predict Success in ABEM Initial Certification: A Multicenter Study

Introduction

There are no existing data on whether performance on the United States Medical Licensing Examination (USMLE) predicts success in American Board of Emergency Medicine (ABEM) certification. The aim of this study was to determine the presence of any association between USMLE scores and first-time success on the ABEM qualifying and oral certification examinations.

Methods

We retrospectively collected USMLE Step 1, Step 2 Clinical Knowledge (CK) scores and pass/fail results from the first attempt at ABEM qualifying and oral examinations from residents graduating between 2009 and 2011 from nine EM programs. A composite score was defined as the sum of USMLE Step 1 and Step 2 CK scores.

Results

Sample was composed of 197 residents. Median Step 1, Step 2 CK and composite scores were 218 ([IQR] 207–232), 228 (IQR 217–239) and 444 (IQR 427–468). First-time pass rates were 95% for the qualifying examination and 93% for both parts of the examination. Step 2 CK and composite scores were better predictors of achieving ABEM initial certification compared to Step 1 score (area under the curve 0.800, 0.759 and 0.656). Step 1 score of 227, Step 2 CK score of 225 and composite score of 444 predicted a 95% chance of passing both boards.

Conclusion

Higher USMLE Step 1, Step 2 CK and composite scores are associated with better performance on ABEM examinations, with Step 2 CK being the strongest predictor. Cutoff scores for USMLE Step 1, Step 2 CK and composite score were established to predict first-time success on ABEM initial certification.

The association of USMLE Step 1 and Step 2 CK scores with residency match specialty and location

BACKGROUND

For future physicians, residency programs offer necessary extended training in specific medical specialties. Medical schools benefit from an understanding of factors that lead their students to match into certain residency specialties. One such factor, often used during the residency application process, is scores on the USA Medical Licensing Exam (USMLE).

OBJECTIVES

To determine the relationship between USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores and students' residency specialty match, and the association between both USMLE scores and state of legal residency (Minnesota) at the time of admission with students staying in-state or leaving the state for residency program.

DESIGN

USMLE scores and residency match data were analyzed from five graduating classes of students at the University of Minnesota Medical School (N = 1054).

RESULTS

A MANOVA found significant differences (p < 0.001) between residency specialties and both USMLE Step 1 and Step 2 CK scores, as well as the combination of the two. Students who matched in Dermatology had the highest mean USMLE scores overall, while students who matched in Family Medicine had the lowest mean scores. Students who went out of state for residency had significantly higher Step 1 scores (p = 0.027) than students who stayed in-state for residency, while there was no significant difference between the groups for Step 2 scores. A significant positive association was found between a student who applied as a legal resident of Minnesota and whether the student stayed in Minnesota for their residency program.

CONCLUSIONS

Residency specialty match was significantly associated with USMLE Step 1 and USMLE Step 2 CK scores, as was staying in-state or leaving the state for residency. Students who were legal residents of the state at the time of application were more likely to stay in-state for residency, regardless of USMLE score.

ABBREVIATIONS

CK: Clinical knowledge; COMLEX: Comprehensive Osteopathic Medical Licensing Examination; GME: Graduate medical education; NRMP: National Resident Matching Program; UME: Undergraduate medical education; USMLE: United States Medical Licensing Examination.

Identifying Best Practices for and Utilities of the Pharmacy Curriculum Outcome Assessment Examination

Objective. A review was conducted to determine implementation strategies, utilities, score interpretation, and limitations of the Pharmacy Curriculum Outcome Assessment (PCOA) examination. Methods. Articles were identified through the PubMed and American Journal of Pharmaceutical Education, and International Pharmaceutical Abstracts databases using the following terms: "Pharmacy Curriculum Outcomes Assessment," "pharmacy comprehensive examination," and "curricular assessment." Studies containing information regarding implementation, utility, and predictive values for US student pharmacists, curricula, and/or PGY1/PGY2 residents were included. Publications from the Academic Medicine Journal, the Accreditation Council for Pharmacy Education (ACPE), and the American Association of Colleges of Pharmacy (ACCP) were included for background information and comparison of predictive utilities of comprehensive examinations in medicine. Results. Ten PCOA and nine residency-related publications were identified. Based on published information, the PCOA may be best used as an additional tool to identify knowledge gaps for third-year student pharmacists. Conclusion. Administering the PCOA to students after they have completed their didactic coursework may yield scores that reflect student knowledge. Predictive utility regarding the North American Pharmacy Licensure Examination (NAPLEX) and potential applications is limited, and more research is required to determine ways to use the PCOA.

The Association Between Premedical Curricular and Admission Requirements and Medical School Performance and Residency Placement: A Study of Two Admission Routes

PURPOSE

The curricular elements of undergraduate premedical education are the subject of an ongoing debate. The Warren Alpert Medical School of Brown University (AMS) matriculates students via the traditional premedical route (TPM) and an eight-year baccalaureate/MD program-the Program in Liberal Medical Education (PLME)-which provides students with a broad and liberal education. Using the juxtaposition of these two admission routes, the authors aimed to determine whether there is an association between highly distinct premedical curricular and admission requirements and medical school performance and residency placement.

METHOD

The cohorts studied included all of the PLME (n = 295) and TPM (n = 215) students who graduated from the AMS between 2010 and 2015. Outcome variables consisted of multiple measures of medical school performance, including standardized multiple-choice examination scores and honors grades, and residency placement. The authors employed unadjusted tests of averages and proportions (independent t tests and chi-square tests) to compare variables.

RESULTS

The TPM students attained marginally, but statistically significantly, higher average scores on standardized multiple-choice examinations than their PLME counterparts. The number of undergraduate premedical science courses completed by PLME students accounted for less than 4% of the variance in key metrics of medical school performance. The residency placement record of the PLME and TPM cohorts proved comparable.

CONCLUSIONS

These findings suggest that the association between medical school performance and residency placement and undergraduate premedical curricular and admission requirements is weak. Further study is needed to determine the optimal premedical preparation of students.

A Plea to Reassess the Role of United States Medical Licensing Examination Step 1 Scores in Residency Selection

The three-step United States Medical Licensing Examination (USMLE) was developed by the National Board of Medical Examiners and the Federation of State Medical Boards to provide medical licensing authorities a uniform evaluation system on which to base licensure. The test results appear to be a good measure of content knowledge and a reasonable predictor of performance on subsequent in-training and certification exams. Nonetheless, it is disconcerting that the test preoccupies so much of students' attention with attendant substantial costs (in time and money) and mental and emotional anguish. There is an increasingly pervasive practice of using the USMLE score, especially the Step 1 component, to screen applicants for residency. This is despite the fact that the test was not designed to be a primary determinant of the likelihood of success in residency. Further, relying on Step 1 scores to filter large numbers of applications has unintended consequences for students and undergraduate medical education curricula. There are many other factors likely to be equally or more predictable of performance during residency. The authors strongly recommend a move away from using test scores alone in the applicant screening process and toward a more holistic evaluation of the skills, attributes, and behaviors sought in future health care providers. They urge more rigorous study of the characteristics of students that predict success in residency, better assessment tools for competencies beyond those assessed by Step 1 that are relevant to success, and nationally comparable measures from those assessments that are easy to interpret and apply.

Predicting and enhancing American Board of Surgery In-Training Examination performance: does writing questions really help?

BACKGROUND

The generative learning model posits that individuals remember content they have generated better than materials created by others. The goals of this study were to evaluate question generation as a study method for the American Board of Surgery In-Training Examination (ABSITE) and determine whether practice test scores and other data predict ABSITE performance.

METHODS

Residents (n = 206) from 6 general surgery programs were randomly assigned to one of the two study conditions. One group wrote questions for practice examinations. All residents took 2 practice examinations.

RESULTS

There was not a significant effect of writing questions on ABSITE score. Practice test scores, United States Medical Licensing Examination Step 1 scores, and previous ABSITE scores were significantly correlated with ABSITE performance.

CONCLUSIONS

The generative learning model was not supported. Performance on practice tests and other data can be used for early identification of residents at risk of performing poorly on the ABSITE.

Identifying and Promoting Best Practices in Residency Application and Selection in a Complex Academic Health Network

Medical education institutions have a social mandate to produce a diverse physician workforce that meets the public's needs. Recent reports have framed the admission process outcome of undergraduate and postgraduate medical education (UGME and PGME) programs as a key determinant of the collective contributions graduating cohorts will make to society, creating a sense of urgency around the issue of who gets accepted. The need for evidence-informed residency application and selection processes is growing because of the increasing size and diversity of the applicant pool and the need for equity, fairness, social accountability, and health human resource planning. The selection literature, however, is dominated by a UGME focus and emphasizes determination of desirable qualities of future physicians and selection instrument reliability and validity. Gaps remain regarding PGME selection, particularly the creation of specialty-specific selection criteria, suitable outcome measures, and reliable selection systems.In this Perspective, the authors describe the University of Toronto's centralized approach to defining system-level best practices for residency application and selection. Over the 2012-2013 academic year, the Best Practices in Application and Selection working group reviewed relevant literature and reports, consulted content experts, surveyed local practices, and conducted iterative stakeholder consultations on draft recommendations. Strong agreement arose around the resulting 13 principles and 24 best practices, which had either empirical support or face validity. These recommendations, which are shared in this article, have been adopted by the university's PGME advisory committee and will inform a national initiative to improve trainees' transition from UGME to PGME in Canada.

General Surgery Residents can Be a Reliable Resource in the Evaluation of Residency Applications

OBJECTIVE

Surgical residents' ability to screen general surgery (GS) applicants has not been previously investigated. The objective of this study was to compare surgical residents' evaluation of Electronic Residency Application Service (ERAS) applicants to that of faculty using a standardized assessment instrument.

DESIGN

A prospective analysis of ERAS applications using a standardized assessment tool.

SETTING

A university-affiliated, academic, county GS residency program.

PARTICIPANTS

Before the interview day, 51 ERAS (2013-2014) applications were reviewed by 10 different assessors (6 GS faculty, including the program director, and 4 GS residents), who evaluated applicants on 10 characteristics (subjective and objective) using a 5-point Likert scale, a total score, and a Global Rating Scale that ranked candidates into deciles.

RESULTS

There were a total of 510 assessments. In 8 of 10 individual domains the interrater reliability (IRR) between residents and faculty was good. The IRRs of the total score and global score were excellent. The Spearman ρ between the total score and final rank list were similar for faculty (-0.558) and residents (-0.592).

CONCLUSIONS

The excellent IRR score between the total and global scores of faculty and residents demonstrates the reliability of GS residents in evaluating ERAS applications. The low correlations between the total score and final rank are consistent with those in previous studies, in which the interview has been demonstrated to be the most important factor in determining final selection.