Do USMLE steps, and ITE score predict the American Board of Internal Medicine Certifying Exam results?

Resident Recruitment in a New Era

ABSTRACT

This chapter focuses on resident recruitment and recent US National Resident Matching Program changes and the impact in the evaluation and ranking of applicants within the specialty of anesthesiology. Recruitment challenges are examined as well as program strategies and potential future directions. Also discussed are DEI initiatives within the recruitment process.

Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations?

PURPOSE

The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency.

METHODS

The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics.

RESULTS

Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ± 63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively.

CONCLUSION

USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.

Resident factors associated with American board of internal medicine certification exam failure

INTRODUCTION

Performance on the certifying examinations such as the American Board of Internal Medicine Certification Exam (ABIM-CE) is of great interest to residents and their residency programs. Identification of factors associated with certification exam result may allow residency programs to recognize and intervene for residents at risk of failing. Despite this, residency programs have few evidence-based predictors of certification exam outcome. The change to pass-or-fail score reporting of the USA Medical Licensing Exam (USMLE) Step 1 removes one such predictor.

MATERIALS AND METHODS

We performed a retrospective study of residents from a medium-sized internal medicine residency program who graduated from 1998 through 2017. We used univariate tests of associations between ABIM-CE result and various demographic and scholastic factors.

RESULTS

Of 166 graduates, 14 (8.4%) failed the ABIM-CE on the first attempt. Failing the first attempt of the ABIM-CE was associated with older median age on entering residency (29 vs 27 years; P = 0.01); lower percentile rank on the Internal Medicine In-Training Examination (IM-ITE) in each of the first, second, and third years of training (P < 0.001 for all); and lower scores on the USMLE Steps 1, 2 Clinical Knowledge, and 3 (P < 0.05 for all). No association was seen between a variety of other scholastic or demographic factors and first-attempt ABIM-CE result.

DISCUSSION

Although USMLE step 1 has changed to a pass-or-fail reporting structure, there are still other characteristics that allow residency programs to identify residents at risk of ABIM-CE first time failure and who may benefit from intervention.

The Upcoming Pass/Fail USMLE Step 1 Score Reporting: An Impact Assessment From Medical School Deans

Background: The US Medical Licensing Examination (USMLE) Step 1 change to pass/fail has been met with mixed reviews, and the impact on medical student education and residency match is unknown. We surveyed medical school student affairs deans regarding their thoughts on the upcoming transition of Step 1 to pass/fail. Methods: A questionnaire was emailed to medical school deans. Deans were asked to rank the importance of the following after the Step 1 reporting change: Step 2 Clinical Knowledge (Step 2 CK), clerkship grades, letters of recommendation, personal statement, medical school reputation, class rank, Medical Student Performance Evaluation, and research. They were asked how the score change will affect curriculum, learning, diversity, and student mental health. Deans were asked to select 5 specialties they thought would be most affected. Results: Regarding perceived importance of residency applications following the scoring change, the most frequent number 1 choice was Step 2 CK. The majority of deans (93.5%, n = 43) felt that the change to pass/fail would benefit medical student education/learning environment; however, most (68.2%, n = 30) did not believe their school curriculum would change. Students applying to dermatology, neurosurgery, orthopedic surgery, ENT, and plastic surgery were felt to be most affected by the scoring change; 58.7% (n = 27) felt it would not adequately address future diversity. Conclusion: The majority of deans feel the USMLE Step 1 change to pass/fail would benefit medical student education. Deans feel that students applying to traditionally more competitive specialties (ie, programs with fewer overall residency positions available) will be most affected.

The Association Between USMLE Step 2 Clinical Knowledge Scores and Residency Performance: A Systematic Review and Meta-Analysis

PURPOSE

With the change in Step 1 score reporting, Step 2 Clinical Knowledge (CK) may become a pivotal factor in resident selection. This systematic review and meta-analysis seeks to synthesize existing observational studies that assess the relationship between Step 2 CK scores and measures of resident performance.

METHOD

The authors searched MEDLINE, Web of Science, and Scopus databases using terms related to Step 2 CK in 2021. Two researchers identified studies investigating the association between Step 2 CK and measures of resident performance and included studies if they contained a bivariate analysis examining Step 2 CK scores' association with an outcome of interest: in-training examination (ITE) scores, board certification examination scores, select Accreditation Council for Graduate Medical Education core competency assessments, overall resident performance evaluations, or other subjective measures of performance. For outcomes that were investigated by 3 or more studies, pooled effect sizes were estimated with random-effects models.

RESULTS

Among 1,355 potential studies, 68 met inclusion criteria and 43 were able to be pooled. There was a moderate positive correlation between Step 2 CK and ITE scores (0.52, 95% CI 0.45-0.59, P < .01). There was a moderate positive correlation between Step 2 CK and ITE scores for both nonsurgical (0.59, 95% CI 0.51-0.66, P < .01) and surgical specialties (0.41, 95% CI 0.33-0.48, P < .01). There was a very weak positive correlation between Step 2 CK scores and subjective measures of resident performance (0.19, 95% CI 0.13-0.25, P < .01).

CONCLUSIONS

This study found Step 2 CK scores have a statistically significant moderate positive association with future examination scores and a statistically significant weak positive correlation with subjective measures of resident performance. These findings are increasingly relevant as Step 2 CK scores will likely become more important in resident selection.

Evaluation of house staff candidates for program fit: a cohort-based controlled study

BACKGROUND

Medical school academic achievements do not necessarily predict house staff job performance. This study explores a selection mechanism that improves house staff-program fit that enhances the Accreditation Council for Graduate Medical Education Milestones performance ratings.

OBJECTIVE

Traditionally, house staff were selected primarily on medical school academic performance. To improve residency performance outcomes, the Program designed a theory-driven selection tool to assess house staff candidates on their personal values and goals fit with Program values and goals. It was hypothesized cohort performance ratings will improve because of the intervention.

METHODS

Prospective quasi-experimental cohort design with data from two house staff cohorts at a university-based categorical Internal Medicine Residency Program. The intervention cohort, comprising 45 house staff from 2016 to 2017, was selected using a Behaviorally Anchored Rating Scales (BARS) tool for program fit. The control cohort, comprising 44 house staff from the prior year, was selected using medical school academic achievement scores. House staff performance was evaluated using ACGME Milestones indicators. The mean scores for each category were compared between the intervention and control cohorts using Student's t-tests with Bonferroni correction and Cohen's d for effect size.

RESULTS

The cohorts were no different in academic performance scores at time of Program entry. The intervention cohort outperformed the control cohort on all 6 dimensions of Milestones by end-PGY1 and 3 of 6 dimensions by mid-PGY3.

CONCLUSION

Selecting house staff based on compatibility with Residency Program values and objectives may yield higher job performance because trainees benefit more from a better fit with the training program.

Medical Schools as Racialized Organizations: How Race-Neutral Structures Sustain Racial Inequality in Medical Education-a Narrative Review

In 2021, The American Association of Medical Colleges released a framework addressing structural racism in academic medicine, following the significant, nationwide Movement for Black Lives. The first step of this framework is to "begin self-reflection and educating ourselves." Indeed, ample evidence shows that medical schools have a long history of racially exclusionary practices. Drawing on racialized organizations theory from the field of sociology, we compile and examine scholarship on the role of race and racism in medical training, focusing on disparities in educational and career outcomes, experiences along racial lines in medical training, and long-term implications. From the entrance into medical school through the residency application process, organizational factors such as reliance on standardized tests to predict future success, a hostile learning climate, and racially biased performance metrics negatively impact the careers of trainees of color, particularly those underrepresented in medicine (URiM). Indeed, in addition to structural biases associated with otherwise "objective" metrics, there are racial disparities across subjective outcomes such as the language used in medical trainees' performance evaluations, even when adjusting for grades and board exam scores. These disadvantages contribute to URIM trainees' lower odds of matching, steering into less competitive and lucrative specialties, and burnout and attrition from academic careers. Additionally, hostile racial climates and less diverse medical schools negatively influence White trainees' interest in practicing in underserved communities, disproportionally racial and ethnic minorities. Trainees' mental health suffers along the way, as do medical schools' recruitment, retention, diversity, and inclusion efforts. Evidence shows that seemingly race-neutral processes and structures within medical education, in conjunction with individuals' biases and interpersonal discrimination, may reproduce and sustain racial inequality among medical trainees. Medical schools whose goals include training a more diverse physician workforce towards addressing racial health disparities require a new playbook.

Association of Gender and Race/Ethnicity with Internal Medicine In-Training Examination Performance in Graduate Medical Education

BACKGROUND

Disparities in objective assessments in graduate medical education such as the In-Training Examination (ITE) that disadvantage women and those self-identifying with race/ethnicities underrepresented in medicine (URiM) are of concern.

OBJECTIVE

Examine ITE trends longitudinally across post-graduate year (PGY) with gender and race/ethnicity.

DESIGN

Longitudinal analysis of resident ITE metrics at 7 internal medicine residency programs, 2014-2019. ITE trends across PGY of women and URiM residents compared to non-URiM men assessed via ANOVA. Those with ITE scores associated with less than 90% probability of passing the American Board of Internal Medicine certification exam (ABIM-CE) were identified and odds of being identified as at-risk between groups were assessed with chi square.

PARTICIPANTS

A total of 689 IM residents, including 330 women and URiM residents (48%).

MAIN MEASURES

ITE score KEY RESULTS: There was a significant difference in ITE score across PGY for women and URiM residents compared to non-URiM men (F_{(2, 1321)} 4.46, p=0.011). Adjusting for program, calendar year, and baseline ITE, women and URiM residents had smaller ITE score gains (adjusted mean change in score between PGY1 and PGY3 (se), non-URiM men 13.1 (0.25) vs women and URiM residents 11.4 (0.28), p<0.001). Women and URiM residents had greater odds of being at potential risk for not passing the ABIM-CE (OR 1.75, 95% CI 1.10 to 2.78) with greatest odds in PGY3 (OR 3.13, 95% CI 1.54 to 6.37).

CONCLUSION

Differences in ITE over training were associated with resident gender and race/ethnicity. Women and URiM residents had smaller ITE score gains across PGY translating into greater odds of potentially being seen as at-risk for not passing the ABIM-CE. Differences in ITE over training may reflect differences in experiences of women and URiM residents during training and may lead to further disparities.

Development of a Longitudinal Research Curriculum for Pediatric Emergency Medicine Fellowship

n/a

Mission-Based Filters in the Electronic Residency Application Service: Saving Time and Promoting Diversity

BACKGROUND

Holistic review promotes diversity, but widespread implementation remains limited.

OBJECTIVE

We aimed to develop a practical approach to incorporate holistic review principles in screening applicants in the Electronic Residency Application Service (ERAS) and to assess the impact on diversity.

METHODS

Three residency programs (internal medicine [IM], pediatrics, and surgery) at McGovern Medical School developed filters to identify applicants with experiences/attributes aligned with the institutional mission. These filters were retroactively applied to each program's 2019-2020 applicant pool using built-in ERAS capabilities to group applicants by user-defined features. We compared the demographics of applicants reviewed during the cycle with those identified retrospectively through experiences/attributes filters.

RESULTS

The IM, pediatrics, and surgery programs received 3527, 1341, and 1313 applications, respectively, in 2019-2020. Retrospective use of experiences/attributes filters, without scores, narrowed the IM applicant pool for review to 1301 compared to 1323 applicants reviewed during actual recruitment, while the pediatrics filters identified 514 applicants compared to 384 at baseline. The surgery filters resulted in 582 applicants, but data were missing for baseline comparison. Compared to the baseline screening approach utilizing scores, mission-based filters increased the proportions of underrepresented in medicine applicants selected for review in IM (54.8% [95% CI 52.1-57.5] vs 22.7% [20.4-24.9], P < .0001) and pediatrics (63.2% [95% CI 59.1-67.4] vs 25.3% [20.9-29.6], P < .0001).

CONCLUSIONS

Program directors can leverage existing ERAS features to conduct application screening in alignment with holistic review principles. Widespread implementation could have important repercussions for enhancing physician workforce diversity.

Curriculum Transition From Lecture-Based to Team-Based Learning is Associated With Improved Performance on Internal Medicine In-Training Examination

BACKGROUND

Team-based learning (TBL) is an alternative to traditional lectures in graduate medical education, but evidence is scarce regarding its impact on knowledge acquisition and standardized testing performance.

OBJECTIVE

We examined the association between resident performance on the Internal Medicine In-Training Examination (IM-ITE) and these 2 educational methods.

METHODS

In 2013, the internal medicine residency program at Albany Medical College transitioned from a lecture-based curriculum to TBL. Residents enrolled in academic years 2011-2012 and 2012-2013 comprised the lecture cohort, and those enrolled in 2015-2016 and 2016-2017 the TBL cohort. Covariates included the type of medical school attended, gender, and United States Medical Licensing Examination Step 2 Clinical Knowledge scores. We performed univariate analysis and multivariable regression to determine the association between covariates and ITE scores.

RESULTS

Of 120 residents, 60 were in the lecture cohort and 60 in the TBL cohort. The IM-ITE percent correct scores were higher with TBL than lecture (PGY-1 61.0% vs 55.0%, P < .001; PGY-2 69.0% vs 59.7%, P < .001; PGY-3 73.2% vs 61.7%, P < .001). In a multivariable regression analysis of 3 PGYs combined, the transition from lecture to TBL resulted in an increase in IM-ITE Z-score of 0.415 (P < .001), equivalent to 0.415 SD, when including the effects of all covariates.

CONCLUSIONS

Compared to a lecture-based curriculum, TBL was associated with improved resident medical knowledge acquisition as evidenced by higher IM-ITE scores.

US Medical Licensing Exam Performance and American Board of Surgery Qualifying and Certifying Examinations

BACKGROUND

Program directors use United States Medical Licensing Exam (USMLE) scores as criteria for ranking applicants. First-time pass rates of the American Board of Surgery (ABS) Qualifying (QE) and Certifying (CE) Examinations have become important measures of residency program quality. USMLE Step 1 will become pass/fail in 2022.

STUDY DESIGN

American Board of Surgery QE and CE success rates were assessed considering multiple characteristics of highly ranked (top 20) applicants to 22 general surgery programs in 2011. Chi-square, t-test, Wilcoxon Rank sum, linear and logistic regression were used, as appropriate.

RESULTS

The QE and CE first attempt pass rates were 96% (235/244) and 86% (190/221), respectively. QE/CE success was not significantly associated with sex, race, research experience, or publications. Alpha Omega Alpha (AΩA) status was associated with success on the index CE (98% vs 83%; p = 0.008). Step 1 and Step 2 Clinical Knowledge (CK) scores of surgeons who passed QE on their first attempt were higher than scores of those who failed (Step 1: 233 vs 218; p = 0.016); (Step 2CK: 244 vs 228, p = 0.009). For every 10-point increase in Step 1 and 2CK scores, the odds of passing CE on the first attempt increased 1.5 times (95% CI 1.12, 2.0; p = 0.006) and 1.5 times (95% CI 1.11, 2.02, p = 0.008), respectively. For every 10-point increase in Steps 1 and 2CK scores, the odds of passing the QE on the first attempt increased 1.85 times (95% CI 1.11, 3.09; p = 0.018) and 1.86 times (95% CI 1.14, 3.06, p = 0.013), respectively.

CONCLUSIONS

USMLE Step 1 and Step 2 CK examination scores correlate with American Board of Surgery QE and CE performance and success. The USMLE decision to transition Step 1 to a pass/fail examination will require program directors to identify other factors that predict ABS performance for ranking applicants.

Evidence-Based Selection of Surgical Residents

Residency programs should use a systematic method of recruitment that begins with defining unique desired candidate attributes. Commonly sought-after characteristics may be delineated via the residency application. Scores from standardized examinations taken in medical school predict academic success, and may correlate to overall performance. Strong letters of recommendation and a personal history of prior success outside the medical field both forecast success in residency. Interviews are crucial to determining fit within a program, and remain a valid measure of an applicant's ability to prosper in a particular program, even with many interviews being completed in the virtual realm.

Do the selection criteria of internal medicine residency program predict resident performance?

BACKGROUND

Well-performing physician reflects the success of the residency program in selecting the best candidates for training. This study aimed to evaluate the selection criteria, mainly the United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) results and applicants' status as international or locally trained applicants, used by the medical education department and the internal medicine residency program in Hamad Medical Corporation in Qatar to predict the residents' performance during their training.

METHODS

A retrospective chart review was performed for three batches of graduates who started residency training in 2011, 2012, and 2013. Each group completed 4 years of training. The USMLE Step 2 CK status of the applicant, in-training exam (ITE) scores, formative evaluation scores, Arab Board written and clinical exams pass rate, and other indicators were analyzed. Statistical analysis included chi squares and independent t-test to identify associations. Multivariable analyses were conducted using logistic and linear regressions to test for adjusted associations.

RESULTS

The study included 118 (81 international/37 locally trained applicants) internal medicine residents. The ITE score correlated positively with the USMLE Step 2 CK score (r = 0.621, r = 0.587, r = 0.576, r = 0.571, p < 0.001) over the 4 years of training and among the international compared with locally trained applicants (p < 0.001). The rate of passing part 1 and 2 written exam of the Arab Board was higher in international than in local applicants, whereas clinical Arab Board exam and formative evaluation were not associated with any criteria.

CONCLUSIONS

Higher USMLE Step 2 CK score correlated with better performance on ITE but not with other performance indicators, whereas international applicants did better in both ITE and Arab Board written exam than local applicants. These variables may provide reasonable predictors of well-performing physicians.

Assessment of Plastic Surgery Residency Applications without United States Medical Licensing Examination Step 1 Scores

SUMMARY

The United States Medical Licensing Examination announced the changing of Step 1 score reporting from a three-digit number to pass/fail beginning on January 1, 2022. Plastic surgery residency programs have traditionally used United States Medical Licensing Examination Step 1 scores to compare plastic surgery residency applicants. Without a numerical score, the plastic surgery residency application review process will likely change. This article discusses advantages, disadvantages, and steps forward for residency programs related to the upcoming change. The authors encourage programs to continue to seek innovative methods of objectively and holistically evaluating applications.

Relationship Between Standardized Test Scores and Board Certification Exams in a Combined Internal Medicine/Pediatrics Residency Program

Background

Combined Internal Medicine/Pediatrics (Med/Peds) residencies rely on categorical program data to predict pass rates for the American Board of Internal Medicine Certifying Exam (ABIM-CE) and the American Board of Pediatrics Certifying Exam (ABP-CE). There is insufficient literature describing what best predicts a Med/Peds resident passing board exams. In this study, we aimed to determine how standardized test scores predict performance on ABIM-CE and ABP-CE for Med/Peds residents.

Methodology

We analyzed prior exam scores for 91/96 (95%) residents in a Med/Peds program from 2008 to 2017. Scores from the United States Medical Licensing Examination (USMLE) Steps 1 and 2 Clinical Knowledge (CK) and In-Training Exams in Internal Medicine (ITE-IM) and Pediatrics (ITE-P) were analyzed with the corresponding ABIM-CE and ABP-CE first-time scores. Linear and logistic regression were applied to predict board scores/passage.

Results

USMLE 1 and 2 CK, ITE-IM, and ITE-P scores had a linear relationship with both ABIM-CE and ABP-CE scores. In the linear regression, adjusted R² values showed low-to-moderate predictive ability (R² = 0.11-0.35), with the highest predictor of ABIM-CE and ABP-CE being USMLE Step 1 (0.35) and Postgraduate Year 1 (PGY-1) ITE-IM (0.33), respectively. Logistic regression showed odds ratios of passing board certifications ranging from 1.05 to 1.53 per point increase on the prior exam score. The PGY-3 ITE-IM was the best predictor of passing both certifying exams.

Conclusions

In one Med/Peds program, USMLE Steps 1 and 2 and all ITE-IM and ITE-P scores predicted certifying exam scores and passage. This provides Med/Peds-specific data to allow individualized resident counseling and guide programmatic improvements targeted to board performance.

Educational and Personal Opportunity Costs of Medical Student Preparation for the United States Medical Licensing Examination Step 1 Exam: A Single-Center Study

Purpose

To assess the degree to which medical students choose to disengage from their regular preclinical curriculum and extracurricular activities in order to focus on United States Medical Licensing Examination (USMLE) Step 1 exam preparation, as well as learner-perceived effects of Step 1 preparation on their physical, social, and mental health.

Method

Online survey of medical students who have taken the USMLE Step 1 exam at a single large Midwestern academic medical center.

Results

The response rate was 54%. Students often reported absenteeism from a variety of preclinical curricular activities, including lectures (44%) and didactics focusing on medical ethics (37%), clinical skills (28%), and encounters with actual and standardized patients (9%) in order to study for USMLE Step 1. Many students also forewent extracurricular opportunities including research (53%), elective patient care opportunities (45%), community service (39%), and healthcare advocacy experiences (38%) in order to study for USMLE Step 1. Majorities of students identified Step 1 preparation as a cause of burnout (79%) or significant anxiety or depression (61%), for which nearly a third sought mental healthcare; students also reported Step 1 preparation as a cause of engaging in dangerous behaviors such as illicit prescription stimulant use as well as driving or providing patient care while impaired by fatigue. In narrative comments, students frequently described Step 1 to be a barrier to their development into effective clinicians, the traditional medical school curriculum to be a barrier to performance on Step 1, or both.

Conclusions

Medical students often prioritize Step 1 exam preparation over engaging with the standard preclinical curriculum, extracurricular opportunities, and activities to promote wellbeing. These findings have implications for the emphasis residency program directors place on single high-stakes standardized exams in the resident recruitment process.

Preparation for the United States Medical Licensing Examinations in the Face of COVID-19

The COVID-19 pandemic has led to the canceling and rescheduling of the United States Medical Licensing Examination (USMLE) examinations due to the nationwide closure of the Prometric testing centers, which poses a significant challenge to medical students. The rescheduling of a high-stakes board examination leads to significant stress and potential burnout. Students may need guidance to decrease anxiety and reframe their study plan while maintaining their knowledge. Here, we combined board examination coaching tips with specific worked examples to describe how to prevent burnout, give recommendations for scheduling, and suggest practical approaches to USMLE and other high-stakes examinations.