The Relationship Between US Medical Licensing Examination Step Scores and ABR Core Examination Outcome and Performance: A Multi-institutional Study

Perspectives From the RadDiscord Annual Survey: Overview of the Top Study Tools and Evaluation of Study Time and Various Resources

RATIONALE AND OBJECTIVES

Each year, senior radiology residents take the American Board of Radiology Qualifying (Core) exam to evaluate competency. Approximately 10% of first-time examinees will fail this exam (1). Understanding factors that contribute to success will help residency program directors and trainees prepare for future exams. RadDiscord (www.raddiscord.org), an international radiology educational community, is in the unique position to evaluate different study materials and resources. The goal of this paper is to report the results from the RadDiscord survey and analyze the factors that correlate with higher exam performance and passing.

MATERIALS AND METHODS

Following the February 2021, June 2021, and June 2022 exams, RadDiscord members were provided an anonymous survey, collecting information on study resources and exam scores. The collected data were analyzed using various statistical methods. Both descriptive and inferential analyses were performed.

RESULTS

A total of 318 residents responded (95% passed). Significant variability in Qualifying (Core) exam performance and perceived quality of internal didactics existed between program types. Residents who did less than 2000 practice questions performed lower on the exam. The Diagnostic Radiology In-Training (DXIT) exam was the most predictive for passing and performance. Qualifying (Core) exam performance negatively correlated with study time, though certain residents did receive some benefit from study time.

CONCLUSION

Many factors correlate with passing and Qualifying (Core) exam performance. Residency programs with fewer resources should consider alternative ways to support residents beyond offering study time. Residents who complete at least 2000 practice questions are more likely to pass and DXIT results can be a useful gauge to identify exam readiness.

Resident factors associated with American board of internal medicine certification exam failure

INTRODUCTION

Performance on the certifying examinations such as the American Board of Internal Medicine Certification Exam (ABIM-CE) is of great interest to residents and their residency programs. Identification of factors associated with certification exam result may allow residency programs to recognize and intervene for residents at risk of failing. Despite this, residency programs have few evidence-based predictors of certification exam outcome. The change to pass-or-fail score reporting of the USA Medical Licensing Exam (USMLE) Step 1 removes one such predictor.

MATERIALS AND METHODS

We performed a retrospective study of residents from a medium-sized internal medicine residency program who graduated from 1998 through 2017. We used univariate tests of associations between ABIM-CE result and various demographic and scholastic factors.

RESULTS

Of 166 graduates, 14 (8.4%) failed the ABIM-CE on the first attempt. Failing the first attempt of the ABIM-CE was associated with older median age on entering residency (29 vs 27 years; P = 0.01); lower percentile rank on the Internal Medicine In-Training Examination (IM-ITE) in each of the first, second, and third years of training (P < 0.001 for all); and lower scores on the USMLE Steps 1, 2 Clinical Knowledge, and 3 (P < 0.05 for all). No association was seen between a variety of other scholastic or demographic factors and first-attempt ABIM-CE result.

DISCUSSION

Although USMLE step 1 has changed to a pass-or-fail reporting structure, there are still other characteristics that allow residency programs to identify residents at risk of ABIM-CE first time failure and who may benefit from intervention.

Unintended Consequence: Diversity as a Casualty of Eliminating United States Medical Licensing Examination Step 1 Scores

PURPOSE

The purpose of this study was to use a discrete-choice experiment to model the trade-offs evaluators make between academic attributes and demographics when the United States Medical Licensing Examination (USMLE) Step 1 switches to pass/fail.

METHODS

A discrete-choice experiment was administered to faculty members from a geographically diverse mix of 14 academic and community radiology departments in the United States from August through November 2020. Reviewers reviewed 10 applicant pairs with numeric Step 1 scores (part 1) and 10 applicant pairs with a pass Step 1 result (part 2). Applicant attributes included medical school rank, gender, race/ethnicity, USMLE Step 1 score, USMLE Step 2 score, class rank, clerkship honors, and publications. Conditional logistic regression modeled the influence of attribute levels.

RESULTS

Two hundred twelve evaluators completed the study (response rate 59%). The most influential attribute was Step 1 score in part 1 and medical school rank in part 2. The relative importance of race/ethnicity and gender decreased by 25% and 29%, respectively, when Step 1 switches to pass/fail. Evaluators weigh race/ethnicity the strongest when applicants have the same Step 1 score (preference weights of 0.85 for African American, 1.42 for Hispanic, and 0 for White and Asian applicants). Race/ethnicity is relatively more important when Step 1 scores are higher (preference weights of 1.58 for African American, 0.90 for Hispanic, and 0 for White and Asian applicants).

CONCLUSIONS

The loss of numeric Step 1 scores reduced the residency evaluator preference for diversity. Reviewers prioritize underrepresented-in-medicine applicants when Step 1 scores are higher and comparable with White and Asian applicants.

The Association Between USMLE Step 2 Clinical Knowledge Scores and Residency Performance: A Systematic Review and Meta-Analysis

PURPOSE

With the change in Step 1 score reporting, Step 2 Clinical Knowledge (CK) may become a pivotal factor in resident selection. This systematic review and meta-analysis seeks to synthesize existing observational studies that assess the relationship between Step 2 CK scores and measures of resident performance.

METHOD

The authors searched MEDLINE, Web of Science, and Scopus databases using terms related to Step 2 CK in 2021. Two researchers identified studies investigating the association between Step 2 CK and measures of resident performance and included studies if they contained a bivariate analysis examining Step 2 CK scores' association with an outcome of interest: in-training examination (ITE) scores, board certification examination scores, select Accreditation Council for Graduate Medical Education core competency assessments, overall resident performance evaluations, or other subjective measures of performance. For outcomes that were investigated by 3 or more studies, pooled effect sizes were estimated with random-effects models.

RESULTS

Among 1,355 potential studies, 68 met inclusion criteria and 43 were able to be pooled. There was a moderate positive correlation between Step 2 CK and ITE scores (0.52, 95% CI 0.45-0.59, P < .01). There was a moderate positive correlation between Step 2 CK and ITE scores for both nonsurgical (0.59, 95% CI 0.51-0.66, P < .01) and surgical specialties (0.41, 95% CI 0.33-0.48, P < .01). There was a very weak positive correlation between Step 2 CK scores and subjective measures of resident performance (0.19, 95% CI 0.13-0.25, P < .01).

CONCLUSIONS

This study found Step 2 CK scores have a statistically significant moderate positive association with future examination scores and a statistically significant weak positive correlation with subjective measures of resident performance. These findings are increasingly relevant as Step 2 CK scores will likely become more important in resident selection.

Attending Reviews of Residents Correlate with ABR Qualifying (Core) Examination Failure

RATIONALE AND OBJECTIVES

Since the American Board of Radiology (ABR) instituted the new system of board certification, there has been much discussion as to the test's validity. We decided to evaluate if subjective evaluation of resident performance correlated with ABR Qualifying (Core) Examination performance at this single institution.

MATERIALS AND METHODS

Data regarding resident evaluation scores by attending physicians and passage of board examinations was gathered regarding residents who had taken the ABR Qualifying (Core) Examination from 2013 through 2019 for a total of 42 residents, eight of whom failed the ABR Qualifying (Core) Examination on their first attempt. A univariate analysis comparing scores with resident passage or failure of the ABR Qualifying (Core) Examination on the first attempt and analyses correcting for class year only and class year and number of evaluations was performed.

RESULTS

The non-weighted average evaluation score of years 1, 2, and 3 was 80.24% for those who failed the ABR Qualifying (Core) Examination and 83.71 % for those who passed. On univariate analysis along with analyses correcting for class year only and class year along with number of evaluations, there was a statistically significant correlation with decreased evaluation scores averaged over the three years of residency and failure of the ABR Qualifying (Core) Examination (p = 0.0102, p = 0.003, and p = 0.0043). The statistical significance held for the average numerical score in each individual year of training in all analyses except for year 1 of the univariate analysis (p = 0.1264).

CONCLUSION

At the studied institution, there was a statistically significant correlation between lower subjective faculty evaluation scores and failure of the ABR Qualifying (Core) Examination.

Current Views on the New United States Medical Licensing Examination Step 1 Pass/Fail Format: A Review of the Literature

INTRODUCTION

Residency programs have historically used numerical Step 1 scores to screen applicants, making it a career-defining, high-stakes examination. Step 1 scores will be reported as pass/fail starting in January 2022, fundamentally reshaping the residency application review process. This review aimed to identify opinions of physicians and medical students about the new format, identify arguments in support of or against the change, and determine the implications of this change on the residency selection process.

METHODS

A comprehensive PubMed review was performed in May 2021 to identify articles that discussed the new Step 1 format. Non-English and duplicate articles were excluded. Data collected from each article included publication year, specialty, subjects, and key findings.

RESULTS

A total of 81 articles were included, 26 of which discussed the impact of the new format within surgical fields (32.1%). Remaining articles discussed the implications within the medical community as a whole (n = 33, 40.7%) and nonsurgical fields (n = 22, 27.2%). Studies suggest Program Directors will rely on Step 2 Clinical Knowledge (CK) scores, medical school reputation, applicant familiarity, Dean's letters, recommendation letters, and research in lieu of numerical Step 1 scores. In addition, concerns have been raised that the new format will disadvantage international, osteopathic, and minority applicants while increasing stress surrounding Step 2 CK.

CONCLUSIONS

Within the medical community, there are concerns that Step 2 CK will be used to substitute Step 1 and that resident diversity will diminish due to the new Step 1 format. Holistic candidate consideration will be increasingly important.

USMLE Step 3 Scores Have Value in Predicting ABR Core Examination Outcome and Performance: A Multi-institutional Study

RATIONALE AND OBJECTIVES

We analyzed multi-institutional data to determine if Step 3 performance tiers can identify radiology residents with increased risk of Core examination failure and submean performance.

MATERIALS AND METHODS

We collected Step 3 scores (USMLE Step 3 or COMLEX Level 3) and American Board of Radiology (ABR) Core examination outcomes and scores for anonymized residents from 13 different Diagnostic Radiology residency programs taking the ABR Core examination between 2013 and 2019. Step 3 scores were converted to percentiles based on Z-score, with Core outcome and performance analyzed for Step 3 groups based on 50th percentile and based on quintiles. Core outcome was scored as fail when conditionally passed or failed. Core performance was measured by the percent of residents with scores below the mean. Differences between Step 3 groups for Core outcome and Core performance were statistically evaluated.

RESULTS

Data were available for 342 residents. The Core examination failure rate for 121 residents with Step 3 scores <50th percentile was 19.8% (fail relative risk = 2.26), significantly higher than the 2.7% failure rate for the 221 other residents. Of 42 residents with Step 3 scores in the lowest quintile, the Core failure rate increased to 31.0% (fail relative risk = 3.52). Core performance improved with higher Step 3 quintiles.

CONCLUSION

Step 3 licensing scores have value in predicting radiology resident performance on the ABR Core examination, enabling residency programs to target higher risk residents for early assessment and intervention.

Educational and Personal Opportunity Costs of Medical Student Preparation for the United States Medical Licensing Examination Step 1 Exam: A Single-Center Study

Purpose

To assess the degree to which medical students choose to disengage from their regular preclinical curriculum and extracurricular activities in order to focus on United States Medical Licensing Examination (USMLE) Step 1 exam preparation, as well as learner-perceived effects of Step 1 preparation on their physical, social, and mental health.

Method

Online survey of medical students who have taken the USMLE Step 1 exam at a single large Midwestern academic medical center.

Results

The response rate was 54%. Students often reported absenteeism from a variety of preclinical curricular activities, including lectures (44%) and didactics focusing on medical ethics (37%), clinical skills (28%), and encounters with actual and standardized patients (9%) in order to study for USMLE Step 1. Many students also forewent extracurricular opportunities including research (53%), elective patient care opportunities (45%), community service (39%), and healthcare advocacy experiences (38%) in order to study for USMLE Step 1. Majorities of students identified Step 1 preparation as a cause of burnout (79%) or significant anxiety or depression (61%), for which nearly a third sought mental healthcare; students also reported Step 1 preparation as a cause of engaging in dangerous behaviors such as illicit prescription stimulant use as well as driving or providing patient care while impaired by fatigue. In narrative comments, students frequently described Step 1 to be a barrier to their development into effective clinicians, the traditional medical school curriculum to be a barrier to performance on Step 1, or both.

Conclusions

Medical students often prioritize Step 1 exam preparation over engaging with the standard preclinical curriculum, extracurricular opportunities, and activities to promote wellbeing. These findings have implications for the emphasis residency program directors place on single high-stakes standardized exams in the resident recruitment process.