The Value of Numerical USMLE Step 1 Scores in Radiology Resident Selection

Beyond milestones and percentiles: Revisiting non-cognitive and non-interpretive skills in radiology

While there is no precise formula for a great radiology resident, certain attributes and achievements may herald success during training. We briefly review prior works exploring predictive factors and evaluation metrics of top resident performance, noting that those focusing on non-cognitive attributes are over twenty years old. As radiology practice and education has substantially evolved in the interim, we revisit this topic from a contemporary perspective. Inspired by the literature and our own personal experiences, we suggest that the following non-cognitive traits are invaluable for radiology trainees: communication expertise, workplace adaptability, self-awareness, tech savvy and genuine interest in one's individual work and greater community. These characteristics should be highlighted by applicants, sought by selection committees, cultivated by mentors, evaluated by programs and valued by colleagues.

Unintended Consequence: Diversity as a Casualty of Eliminating United States Medical Licensing Examination Step 1 Scores

PURPOSE

The purpose of this study was to use a discrete-choice experiment to model the trade-offs evaluators make between academic attributes and demographics when the United States Medical Licensing Examination (USMLE) Step 1 switches to pass/fail.

METHODS

A discrete-choice experiment was administered to faculty members from a geographically diverse mix of 14 academic and community radiology departments in the United States from August through November 2020. Reviewers reviewed 10 applicant pairs with numeric Step 1 scores (part 1) and 10 applicant pairs with a pass Step 1 result (part 2). Applicant attributes included medical school rank, gender, race/ethnicity, USMLE Step 1 score, USMLE Step 2 score, class rank, clerkship honors, and publications. Conditional logistic regression modeled the influence of attribute levels.

RESULTS

Two hundred twelve evaluators completed the study (response rate 59%). The most influential attribute was Step 1 score in part 1 and medical school rank in part 2. The relative importance of race/ethnicity and gender decreased by 25% and 29%, respectively, when Step 1 switches to pass/fail. Evaluators weigh race/ethnicity the strongest when applicants have the same Step 1 score (preference weights of 0.85 for African American, 1.42 for Hispanic, and 0 for White and Asian applicants). Race/ethnicity is relatively more important when Step 1 scores are higher (preference weights of 1.58 for African American, 0.90 for Hispanic, and 0 for White and Asian applicants).

CONCLUSIONS

The loss of numeric Step 1 scores reduced the residency evaluator preference for diversity. Reviewers prioritize underrepresented-in-medicine applicants when Step 1 scores are higher and comparable with White and Asian applicants.

Athletes in medicine: A systematic review of performance of athletes in medicine

INTRODUCTION

As interest in medicine grows, admissions committees must review an increasingly competitive pool of medical school and residency candidates. Nearly all admissions committees have moved towards a holistic review, which considers an applicant's experiences and attributes in addition to academic metrics. As such, identifying nonacademic predictors of success in medicine is necessary. Parallels between skills necessary to succeed in athletics and in medicine have been drawn, including teamwork, discipline and resiliency. This systematic review synthesises the current literature to evaluate the relationship between participation in athletics and performance in medicine.

METHODS

The authors searched five databases to conduct a systematic review following PRISMA guidelines. Included studies assessed medical students, residents or attending physicians in the United States or Canada and used prior athletic participation as a predictor or explanatory variable. The review examined associations between prior athletic participation and outcomes in medical school, residency and/or as an attending physician.

RESULTS

Eighteen studies evaluating medical students (78%), residents (28%) or attending physicians (6%) met inclusion criteria for this systematic review. Twelve (67%) studies specifically assessed participants based on skill level, and five (28%) studies specifically assessed participants based on type of athletic participation (team versus individual). Sixteen studies (89%) found that former athletes performed significantly better than their counterparts (p < 0.05). These studies found significant associations between prior athletic participation and better outcomes in multiple performance indicators, including exam scores, faculty ratings, surgical errors and burnout.

CONCLUSIONS

Current literature, although limited, suggests that prior participation in athletics may be a predictor of success in medical school and residency. This was demonstrated through objective scoring methods, such as USMLE, and subjective outcomes, such as faculty ratings and burnout. Specifically, multiple studies indicate that former athletes demonstrated increased surgical skill proficiency and decreased burnout as medical students and residents.

The USMLE® STEP 1 Pass or Fail Era of the Vascular Surgery Residency Application Process: Implications for Structural Bias and Recommendations

MOTIVES BEHIND THE SHIFT

USMLE® STEP 1 score reporting has been changed to a binary pass/fail format since January 26, 2022. The motives behind this change were (1) the questionable validity of using USMLE® STEP 1 as a screening tool during the candidate selection process and (2) the negative impact of using standardized examination scores as an initial gatekeeping threshold for the underrepresented in medicine (URiM) candidates applying to GME programs, given their generally lower mean standardized exams scores compared to non-URiM students. The USMLE® administrators justified this change as a tactic to enhance the overall educational experience for all students and to increase the representation of URiM groups. Moreover, they advised the program directors (PDs) to give more attention to other important qualities and components such as the applicant's personality traits, leadership roles and other extracurricular accomplishments, as part of a holistic evaluation strategy. IMPACT OF USMLE® STEP 1 PASS/FAIL: At this early stage, it is unclear how this change will impact Vascular Surgery Integrated residency (VSIR) programs. Several questions are outstanding, most importantly, how VSIR PDs will evaluate applicants absent the variable which heretofore was the primary screening tool. Our previously published survey showed that VSIR PDs will move their attention to other measures such as USMLE® STEP 2CK and letters of recommendation during the VSIR selection process. Furthermore, more emphasis on subjective measures such as the applicant's medical school rank and extracurricular student activities is expected. Given the expected higher weight of USMLE® STEP 2CK in the selection process than ever, many anticipate that medical students will dedicate more of their limited time to its preparation at the expense of both clinical and non-clinical activities. Potentially leaving less time to explore specialty pathways and to determine whether VS is the appropriate career for them.

A FRAMEWORK FOR CANDIDATE EVALUATION

The critical juncture in the VSIR candidate evaluation paradigm presents an opportunity to thoughtfully transform the process via current (Standardized Letter of Recommendation, USMLE® STEP 2CK, and clinical research) and future (Emotional Intelligence, Structure Interview and Personality Assessment) measures which constitute a framework to follow in the USMLE® STEP 1 pass/fail era.

Current Views on the New United States Medical Licensing Examination Step 1 Pass/Fail Format: A Review of the Literature

INTRODUCTION

Residency programs have historically used numerical Step 1 scores to screen applicants, making it a career-defining, high-stakes examination. Step 1 scores will be reported as pass/fail starting in January 2022, fundamentally reshaping the residency application review process. This review aimed to identify opinions of physicians and medical students about the new format, identify arguments in support of or against the change, and determine the implications of this change on the residency selection process.

METHODS

A comprehensive PubMed review was performed in May 2021 to identify articles that discussed the new Step 1 format. Non-English and duplicate articles were excluded. Data collected from each article included publication year, specialty, subjects, and key findings.

RESULTS

A total of 81 articles were included, 26 of which discussed the impact of the new format within surgical fields (32.1%). Remaining articles discussed the implications within the medical community as a whole (n = 33, 40.7%) and nonsurgical fields (n = 22, 27.2%). Studies suggest Program Directors will rely on Step 2 Clinical Knowledge (CK) scores, medical school reputation, applicant familiarity, Dean's letters, recommendation letters, and research in lieu of numerical Step 1 scores. In addition, concerns have been raised that the new format will disadvantage international, osteopathic, and minority applicants while increasing stress surrounding Step 2 CK.

CONCLUSIONS

Within the medical community, there are concerns that Step 2 CK will be used to substitute Step 1 and that resident diversity will diminish due to the new Step 1 format. Holistic candidate consideration will be increasingly important.

Potential Impact of a Pass or Fail United States Medical Licensing Exam Step 1 Scoring System on Radiology Residency Applications

RATIONALE AND OBJECTIVES

To assess the anticipated impact of the change in United States Medical Licensing Examination Step 1 scoring from numerical to pass or fail on the future selection of radiology residents.

MATERIALS AND METHODS

An anonymous electronic 14-item survey was distributed to 308 members of the Association of Program Directors in Radiology and included questions regarding the anticipated importance of various application metrics when Step 1 becomes pass or fail. Secondary analyses compared responses based on the current use of a Step 1 scoring screen.

RESULTS

Eighty eight respondents (28.6% [88/308]) completed the survey. Most (64% [56/88]) noted that the United States Medical Licensing Examination Step 2 Clinical Knowledge (CK) score will likely be one of the top three most important factors in assessing applicants, followed by class ranking or quartile (51% [45/88]) and the Medical Student Performance Evaluation/Dean's Letter (42% [37/88]). Over 90% (82/88) of respondents anticipate potentially or definitively requiring Step 2 CK scores before application review, and 50% (44/88) of respondents anticipate extending interview invites at a later date to receive Step 2 CK scores. These results did not significantly differ between programs who currently use a Step 1 scoring screen and those who do not.

CONCLUSION

As Step 1 transitions from a numerical score to pass or fail, radiology residency program directors will likely rely on Step 2 CK scores as an objective and standardized metric to screen applicants. Further investigation is needed to identify other objective metrics to evaluate applicants before Step 1 becomes pass or fail.

Three Decades Later: A Scoping Review of the Literature Related to the United States Medical Licensing Examination

PURPOSE

To conduct a scoping review of the timing, scope, and purpose of literature related to the United States Medical Licensing Examination (USMLE) given the recent impetus to revise USMLE scoring.

METHOD

The authors searched PubMed, PsycInfo, and ERIC for relevant articles published from 1990 to 2019. Articles selected for review were labeled as research or commentaries and coded by USMLE Step level, sample characteristics (e.g., year(s), single/multiple institutions), how scores were used (e.g., predictor/outcome/descriptor), and purpose (e.g., clarification/justification/description).

RESULTS

Of the 741 articles meeting inclusion criteria, 636 were research and 105 were commentaries. Publication totals in the past 5 years exceeded those of the first 20 years.Step 1 was the sole focus of 38%, and included in 84%, of all publications. Approximately half of all research articles used scores as a predictor or outcome measure related to other curricular/assessment efforts, with a marked increase in the use of scores as predictors in the past 10 years. The overwhelming majority of studies were classified as descriptive in purpose.

CONCLUSIONS

Nearly 30 years after the inception of the USMLE, aspirations for its predictive utility are rising faster than evidence supporting the manner in which the scores are used. A closer look is warranted to systematically review and analyze the contexts and purposes for which USMLE scores can productively be used. Future research should explore cognitive and noncognitive factors that can be used in conjunction with constrained use of USMLE results to inform evaluation of medical students and schools and to support the residency selection process.

The Relationship Between US Medical Licensing Examination Step Scores and ABR Core Examination Outcome and Performance: A Multi-institutional Study

PURPOSE

We analyzed multi-institutional data to understand the relationship of US Medical Licensing Examination (USMLE) Step scores to ABR Core examination performance to identify Step score tiers that stratify radiology residents into different Core performance groups.

METHODS

We collected USMLE Step scores and ABR Core examination outcomes and scores for anonymized residents from 13 different diagnostic radiology residency programs taking the ABR Core examination between 2013 and 2019. USMLE scores were grouped into noniles using z scores and then aggregated into three tiers based on similar Core examination pass-or-fail outcomes. Core performance was grouped using standard deviation from the mean and then measured by the percent of residents with scores below the mean. Differences between Step tiers for Core outcome and Core performance were statistically evaluated (P < .05 considered significant).

RESULTS

Differences in Step 1 terciles Core failure rates (45.9%, 11.9%, and 3.0%, from lowest to highest Step tiers; n = 416) and below-mean Core performance (83.8%, 54.1%, and 21.1%, respectively; n = 402) were significant. Differences in Step 2 groups Core failure rates (30.0%, 10.6%, and 2.0%, from lowest to highest Step tiers; n = 387) and below-mean Core performance (80.0%, 43.7%, and 14.0%, respectively; n = 380) were significant. Step 2 results modified Core outcome and performance predictions for residents in Step 1 terciles of varying statistical significance.

CONCLUSIONS

Tiered scoring of USMLE Step results has value in predicting radiology resident performance on the ABR Core examination; effective stratification of radiology resident applicants can be done without reporting numerical Step scores.