The USMLE Step 1 Pass/Fail Reporting Proposal: The APDR Position

Evaluating the medical curriculum: Bias, problems, solutions

Medical school curriculums have increasingly shifted to an integrated curriculum and have been replacing lecture with 'flipped classroom' approaches. Analyses of the benefits of the integrated curriculum and flipped classroom model typically report enhanced student performance. However, the question is whether institutional self-evaluation of curricular success is biased to demonstrate success that may not objectively exist and/or whether such biased data are presented during Liaison Committee on Medical Education (LCME) site visits. An objective determination of curricular effectiveness requires an absence of bias and of efforts to put an institutional 'thumb on the scale' to obtain desired results. In addition, bias may exist in the rationale for implementing these curricular changes in the first place; these can include, for example, with respect to career advancement as well as ideological motivation. Thus, in this paper I examine potential problems with institutional bias with evaluation of curriculum and how to overcome these.

Does Masked Interviewing Encourage Holistic Review in Residency Selection? A Mixed-Methods Study

Problem: Medical educators increasingly champion holistic review. However, in U.S. residency selection, holistic review has been difficult to implement, hindered by a reliance on standardized academic criteria such as board scores. Masking faculty interviewers to applicants' academic files is a potential means of promoting holistic residency selection by increasing the interview's ability to make a discrete contribution to evaluation. However, little research has directly analyzed the effects of masking on how residency selection committees evaluate applicants. This mixed-methods study examined how masking interviews altered residency selection in an anesthesiology program at a large U.S. academic medical center. Intervention: During the 2019-2020 residency selection season in the University of Pennsylvania's Department of Anesthesiology & Critical Care, we masked interviewers to the major academic components of candidates' application files (board scores, transcripts, letters) on approximately half of interview days. The intent of the masking intervention was to mitigate the tendency of interviewers to form predispositions about candidates based on standardized academic criteria and thereby allow the interview to make a more independent contribution to candidate evaluation. Context: Our examination of the masking intervention used a concurrent, partially mixed, equal-status mixed-methods design guided by a pragmatist approach. We audio-recorded selection committee meetings and qualitatively analyzed them to explore how masking affected the process of candidate evaluation. We also collected independent candidate ratings from interviewers and consensus committee ratings and statistically compared ratings of candidates interviewed on masked days to ratings from conventional days. Impact: In conventional committee meetings, interviewers focused on how to reconcile academic metrics and interviews, and their evaluations of interviews were framed according to predispositions about candidates formed through perusal of application files. In masked meetings, members instead spent considerable effort evaluating candidates' "fit" and whether they came off as tactful. Masked interviewers gave halting opinions of candidates and sometimes pushed for committee leaders to reveal academic information, leading to masking breaches. Higher USMLE Step 1 score and higher medical school ranking were statistically associated with more favorable consensus rating. We found no significant differences in rating outcomes between masked and conventional interview days. Lessons learned: Elimination of academic metrics during the residency interview phase does not straightforwardly promote holistic review. While critical reflection among medical educators about the fairness and utility of such metrics has been productive, research and intervention should focus on the more proximate topic of how programs apply academic and other criteria to evaluate applicants.

Effect of United States Medical Licensing Examination Score Cutoffs on Recruitment of Underrepresented Applicants in the Urology Match

OBJECTIVE

To determine how the use of United States Medical Licensing Examination (USMLE) score cutoffs during the screening process of the Urology Residency Match Program may affect recruitment of applicants who are underrepresented in medicine (URM).

MATERIALS AND METHODS

Deidentified data from the Association of American Medical Colleges' (AAMC) Electronic Residency Application Service (ERAS) system was reviewed, representing all applicants to our institution's urology residency program from 2018 to 2022. We analyzed self-reported demographic variables including race/ethnicity, age, sex/gender, as well as USMLE Step 1 and Step 2 scores. Chi-square tests and ANOVA were used to determine the association between race/ethnicity and other sociodemographic factors and academic metrics. Applicants were stratified according to USMLE Step 1 cutoff scores and the distribution of applicants by race/ethnicity was assessed using a Gaussian nonlinear regression fit.

RESULTS

A total of 1258 applicants submitted applications to our program during the 5-year period, including 872 males (69.3%) and 386 females (30.7%). Most applicants were White (43.5%), followed by Asian (28.3%), Hispanic/Latino (11.7%), and Black (7.0%). There was an association between race/ethnicity and USMLE scores. Median USMLE Step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232, respectively (P < .001). As cutoff score increases, percentage of URM applicants decreases.

CONCLUSION

The use of cutoffs based on USMLE scores disproportionately affects URM applicants. Transitioning from numeric scores to pass/fail may enhance holistic review processes and increase the representation of URM applicants offered interviews at urology residency programs.

Underrepresented Applicants Post-USMLE Pass/Fail: A National Survey of Competitive Residency Directors

Objective

To report the selection criteria important to residency program directors (PDs) and whether they believe pass/fail scoring will impact underrepresented in medicine (URM), International Medical Graduate (IMG), or osteopathic (DO) residency applicants after the United States Medical Licensing Examination (USMLE) changed Step 1 score reporting to pass/fail in January 2022.

Methods

A Qualtrics survey was sent between August 2022 and January 2023 to 1141 US PDs from specialties with traditionally low residency selection rates: dermatology, ophthalmology, otolaryngology, orthopedic surgery, neurosurgery, interventional radiology, diagnostic radiology, radiation oncology, thoracic surgery, vascular surgery, plastic surgery, and urology. Contact information was obtained from AMA, FRIEDA, or program websites.

Results

We received 433 responses (38%). When asked to anticipate the difficulty student groups will face matching into their specialty, PDs reported: for URM, 24.0% increased, 46.0% unchanged, and 30.0% decreased; for DO, 49.19% increased, 44.58% unchanged, and 6.23% decreased and for IMG, 56.35% increased, 39.72% unchanged, and 3.93% decreased. When asked to rank the most important selection factors, the top two responses were Step 2 CK score and away rotation participation at their site.

Conclusion

PDs overwhelmingly believed residency selection difficulty would either increase or remain unchanged for DO (93.77%) and IMG (96.07%). In contrast, 76.0% reported difficulty for URM students would either decrease or remain unchanged. PDs ranked Step 2 CK score and away rotation participation as the most important selection factors. Despite PDs' belief that the Step 1 pass/fail scoring system may mitigate one barrier for URM students, emphasis on Step 2 CK and away rotations place additional barriers.

Predictors of Ophthalmology Resident Performance From Medical Student Application Materials

OBJECTIVE

To determine whether elements in ophthalmology residency applications are predictors of future resident performance.

DESIGN

This multi-institutional, cross-sectional, observational study retrospectively reviewed the residency application materials of ophthalmology residents who graduated from residency from 2006 through 2018. Resident performance was scored by 2 faculty reviewers in 4 domains (clinical, surgical, academic, and global performance). Correlation between specific elements of the residency application and resident performance was assessed by Spearman correlation coefficients (univariate) and linear regression (multivariate) for continuous variables and logistic regression (multivariate) for categorical variables.

SETTING

Seven ophthalmology residency programs in the US.

PARTICIPANTS

Ophthalmology residents who graduated from their residency program.

RESULTS

High-performing residents were a diverse group, in terms of sex, ethnicity, visa status, and educational background. Residents with United States Medical Licensing Examination Step 1 scores higher than the national average for that year had significantly higher scores in all 4 performance domains than those who scored at or below the mean (all domains P < 0.05). Residents who had honors in at least 4 core clerkships and who were members of Alpha Omega Alpha Medical Honor Society also had higher scores in all 4 performance domains (all domains P ≤ 0.04). Step 1 score (ρ=0.26, P < 0.001) and the difference between Step 1 score and the national average for that year (ρ=0.19, P = 0.009) positively correlated with total resident performance scores. Residents who passed the American Board of Ophthalmology Written Qualifying Examination or Oral Examination on their first attempt had significantly higher Step 1/2 scores (P ≤ 0.005), Ophthalmology Knowledge Assessment Program scores (P = 0.001), and resident performance scores (P ≤ 0.004).

CONCLUSIONS

In this new landscape of increasing numbers of applicants to residency programs and changing of the Step 1 score to pass/fail, our findings may help guide selection committees as they holistically review applicants to select exceptional future residents in ophthalmology.

Rapidly Increasing Number and Cost of Residency Applications in Surgery

BACKGROUND

Surgical specialties represent the most competitive residency positions to obtain, and applicants for these spots may apply to increasingly more programs in an attempt to match. We describe trends in residency applications across all surgical specialties from 2017 to 2021 application cycles.

MATERIALS AND METHODS

This review of the 2017, 2018, 2019, 2020, and 2021 surgical residency application cycles used the American Association of Medical Colleges' Electronic Residency Application Service (ERAS) databases. Data from a total of 72,171 applicants to United States surgical residencies during the study's time period were included. The 2021 ERAS fee schedule was used to calculate the cost of applications.

RESULTS

The number of applicants across the study interval remains unchanged. Trends suggest more women and underrepresented minorities in medicine are applying to surgical residencies today compared to 5 years ago. The mean number of applications per applicant increased 32.0% from 39.3 in 2017 to 51.8 in 2021, resulting in an increased application fee cost of $329 per applicant. Mean total cost for application fees alone in 2021 were $1211 per applicant. In 2021, the cost of applying to surgical residency for all applicants was over $26 million, a nearly $8 million increase from 2017.

CONCLUSIONS

There has been an increase in the number of applications per applicant in the past 5 residency application cycles. Increased applications create barriers and burdens for applicants, and residency program personnel. These rapid increases are unsustainable and require intervention, although a viable solution has yet to be elucidated.

International Medical Graduates in Radiology Residencies: Demographics, Performance, and Visa Issues

RATIONALE AND OBJECTIVES

The number of international medical graduates (IMG) in radiology residencies has varied from year to year even as the number of candidates continues to grow. It is unclear from which countries the IMGs are arriving and what visas are being used to accommodate them.

MATERIALS AND METHODS

We sent a survey to 195 program directors (PD) in diagnostic radiology (DR) inquiring about the number and nationality of IMG residents in their program, their attitudes about IMG candidates, the performance of their IMG trainees, and the visas that are offered.

RESULTS

We received responses from 121 of 195 (62.1%) DR programs (121/149 =81.2% of actionable emails). 80/121 (66.1%) had at least one IMG in their DR residency program and the countries of origin included India (36), Iran (30), Saudi Arabia (24), Egypt (16), Canada (14), Brazil (14), and Pakistan (9), as the most common. While most programs (76/104, 73.1%) offered J1 visas, 23/99 (23.2%) provided H-1B visas to trainees. IMG DR residents overall performed as well as American graduates, with an equal number of PDs saying IMGs performed better and worse than American graduates. PDs' issues with IMGs centered on visas: (1) expense, (2) lack of familiarity, (3) Educational Commission for Foreign Medical Graduates regulations, and (4) time commitment in submitting paperwork.

CONCLUSION

Most radiology IMG residents originate from India and Middle Eastern countries. Once enrolled, IMG residents perform similarly to US graduates. However, adding IMG candidates to the training program requires overcoming bureaucratic and monetary hurdles around visas.

SUMMARY SENTENCE

Most DR international medical graduate residents are from India or the Middle East. Although their performance is the same as American graduates in general, PDs note the monetary and bureaucratic hassles accompanying their recruitment.

Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School

Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.

The Upcoming Pass/Fail USMLE Step 1 Score Reporting: An Impact Assessment From Medical School Deans

Background: The US Medical Licensing Examination (USMLE) Step 1 change to pass/fail has been met with mixed reviews, and the impact on medical student education and residency match is unknown. We surveyed medical school student affairs deans regarding their thoughts on the upcoming transition of Step 1 to pass/fail. Methods: A questionnaire was emailed to medical school deans. Deans were asked to rank the importance of the following after the Step 1 reporting change: Step 2 Clinical Knowledge (Step 2 CK), clerkship grades, letters of recommendation, personal statement, medical school reputation, class rank, Medical Student Performance Evaluation, and research. They were asked how the score change will affect curriculum, learning, diversity, and student mental health. Deans were asked to select 5 specialties they thought would be most affected. Results: Regarding perceived importance of residency applications following the scoring change, the most frequent number 1 choice was Step 2 CK. The majority of deans (93.5%, n = 43) felt that the change to pass/fail would benefit medical student education/learning environment; however, most (68.2%, n = 30) did not believe their school curriculum would change. Students applying to dermatology, neurosurgery, orthopedic surgery, ENT, and plastic surgery were felt to be most affected by the scoring change; 58.7% (n = 27) felt it would not adequately address future diversity. Conclusion: The majority of deans feel the USMLE Step 1 change to pass/fail would benefit medical student education. Deans feel that students applying to traditionally more competitive specialties (ie, programs with fewer overall residency positions available) will be most affected.

The USMLE® STEP 1 Pass or Fail Era of the Vascular Surgery Residency Application Process: Implications for Structural Bias and Recommendations

MOTIVES BEHIND THE SHIFT

USMLE® STEP 1 score reporting has been changed to a binary pass/fail format since January 26, 2022. The motives behind this change were (1) the questionable validity of using USMLE® STEP 1 as a screening tool during the candidate selection process and (2) the negative impact of using standardized examination scores as an initial gatekeeping threshold for the underrepresented in medicine (URiM) candidates applying to GME programs, given their generally lower mean standardized exams scores compared to non-URiM students. The USMLE® administrators justified this change as a tactic to enhance the overall educational experience for all students and to increase the representation of URiM groups. Moreover, they advised the program directors (PDs) to give more attention to other important qualities and components such as the applicant's personality traits, leadership roles and other extracurricular accomplishments, as part of a holistic evaluation strategy. IMPACT OF USMLE® STEP 1 PASS/FAIL: At this early stage, it is unclear how this change will impact Vascular Surgery Integrated residency (VSIR) programs. Several questions are outstanding, most importantly, how VSIR PDs will evaluate applicants absent the variable which heretofore was the primary screening tool. Our previously published survey showed that VSIR PDs will move their attention to other measures such as USMLE® STEP 2CK and letters of recommendation during the VSIR selection process. Furthermore, more emphasis on subjective measures such as the applicant's medical school rank and extracurricular student activities is expected. Given the expected higher weight of USMLE® STEP 2CK in the selection process than ever, many anticipate that medical students will dedicate more of their limited time to its preparation at the expense of both clinical and non-clinical activities. Potentially leaving less time to explore specialty pathways and to determine whether VS is the appropriate career for them.

A FRAMEWORK FOR CANDIDATE EVALUATION

The critical juncture in the VSIR candidate evaluation paradigm presents an opportunity to thoughtfully transform the process via current (Standardized Letter of Recommendation, USMLE® STEP 2CK, and clinical research) and future (Emotional Intelligence, Structure Interview and Personality Assessment) measures which constitute a framework to follow in the USMLE® STEP 1 pass/fail era.

The US Residency Selection Process After the United States Medical Licensing Examination Step 1 Pass/Fail Change: Overview for Applicants and Educators

The United States Medical Licensing Examination (USMLE) Step 1, arguably the most significant assessment in the USMLE examination series, changed from a 3-digit score to a pass/fail outcome in January 2022. Given the rapidly evolving body of literature on this subject, this paper aims to provide a comprehensive review of the historical context and impact of this change on various stakeholders involved in residency selection. For this, relevant keyword-based searches were performed in PubMed, Google Scholar, and Scopus to identify relevant literature. Given the unique history of USMLE Step 1 in the US residency selection process and the score's correlation with future performance in board-certifying examinations in different specialties, this scoring change is predicted to significantly impact US Doctor of Medicine students, US Doctor of Osteopathic Medicine students, international medical graduates, and residency program directors, among others. The significance and the rationale of the pass/fail change along with the implications for both residency applicants and educators are also summarized in this paper. Although medical programs, academic institutions, and residency organizing bodies across the United States have swiftly stepped up to ensure a seamless transition and have attempted to ensure equity for all, the conversion process carries considerable uncertainty for residency applicants. For educators, the increasing number of applications conflicts with holistic application screening, leading to the expected greater use of objective measures, with USMLE Step 2 Clinical Knowledge likely becoming the preferred screening tool in lieu of Step 1.

The next step: Replacing step 1 as a metric for residency application

Objective

As of January 2022, USMLE Step 1 scores are reported as pass/fail. Historically, Step 1 scores have been a critical component of residency applications, representing one of the few metrics standardized across all applicants independent of the school they attended. In competitive specialties, such as otolaryngology, programs routinely get 100+ applicants for each residency spot and use Step 1 as a screening tool. This study seeks to assess quantifiable metrics in the application that highly competitive residency programs could use for screening in place of Step 1 scores.

Methods

Otolaryngology applications to an academic medical center for the 2019-20 and 2020-21 ERAS cycles were reviewed. Board scores and quantitative research data were extracted. The relationships between Step 1 score and the other metrics were examined by computing Pearson's correlation coefficients and building regression models. Similar analyses were done separately for three different score tiers defined by Step 1 cutoffs at 220 points and 250 points.

Results

Step 2 score was the only variable that had meaningful correlation with Step 1 score (R = .67, p < 2.2e-16). No other objective metric such as journal articles, posters, or oral presentations correlated with Step 1 scores.

Conclusion

Step 1 scores were moderately correlated with Step 2 scores; however, using a Step 2 cutoff instead of a Step 1 cutoff would identify a different cohort of applicants for interview. No other quantifiable application metric had a positive correlation. In future match cycles, highly competitive residency programs will need to adopt new methods to screen candidates.Level of Evidence: Level 3.

Pre-clerkship National Board of Medical Examiners Subject Examinations Versus End-of-Semester Final Examinations: How Well Do They Assess Preparedness for the United States Medical Licensing Examination Step 1?

Background The ability to provide performance insights of various United States Medical Licensing Examination (USMLE) Step 1 assessments is of great importance to medical educators. Two custom pre-clerkship assessments used at the Kirk Kerkorian School of Medicine at the University of Nevada, Las Vegas (KSOM) are National Board of Medical Examiners (NBME)-derived end-of-semester final examinations and subject examinations. The authors sought to determine if performance on these custom assessments can provide feedback on a medical student's readiness to undertake the USMLE Step 1 examination. Methodology Deidentified student performance data were provided by institutional databases for the KSOM graduating class of 2023 (N = 60). Pearson correlation analyses were utilized to evaluate the strength of the correlation between USMLE Step 1 performance and NBME subject examinations versus NBME end-of-semester final examinations. Results The results indicated that the NBME end-of-semester final examinations have a statistically higher correlation to the USMLE Step 1 score than the majority of the individual NBME subject examinations. However, the mean NBME subject examination score (Semester 1: r = 0.53, p < 0.05; Semester 2: r = 0.58, p < 0.05) demonstrated significantly higher correlation to the USMLE Step 1 performance than the NBME end-of-semester final examination score for both Semesters 1 and 2 (Semester 1: r = 0.50, p < 0.05; Semester 2: r = 0.48, p < 0.05). Conclusions These results showed that the mean of the NBME subject examination score was a better metric to assess readiness for the USMLE Step 1 than the NBME end-of-semester final examinations. However, each NBME end-of-semester final examination score showed a better correlation than the majority of the NBME subject examinations.

Predicting Academic Performance during Plastic Surgery Residency: Can Step 2 Scores Reliably Replace Step 1?

OBJECTIVE

Step 1 will transition to a pass/fail system in 2022. This study aimed to characterize the effects of this change on integrated plastic surgery program directors' selection criteria and assess whether Step 2 Clinical Knowledge (CK) can replace Step 1 as an application selection metric.

DESIGN

Online survey that was administered to a collaborative group of ten plastic surgery program directors collecting USMLE Step 1, Step 2 CK, In-Service, and written board scores for 3 years of graduated integrated residents.

SETTING

Ten academic integrated plastic surgery programs.

PARTICIPANTS

Data from 80 graduated integrated plastic surgery residents.

RESULTS

Across 80 included integrated residents, mean (SD) Step 1 score was 247 (13), Step 2 CK was 249 (13), PGY1-6 In-Service percentiles varied from 45 to 53 percentile, and written board pass rate was 98.3%. Both Step 1 and Step 2 CK correlated highly with In-Service percentiles (both p < 0.001), with Step 2 CK scores correlating similarly with In-Service performance compared to Step 1 (rho 0.359 vs. 0.355, respectively). Across applicant characteristics, program directors reported the highest relative increase in Step 2 CK importance after Step 1 transitions to pass/fail.

CONCLUSIONS

Step 2 CK correlates similarly with plastic surgery In-Service performance compared to Step 1. While Step scores do not necessarily correlate with residency performance, Step 2 CK may also be used as an application screening metric for programs seeking objective data to differentiate plastic surgery applicants.

Current Views on the New United States Medical Licensing Examination Step 1 Pass/Fail Format: A Review of the Literature

INTRODUCTION

Residency programs have historically used numerical Step 1 scores to screen applicants, making it a career-defining, high-stakes examination. Step 1 scores will be reported as pass/fail starting in January 2022, fundamentally reshaping the residency application review process. This review aimed to identify opinions of physicians and medical students about the new format, identify arguments in support of or against the change, and determine the implications of this change on the residency selection process.

METHODS

A comprehensive PubMed review was performed in May 2021 to identify articles that discussed the new Step 1 format. Non-English and duplicate articles were excluded. Data collected from each article included publication year, specialty, subjects, and key findings.

RESULTS

A total of 81 articles were included, 26 of which discussed the impact of the new format within surgical fields (32.1%). Remaining articles discussed the implications within the medical community as a whole (n = 33, 40.7%) and nonsurgical fields (n = 22, 27.2%). Studies suggest Program Directors will rely on Step 2 Clinical Knowledge (CK) scores, medical school reputation, applicant familiarity, Dean's letters, recommendation letters, and research in lieu of numerical Step 1 scores. In addition, concerns have been raised that the new format will disadvantage international, osteopathic, and minority applicants while increasing stress surrounding Step 2 CK.

CONCLUSIONS

Within the medical community, there are concerns that Step 2 CK will be used to substitute Step 1 and that resident diversity will diminish due to the new Step 1 format. Holistic candidate consideration will be increasingly important.

Perceptions of USMLE Step 1 Pass/Fail Score Reporting Among Orthopedic Surgery Residency Program Directors

The United States Medical Licensing Examination (USMLE) Step 1 examination will transition from graded to pass/fail scoring starting no earlier than January 2022. Orthopedic surgery residency programs will need to adapt to these changes. The goal of this study was to investigate the perceptions of orthopedic surgery residency program directors on the change of Step 1 from a graded to a pass/fail examination. We also investigated how the change would affect the other factors that are typically considered in the selection of orthopedic surgery residents. A survey was distributed to 161 directors of allopathic orthopedic surgery programs. Contact information was obtained from a national database. Of those contacted, 75 (46.6%) program directors responded. Most (85.3%) did not support the pass/fail change. Most believe that greater importance will be placed on the Step 2 Clinical Knowledge examination (96.0%), audition elective with their department (84.0%), personal knowledge of the applicant (78.7%), grades (74.7%), letters of recommendation from recognizable orthopedic surgeons (74.7%), and Alpha Omega Alpha status (69.3%). Most also believe that this change will advantage allopathic students who attend highly regarded schools (58.7%). Most of the program directors support a graded preclinical curriculum (69.3%) and caps on the number of orthopedic surgery residency applications (70.7%). Although most orthopedic surgery program directors disagree with the change to a pass/fail Step 1 examination, residency programs will need to reevaluate how they screen applicants for an interview once the scored Step 1 is no longer available. With this change, other factors, such as Step 2 score, audition rotations, and grades in clerkships, will be emphasized more heavily. [Orthopedics. 2022;45(1):e30-e34.].

Innovation in Resident Selection: Life Without Step 1

INTRODUCTION

The announcement of Step 1 shifting to a Pass/Fail metric has prompted resident selection committees (RSCs) to pursue objective methods of evaluating prospective residents. Regardless of the program's specialty or affiliated hospital/school, RSCs universally aim to recognize and choose applicants who are an "optimal fit" to their programs.¹ An optimal fit can be defined as a candidate who thrives in the clinical and academic setting, both contributing to and benefiting from their respective training environments.

OBJECTIVE

The objective of this scoping review is to evaluate alternative, innovative methods by which RSCs can evaluate applicants and predict success during residency. Objective methods include: Step 2 scores, Traditionally Used Metrics (core clerkship scores), interview performance, musical talent, sports involvement, AOA membership, research publications, unprofessional behavior, Dean's letters, Rank list, judgement testing, and specialty-specific shelf exams.^13-15.

METHODS

A scoping review was performed in compliance with the guidelines indicated by the PRISMA Protocol for scoping review.¹⁸ 9308 results were identified in the original PubMed search for articles with the key words "Resident Success". Abstract screening and application of inclusion and exclusion criteria yielded 97 articles that were critically appraised via review of full manuscript.

RESULTS

Of the articles that focused on personality traits, situational judgement testing, and specialty specific pre-assessment, all of them demonstrated some level of predictability for resident success. Standardized Letter of Recommendations, Traditionally Used Metrics, and STEP 2 did not show a unanimous consensus in demonstrating predictability of a resident's success, this is because some articles suggested predictability and some articles disputed predictability.

CONCLUSION

The authors found personality traits, situational judgement testing, and specialty specific assessments to be predictive in selecting successful residents. Further research should aim to analyze exactly how RSCs utilize these assessment tools to aid in screening their large and competitive applicant pools to find residents that will be successful in their program.

Potential Impact of a Pass or Fail United States Medical Licensing Exam Step 1 Scoring System on Radiology Residency Applications

RATIONALE AND OBJECTIVES

To assess the anticipated impact of the change in United States Medical Licensing Examination Step 1 scoring from numerical to pass or fail on the future selection of radiology residents.

MATERIALS AND METHODS

An anonymous electronic 14-item survey was distributed to 308 members of the Association of Program Directors in Radiology and included questions regarding the anticipated importance of various application metrics when Step 1 becomes pass or fail. Secondary analyses compared responses based on the current use of a Step 1 scoring screen.

RESULTS

Eighty eight respondents (28.6% [88/308]) completed the survey. Most (64% [56/88]) noted that the United States Medical Licensing Examination Step 2 Clinical Knowledge (CK) score will likely be one of the top three most important factors in assessing applicants, followed by class ranking or quartile (51% [45/88]) and the Medical Student Performance Evaluation/Dean's Letter (42% [37/88]). Over 90% (82/88) of respondents anticipate potentially or definitively requiring Step 2 CK scores before application review, and 50% (44/88) of respondents anticipate extending interview invites at a later date to receive Step 2 CK scores. These results did not significantly differ between programs who currently use a Step 1 scoring screen and those who do not.

CONCLUSION

As Step 1 transitions from a numerical score to pass or fail, radiology residency program directors will likely rely on Step 2 CK scores as an objective and standardized metric to screen applicants. Further investigation is needed to identify other objective metrics to evaluate applicants before Step 1 becomes pass or fail.

Analyzing the Landscape of the 2021 Virtual Match: A Nationwide Survey of Radiology Programs' Stakeholders

RATIONALE AND OBJECTIVES

We aimed to provide insights into the adaptive strategies, benefits, and challenges faced by the radiology programs during the 2021 residency virtual Match. Furthermore, we explored the potential impacts of related topics, such as diversity and social media use on the Match process and outcomes.

MATERIALS AND METHODS

A cross-sectional survey of 31 questions was designed and distributed via e-mails to individuals involved radiology programs match process during the 2021 Match. Descriptive statistics were used to analyze the results of most questions. Two questions comparing the changes in factors influencing the selection of applicants on a Likert scale of 1-5 were analyzed using paired t-test and Wilcoxon signed-rank test where p-value <0.05 was considered statistically significant.

RESULTS

Responses from 125 participants were analyzed. The following factors carried less weight in evaluating applicants during 2021 Match: away rotations (p < 0.01), no failed attempts in USMLE Step 1/CK (p < 0.01), grades in radiology clerkship (p < 0.04), and class rank/quartile (p < 0.04), while personal statements were more important (p < 0.03). Out of the 125 respondents, 80 (64%) and 58 (47%) strongly or somewhat agree on the effectiveness of virtual interviews in gauging applicants' candidacy and showing their programs' advantages, respectively. Advantages of virtual interviews included decreased cost, time flexibility, less faculty burden, and an increased number of offered interviews according to 81% (101/125), 46% (58/125), 40%, (50/125), and 34% (43/125), respectively. The most helpful platforms that showcased program advantages were program websites followed by Twitter and Instagram.

CONCLUSION

Most radiology programs were able to adjust to the virtual interview process, and the majority agree on their effectiveness citing many benefits. However, there were mixed opinions if it could be sustained in future cycles.

Pass/Fail USMLE Step 1 Scoring-A Radiology Program Director Survey

Rationale and Objectives

In February 2020, administrators for the US medical licensing examination (USMLE) announced that Step 1 reporting would change to pass/fail in hopes of reducing the overemphasis of USMLE performance on the residency selection system and improving medical student well-being. Our objective was to determine the perspectives of diagnostic radiology (DR), interventional radiology (IR), and nuclear medicine (NM) program directors (PDs) regarding pass/fail USMLE Step 1 scoring.

Materials and Methods

A survey composed of thirteen questions on a three-point Likert scale, five demographic questions, and a free-text question was distributed to 179 DR, 84 IR, and 34 NM PDs from ACGME-accredited residency programs.

Results

In total, 140 unique responses were obtained (response rate = 47.1%). The PD respondents had a male predominance of 79.1%, average age of 46 ± 7.2 years, and average tenure of 5.9 ± 5.2 years. A majority of PDs (69.6%) disagreed that the change is a good idea, and a minority (21.6%) believe the change will improve medical student well-being. Further, 90.7% of PDs believe a pass/fail format will make it more difficult to objectively compare applicants and most will place more emphasis on USMLE Step 2 scores and medical school reputation (89.3% and 72.7%, respectively).

Conclusion

The lasting impact of pass/fail Step 1 scoring are uncertain and many radiology PDs do not support this change. While the central motivations to reduce the overemphasis on USMLE Step 1 performance and improve medical student well-being are admirable, it remains to be seen if pass/fail scoring will accomplish these goals.

Mastering the Match: Recruiting a Successful Surgery Resident

PURPOSE OF REVIEW

The objective of general surgery residency is to produce competent surgeons. At a minimum this requires being procedurally and clinically capable and able to pass the board exams. Recruitment is designed to select those residents who can successfully do so. But there is more to being a successful resident than that. In this review, we attempt to define a "successful resident" and how to attract them to your program.

RECENT FINDINGS

Resident applicants are still most concerned with matching to a program that will prepare them for a surgery career. Though there is variation of importance for different applicants, resident life, comradery, and relationships with faculty or mentors do factor into residency ranking. The program website remains the most utilized resource for applicants. However, social media (SM) has an increasing role in applicants' evaluation of a program. SM and the preinterview gathering seem to expose the subjective aspects of a program most effectively. Additional assessments evaluating personality, grit or career goals may assist in screening applicants for good "fit."

SUMMARY

In order to recruit successful residents, it is necessary to determine which applicant attributes are important to the program. Additionally, a program must maintain an updated website with clearly delineated resident expectations and program strengths. The screening and interview process must be maximized to target residents with career goals complimentary to available program opportunities. If SM is utilized, post should be frequent with relevant information pertaining to both resident life and educational or clinical opportunities.

Surgical clerkship: Do examination scores correlate with clinical performance?

BACKGROUND

This study aims to determine if there are correlations between clinical performance and objective grading parameters for medical students in the third-year surgery clerkship.

METHODS

Clerkship grades were compiled from 2016 to 2020. Performance on clinical rotations, NBME shelf exam, oral exam, and weekly quizzes were reviewed. Students were divided into quartiles (Q1-Q4) based on clinical performance. Standard statistical analysis was performed.

RESULTS

There were 625 students included in the study. Students in Q1+Q2 were more likely than those in Q3+Q4 to score in the top quartile on the shelf exam (29% vs. 19%, p = 0.002), oral exam (24% vs. 17%, p = 0.032), and quizzes (22% vs. 15%, p = 0.024). However, there was negligible correlation between clinical performance and performance on objective measures: shelf exam (R² = 0.027, p < 0.001), oral exam (R² = 0.021, p < 0.001), and weekly quizzes (R² = 0.053, p = 0.092).

CONCLUSIONS

Clinical performance does not correlate with objective grading parameters for medical students in the third-year surgery clerkship.

Applicant Familiarity Becomes Most Important Evaluation Factor in USMLE Step I Conversion to Pass/Fail: A Survey of Plastic Surgery Program Directors

BACKGROUND

In 2020, Step 1 of the United States Medical Licensing Examination (USMLE) changed to a pass/fail reporting. Step 1 has been one of the main factors for both inviting applicants for interviews and for ranking in Plastic Surgery. Due to this change, we hypothesize that Step 2 CK - currently the only remaining, universal quantitative metric - will become the main factor in the residency selection process.

METHODS

A survey-based cross-sectional study of United States (US) integrated plastic surgery program directors (PSPDs) investigated the factors that would assume importance following the change in the reporting pattern.

RESULTS

Respondents reported that personal prior knowledge of the applicant, Letters of recommendation (LORs), Step 2 CK scores, and away rotation at the institution of interest would become the most important factors (median ratings of 5, 4.5, 4.5, 4.5, respectively on a 5-point Likert scale). Eighty-three percent of respondents were strongly dissatisfied with the conversion to pass/fail reporting. LOR's received the highest ranking (median,1; IQR,1-2) as the component used for offering away rotations after the implementation of the pass/fail reporting, followed by the applicant's medical school (median, 3; IQR, 3-4), and grades obtained during medical school (median,3; IQR,1.75-4). Standardized assessment during rotations are recommended by 67% of PSPDs.

CONCLUSIONS

Future emphasis will be placed primarily on subjective metrics, including applicant familiarity. Step 2 CK, LORs, and away rotation at the institution of interest are other factors of importance. PSPDs welcome the adoption of objective assessments of patient care and medical knowledge to improve the current selection process.

The Single Match: Reflections on the National Resident Matching Program's Sustained Partnership With Learners

In 2020, the National Resident Matching Program (NRMP) sponsored the inaugural "Single Match"-the first time that seniors and graduates of U.S. MD-granting and DO-granting schools participated in one Match. In honor of the Single Match milestone, the authors examine the NRMP's history, reflecting on the organization's efforts since the 1950s to support learners and the graduate medical education community by fostering a responsive, robust matching program while remaining true to its founding principles to provide parity of experience for applicants and reduce coercive practices. The chaos and stress associated with the pre-Match days in the 1920s and 1930s that led to the call for a national clearinghouse are highlighted as are significant NRMP accomplishments, from the organization's incorporation as a 501(c)(3) organization in 1953 as a simple internship placement system through the first Single Match. Recognizing that the current transition to residency is not without its stressors, the authors note that the NRMP remains committed and willing to continue to evolve and identify innovative and meaningful ways to address learner needs and improve the transition to residency.

Selecting the Next Generation of Surgeons: General Surgery Program Directors and Coordinators Perspective on USMLE Changes and Holistic Approach

Introduction

The United States Medical Licensing Examination (USMLE) was designed as a universal assessment tool for states to determine physician’s medical licensure's candidacy. Recent changes in the USMLE exam have changed the way future surgical residency candidate applications will be reviewed. The survey aimed to assess the effect of changes in USMLE exams—USMLE Step 1 pass/fail, complete dissolution of USMLE clinical skills exam, and the role of holistic review in future surgical residency candidacy selection.

Methods

An anonymous online survey was created and distributed to general surgery program directors and coordinators across the USA. The survey aimed to assess attitudes toward changes to USMLE exams and the potential changes with a holistic review of candidate applications.

Results

The response rate was 63.7%. Most program directors and coordinators disagree with changing USMLE Step 1 to a pass/fail scoring system. The majority felt that contacts, the medical school's name, and performance in clinical electives and sub-internships would hold more significance. They also believe that a holistic review of application will decrease socioeconomic discrepancies and promote a more diverse and inclusive resident cohort.

Conclusion

Step 2 clinical knowledge (CK) will gain more importance in future residency matches because of the change in the scoring system of Step 1. The medical school's name, personal contacts, and clinical performance in rotations will hold more significance.

Factors Influential in the Selection of Radiology Residents in the Post-Step 1 World: A Discrete Choice Experiment

OBJECTIVES

Reporting of United States Medical Licensing Examination Step 1 results will transition from a numerical score to a pass or fail result. We sought an objective analysis to determine changes in the relative importance of resident application attributes when numerical Step 1 results are replaced.

METHODS

A discrete choice experiment was designed to model radiology resident selection and determine the relative weights of various application factors when paired with a numerical or pass or fail Step 1 result. Faculty involved in resident selection at 14 US radiology programs chose between hypothetical pairs of applicant profiles between August and November 2020. A conditional logistic regression model assessed the relative weights of the attributes, and odds ratios (ORs) were calculated.

RESULTS

There were 212 participants. When a numerical Step 1 score was provided, the most influential attributes were medical school (OR: 2.35, 95% confidence interval [CI]: 2.07-2.67), Black or Hispanic race or ethnicity (OR: 2.04, 95% CI: 1.79-2.38), and Step 1 score (OR: 1.8, 95% CI: 1.69-1.95). When Step 1 was reported as pass, the applicant's medical school grew in influence (OR: 2.78, 95% CI: 2.42-3.18), and there was a significant increase in influence of Step 2 scores (OR: 1.31, 95% CI: 1.23-1.40 versus OR 1.57, 95% CI: 1.46-1.69). There was little change in the relative influence of race or ethnicity, gender, class rank, or clerkship honors.

DISCUSSION

When Step 1 reporting transitions to pass or fail, medical school prestige gains outsized influence and Step 2 scores partly fill the gap left by Step 1 examination as a single metric of decisive importance in application decisions.

Radiology resident selection factors predict resident performance

PURPOSE

To determine selection factors that predict radiology resident performance.

METHODS

59 consecutive radiology residents from 2002 to 2015 were ranked on performance during residency. Correlations and multiple regression analyses were performed to predict resident performance from the following selection factors: United States Medical Licensing Exam (USMLE) Step 1 score, medical school rank, Alpha Omega Alpha (AOA) membership, honors in clinical rotations, Medical Student Performance Evaluation (MSPE), and interview score. Results were compared against predictions from Match rank position.

RESULTS

Five selection factors showed significant or marginally significant correlations with resident performance (r = 0.2 to 0.3). The interview score was not significantly correlated. A multiple regression model comprised of the USMLE Step 1 score, medical school rank, AOA membership, and interview score predicted resident performance, with an adjusted R² of 0.19. The interview score was included in the model but did not achieve statistical significance. Match rank did not predict resident performance, with an R² of 0.01.

CONCLUSIONS

A multiple regression model comprised of the USMLE Step 1 score, medical school rank, and AOA membership predicted radiology resident performance and may assist with resident selection.

USMLE Step 1 Is Pass/Fail - Should the ABSITE Follow Suit?

The United States Medical Licensing Examination (USMLE) Step 1 was recently made pass/fail. This decision was controversial largely because of the reliance on USMLE Step 1 scores in resident selection. However, these scores do not correlate with resident ability. In this manuscript, we consider if the American Board of Surgery In-Training Examination (ABSITE) should be pass/fail as well. The ABSITE has been used for "high-stakes" purposes, such as preliminary resident advancement and prospective fellow evaluation, for which it was not intended. Moreover, similar to the USMLE Step 1 exam, ABSITE scores have demonstrated no correlation with clinical ability. A pass/fail ABSITE would return the exam to its original purpose and minimize an over-reliance on scores. Moving forward, new objective measures will need to be developed to assess surgical trainees in a more holistic manner.

Internal medicine residency program director perceptions of USMLE Step 1 pass/fail scoring: A cross-sectional survey

The United States Medical Licensing Examination Step 1 will transition to a pass/fail exam starting no earlier than January 2022. Internal medicine residency programs will need to adapt to these changes. The purpose of this study was to investigate: 1. internal medicine residency program directors’ perceptions on the change of Step 1 to a pass/fail exam, and 2. the impact on other factors considered for internal medicine residency selection. A validated REDCap survey was sent to 548 program directors at active Accreditation Council for Graduate Medical Education internal medicine residency programs. Contact information from the American Medical Association's Fellowship and Residency Electronic Interactive Database was used. The survey had 123 respondents (22.4%). Most internal medicine program directors do not support the pass/fail change. A greater importance will be placed on Step 2 Clinical Knowledge exam, personal knowledge of the applicant, clerkship grades, and audition electives. Allopathic students from less highly regarded medical schools, as well as osteopathic and international students, will be disadvantaged. About half believe that schools should adopt a graded pre-clinical curriculum (51.2%) and that there should be residency application caps (54.5%). Internal medicine program directors mostly disagree with the pass/fail Step 1 transition. Residency programs will need to reevaluate how applicants are evaluated. Other factors, such as Step 2 Clinical Knowledge score, personal knowledge of the applicant, grades in clerkships, and audition rotations will now be emphasized more heavily.

Medical Student Attitudes toward USMLE Step 1 and Health Systems Science - A Multi-Institutional Survey

Phenomenon: Because of its importance in residency selection, the United States Medical Licensing Examination Step 1 occupies a critical position in medical education, stimulating national debate about appropriate score use, equitable selection criteria, and the goals of undergraduate medical education. Yet, student perspectives on these issues and their implications for engagement with health systems science-related curricular content are relatively underexplored. Approach: We conducted an online survey of medical students at 19 American allopathic medical schools from March-July, 2019. Survey items were designed to elicit student opinions on the Step 1 examination and the impact of the examination on their engagement with new, non-test curricular content related to health systems science. Findings: A total of 2856 students participated in the survey, representing 23.5% of those invited. While 87% of students agreed that doing well on the Step 1 exam was their top priority, 56% disagreed that studying for Step 1 had a positive impact on engagement in the medical school curriculum. Eighty-two percent of students disagreed that Step 1 scores should be the top item residency programs use to offer interviews. When asked whether Step 1 results should be reported pass/fail with no numeric score, 55% of students agreed, while 33% disagreed. The majority of medical students agreed that health systems science topics were important but disagreed that studying for Step 1 helped learn this content. Students reported being more motivated to study a topic if it was on the exam, part of a course grade, prioritized by residency program directors, or if it would make them a better physician in the future. Insights: These results confirm the primacy of the United States Medical Licensing Examination Step 1 exam in preclinical medical education and demonstrate the need to balance the objectives of medical licensure and residency selection with the goals of the broader medical profession. The survey responses suggest several potential solutions to increase student engagement in health systems science curricula which may be especially important after Step 1 examination results are reported as pass/fail.

Admissions Is Not Enough: The Racial Achievement Gap in Medical Education

The achievement gap is a disparity in academic and standardized test performance that exists between White and underrepresented minority (URM) students that begins as early as preschool and worsens as students progress through the educational system. Medical education is not immune to this inequality. URM medical students are more likely to experience delayed graduation and course failure, even after accounting for science grade point average and Medical College Admission Test performance. Moreover, URM students are more likely to earn lower scores on licensing examinations, which can have a significant impact on their career trajectory, including specialty choice and residency competitiveness. After the release of preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS) and public commentary on these recommendations, the National Board of Medical Examiners and Federation of State Medical Boards announced that the United States Medical Licensing Examination (USMLE) Step 1 would transition from a 3-digit numeric score to pass/fail scoring. Given that another of InCUS's recommendations was to "minimize racial demographic differences that exist in USMLE performance," it is paramount to consider the impact of this scoring change on URM medical students specifically. Holistic admissions are a step in the right direction of acknowledging that URM students often travel a further distance to reach medical school. However, when residency programs emphasize USMLE performance (or any standardized test score) despite persistent test score gaps, medical education contributes to the disproportionate harm URM students face and bolsters segregation across medical specialties. This Perspective provides a brief explanation of the achievement gap, its psychological consequences, and its consequences in medical education; discusses the potential effect of the Step 1 scoring change on URM medical students; and provides a review of strategies to redress this disparity.

Red Flags, Geography, Exam Scores, and Other Factors Used by Program Directors in Determining Which Applicants Are Offered an Interview for Anesthesiology Residency

Objective

The goal of this study was to measure the most important factors in candidate applications that anesthesiology program directors (PDs) use to decide who to invite for an interview, and how that might change once the United States Medical Licensing Examination (USMLE) Step 1 is only reported as pass/fail.

Design

Based on a literature review, a comprehensive list of 27 factors used by PDs to select candidates for the interview was developed. An anonymous survey link was emailed to PDs of all Accreditation Council for Graduate Medical Education (ACGME) accredited Anesthesiology residencies. The survey asked PDs to rank order the top 10 factors they currently consider for making interview invitation, and then to repeat the rank ordering as if the USMLE Step 1 score was instead reported as pass/fail as will be done beginning in 2022.

Results

Forty-five of 159 (28%) PDs responded, with 82% disagreeing with changing the Step 1 score to pass/fail. 84% consider the Step 1 score (77% for Step 2) moderately or very important for selecting an applicant for an interview. The most frequently mentioned “red flags” were failure of a licensing exam, failure of a medical school course, gaps in education without explanation, and criminal history. 69% of PDs agreed that applicants coming from the medical school affiliated with their program would have an advantage over other applicants. Although, the three factors most commonly ranked in the top 10 in importance were the Step 1 score, followed by letters of recommendation, and then the Medical School Performance Evaluation, variability exists in how PDs ranked factors. For example, of the PDs that had Step 1 in the top 10, 27% had it ranked between the 6th and 10th most important. 9% of PDs did not have Step 1 score in the top 10. Core clinical clerkship grades were one of the top 5 factors by 49% of PDs, yet overall was the 6th most common top 10 factor as 36% of PDs did not have core clerkship grades at all in the top 10. Once Step 1 is reported only as pass/fail, PDs had letters of recommendation, Step 2, and the Medical School Performance Evaluation as the most frequently ranked factors in the top 10. 64% of the PDs supported restricting the number of programs a candidate can apply to, with the majority suggesting a limit of 15 to 20 programs per applicant.

Conclusion

Variability exists among anesthesiology PDs in the key criteria for offering an applicant an interview. Once Step 1 is reported as pass/fail, there will be an increased emphasis on Step 2 scores.

The New USMLE Step 1 Paradigm: An Opportunity to Cultivate Diversity of Excellence

The February 2020 announcement that United States Medical Licensing Examination (USMLE) Step 1 results will be reported as pass/fail instead of numerical scores has been controversial. Step 1 scores have played a key role in residency selection, including screening for interviews. Although Step 1 scores are viewed as an objective criterion, they have been shown to disadvantage female and underrepresented minority applicants, cause student anxiety and financial burden, and affect student well-being. Furthermore, Step 1 scores incompletely predict applicants' overall residency performance. With this paradigm shift in Step 1 score reporting, residency programs will have fewer objective, standardized metrics for selection decisions, which may lead to greater emphasis on USMLE Step 2 Clinical Knowledge scores or yield unintended consequences, including shifting weight to metrics such as medical school reputation.Yet, greater breadth in residency selection metrics will better serve both applicants and programs. Some students excel in coursework, others in research or leadership. All factors should be recognized, and broader metrics should be implemented to promote and recognize diversity of excellence. Given the need for metrics for residency selection as well as for a more holistic approach to evaluating residency applicants, assessment during medical school should be revisited and made more meaningful. Another opportunity may involve use of situational judgment tests to predict professionalism and performance on other competencies. It will be important to evaluate the impact of the new Step 1 paradigm and related initiatives going forward. Residency application overload must also be addressed.

How curricular changes influence medical students' perceptions of basic science: A pilot study

Theory

The perceived value of study material may have implications on learning and long-term retention. This study compares the perceived value of basic science of medical students from schools with a traditional “2+2” curriculum and the USMLE Step 1 placed before core clerkships to those from medical schools that have undergone curricular revisions, resulting in shortened pre-clerkship curricula and administration of the USMLE Step 1 after core clerkships.

Hypothesis

We hypothesize that differences in curricula, particularly duration of pre-clerkship curriculum and timing of the USMLE Step 1, affect medical students’ perceived value of basic science.

Methods

A twenty item anonymous questionnaire using a 5-point Likert scale was developed to assess medical students’ perceptions of basic science. The questionnaire was distributed to third-year medical students across four medical schools. Generalized linear models and p-values were calculated comparing the perceived value and use of basic science between medical schools with the USMLE Step 1 before clerkships and 2-years of basic science (BC) and medical schools with the USMLE Step 1 after core clerkships and 1.5-years of basic science (AC).

Results

The questionnaire was distributed to 695 eligible students and completed by 287 students. Students at BC schools tended to view basic science as more essential for clinical practice than students at AC schools across both outcomes (rating independence of basic science and clinical practice, AC school mean = 2.97, BC school mean = 2.73, p = 0.0017; rating importance of basic science to clinical practice, AC school mean = 3.30, BC schools mean = 3.50, p = 0.0135).

Conclusions

Our study suggests that students who have a longer basic science curriculum tend to value basic science greater than students with a shorter basic science curriculum. The timing of the USMLE Step 1 may also influence this relationship. Curricular decisions, such as reductions in pre-clerkship curricula and administration of the USMLE Step 1 after clerkships, may impact medical students’ perceptions of the value of basic science to clinical practice. This can have implications on their future engagement with basic science and should be considered when modifying curriculum.

Beyond High-Stakes Testing: Learner Trust, Educational Commodification, and the Loss of Medical School Professionalism

With ever-growing emphasis on high-stakes testing in medical education, such as the Medical College Admission Test and the United States Medical Licensing Examination Step 1, there has been a recent surge of concerns on the rise of a "Step 1 climate" within U.S. medical schools. The authors propose an alternative source of the "climate problem" in current institutions of medical education. Drawing on the intertwined concepts of trust and professionalism as organizational constructs, the authors propose that the core problem is not hijacking-by-exam but rather a hijackable learning environment weakened by a pernicious and under-recognized tide of commodification within the U.S. medical education system. The authors discuss several factors contributing to this weakening of medicine's control over its learning environments, including erosion of trust in medical school curricula as adequate preparation for entry into the profession, increasing reliance on external profit-driven sources of medical education, and the emergence of an internal medical education marketplace. They call attention to breaches in the core tenets of a profession-namely a logic that differentiates its work from market and managerial forces, along with related slippages in discretionary decision making. The authors suggest reducing reliance on external performance metrics (high-stakes exams and corporate rankings), identifying and investing in alternative metrics that matter, abandoning the marketization of medical education "products," and attending to the language of educational praxis and its potential corruption by market and managerial lexicons. These steps might salvage some self-governing independence implied in the term "profession" and make possible (if not probable) a recovery of a public trust becoming of the term and its training institutions.

The impact of United States Medical Licensing Exam (USMLE) step 1 cutoff scores on recruitment of underrepresented minorities in medicine: A retrospective cross-sectional study

BACKGROUND AND AIMS

United States Medical Licensing Exam (USMLE) scores are the single, most objective criteria for admission into residency programs in the country. Underrepresented minorities in medicine (URiM) are found to have lower USMLE scores compared to their White counterparts. The objective of this study is to examine how USMLE step 1 cutoff scores may exclude self-reported URiM from the residency interview process across various specialties.

METHODS

This was a retrospective cross-sectional study of 10 541 applicants to different residency programs at Zucker School of Medicine at Hofstra/Northwell Health between May 2014 and May 2015. We identified Blacks and Hispanics as URiM. The primary outcome is the percentage of applicants with USMLE step 1 score above different ranges of cutoff score, from 205 to 235 in five-point increments, by race/ethnicity and by URiM status. Secondary outcome is percentages of URiM vs non-URiM above and below mean USMLE step 1 scores by different specialties (internal medicine, obstetrics/gynecology, pediatrics, and psychiatry).

RESULTS

The study sample included 2707 White, 722 Black, 805 Hispanic, 5006 Asian, and 562 Other Race/Ethnicity applicants. Overall, 50.2% were male, 21.3% URiM, 7.4% had limited English proficiency, 67.6% attended international medical schools, and 2.4% are Alpha Omega Alpha Honor Medical Society (AOA) members. The mean (±SD) USMLE step 1 score was significantly greater among non-URiM applicants as compared to URiM applicants (223.7 ± 19.4 vs 216.1 ± 18.4, P < .01, two-sample t-test). Non-URiM applicants were younger, and the percentage of male and AOA applicants was greater among non-URiM applicants as compared to URiM applicants (50.5% vs 47.7%, P = .02, Chi-Square test; 2.9% vs 1.2%, P < .01, Chi-Square test, respectively).

CONCLUSION

Using a USMLE step 1 cutoff score as an initial filter for applicant recruitment and selection could jeopardize the benefits of a diverse residency program. Practical implications are discussed.

The Relationship Between US Medical Licensing Examination Step Scores and ABR Core Examination Outcome and Performance: A Multi-institutional Study

PURPOSE

We analyzed multi-institutional data to understand the relationship of US Medical Licensing Examination (USMLE) Step scores to ABR Core examination performance to identify Step score tiers that stratify radiology residents into different Core performance groups.

METHODS

We collected USMLE Step scores and ABR Core examination outcomes and scores for anonymized residents from 13 different diagnostic radiology residency programs taking the ABR Core examination between 2013 and 2019. USMLE scores were grouped into noniles using z scores and then aggregated into three tiers based on similar Core examination pass-or-fail outcomes. Core performance was grouped using standard deviation from the mean and then measured by the percent of residents with scores below the mean. Differences between Step tiers for Core outcome and Core performance were statistically evaluated (P < .05 considered significant).

RESULTS

Differences in Step 1 terciles Core failure rates (45.9%, 11.9%, and 3.0%, from lowest to highest Step tiers; n = 416) and below-mean Core performance (83.8%, 54.1%, and 21.1%, respectively; n = 402) were significant. Differences in Step 2 groups Core failure rates (30.0%, 10.6%, and 2.0%, from lowest to highest Step tiers; n = 387) and below-mean Core performance (80.0%, 43.7%, and 14.0%, respectively; n = 380) were significant. Step 2 results modified Core outcome and performance predictions for residents in Step 1 terciles of varying statistical significance.

CONCLUSIONS

Tiered scoring of USMLE Step results has value in predicting radiology resident performance on the ABR Core examination; effective stratification of radiology resident applicants can be done without reporting numerical Step scores.

The USMLE Step 1 Pass/Fail Reporting Proposal: Another View

The Association of Program Directors in Radiology recently issued a statement endorsing continued reporting of results of the United States Medical Licensing Examination (USMLE) as a three-digit score. While this position was approved by the Association of Program Directors in Radiology Board of Directors, it does not reflect the opinions of all radiology program directors. Here, we present an argument in support of reporting USMLE results as pass/fail. As a psychometric instrument, the USMLE Step 1 is designed to assess basic science knowledge and intended to inform a binary decision on licensure. Due to a steadily-increasing burden of applications to review, program directors have increasingly relied upon scores for candidate screening. Such use has multiple adverse consequences. Student focus on Step 1 systematically devalues educational content not evaluated on the exam, and the reliance on Step 1 scores almost certainly works against efforts to increase workforce diversity. Moreover, the increasing pressure of "Step 1 Mania" has negative consequences for trainee mental health and wellness. Despite the widespread use of Step 1 scores to select applicants, there are little data to correlate scores to meaningful outcomes related to patient care or clinical practice. We find the current situation untenable, and believe a necessary first step toward reform is making Step 1 a pass/fail only examination.