Challenges Facing Program Directors in the Urology Match

Effect of United States Medical Licensing Examination Score Cutoffs on Recruitment of Underrepresented Applicants in the Urology Match

OBJECTIVE

To determine how the use of United States Medical Licensing Examination (USMLE) score cutoffs during the screening process of the Urology Residency Match Program may affect recruitment of applicants who are underrepresented in medicine (URM).

MATERIALS AND METHODS

Deidentified data from the Association of American Medical Colleges' (AAMC) Electronic Residency Application Service (ERAS) system was reviewed, representing all applicants to our institution's urology residency program from 2018 to 2022. We analyzed self-reported demographic variables including race/ethnicity, age, sex/gender, as well as USMLE Step 1 and Step 2 scores. Chi-square tests and ANOVA were used to determine the association between race/ethnicity and other sociodemographic factors and academic metrics. Applicants were stratified according to USMLE Step 1 cutoff scores and the distribution of applicants by race/ethnicity was assessed using a Gaussian nonlinear regression fit.

RESULTS

A total of 1258 applicants submitted applications to our program during the 5-year period, including 872 males (69.3%) and 386 females (30.7%). Most applicants were White (43.5%), followed by Asian (28.3%), Hispanic/Latino (11.7%), and Black (7.0%). There was an association between race/ethnicity and USMLE scores. Median USMLE Step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232, respectively (P < .001). As cutoff score increases, percentage of URM applicants decreases.

CONCLUSION

The use of cutoffs based on USMLE scores disproportionately affects URM applicants. Transitioning from numeric scores to pass/fail may enhance holistic review processes and increase the representation of URM applicants offered interviews at urology residency programs.

Effect of Holistic Review, Interview Blinding, and Structured Questions in Resident Selection: Can we Predict Who Will Do Well in a Residency Interview?

OBJECTIVE

To examine the Urology residency application process, particularly the interview. Historically, the residency interview has been vulnerable to bias and not determined to be a predictor of future residency performance. Our goal is to determine the relationship between pre-interview metrics and post-interview ranking using best practices for Urology resident selection including holistic review, blinded interviews, and structured behaviorally anchored questions.

METHODS

Applications were assessed on cognitive (Alpha Omega Alpha, class rank, junior year clinical clerkship grades) and non-cognitive attributes (letters of recommendation [LOR], personal statement [PS], demographics, research, personal characteristics) by reviewers blinded to USMLE scores and photograph. Interviewers were blinded to the application other than PS and LORs. Interviews consisted of a structured behaviorally anchored question (SBI) and an unstructured interview (UI). Odds ratios were determined comparing pre-interview and interview impressions.

RESULTS

Fifty-one applicants were included in the analysis. USMLE step 1 score (average 245) was associated with Alpha Omega Alpha, class rank, junior year clinical clerkship, and PS. The UI score was associated with the LOR (P = .04) whereas SBI scores were not (P = .5). Faculty rank was associated with SBI, UI, and overall interview (OI) scores (P < .001). Faculty rank was also associated with LOR. Resident impression of interviewees were associated with faculty interview scores (P = .001) and faculty rank (P < .001).

CONCLUSION

Traditional interviews may be biased toward application materials and may be balanced with behavioral questions. While Step 1 score does not offer additional information over other PI metrics, blinded interviews may offer discriminant validity over a PI rubric.

Characterizing Standardized Letters of Recommendation in Urology Residency Applications

OBJECTIVE

To identify the current formats of standardized letters of recommendation (SLORs) and evaluate their characteristics, the distribution of applicants' ratings, correlation between SLOR domain ratings and conventional application metrics, and potential biases.

METHODS

We evaluated all applications submitted to our residency program for the 2020-2021 urology match. Two main formats of SLOR were identified. We extracted application characteristics and SLOR domain ratings.

RESULTS

Ninety SLORs from 82 applicants were reviewed. Applicants were highly rated among top tiers in both formats. Some correlations were observed between domain ratings and application metrics such as Step 1 and Step 2 Clinical Knowledge scores, and percentage of Honors in core clinical clerkships. No statistically significant differences were found between female and male applicants in terms of domain ratings. Alpha Omega Alpha members received higher ratings in "urology resident potential," "academic urologist potential," and "performance as a sub-intern" domains. Applicants from top 40 US medical schools performed better as sub-interns, and were more likely to be ranked higher. Letters from home institutions were associated with higher ratings in several domains. In-person vs virtual interactions received similar ratings except for "communication".

CONCLUSION

While it is promising to observe such number of SLORs submitted for the first time in urology, the current formats could benefit from further refinement in their structures and domains to distinguish between highly qualified urology applicants more efficiently. Given the transition in Step 1 score reporting to pass/fail outcome, the need for a reliable urology-specific SLOR will be critical.

Systems-Level Reforms to the US Resident Selection Process: A Scoping Review

BACKGROUND

Calls to reform the US resident selection process are growing, given increasing competition and inefficiencies of the current system. Though numerous reforms have been proposed, they have not been comprehensively cataloged.

OBJECTIVE

This scoping review was conducted to characterize and categorize literature proposing systems-level reforms to the resident selection process.

METHODS

Following Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches of Embase, MEDLINE, Scopus, and Web of Science databases were performed for references published from January 2005 to February 2020. Articles were included if they proposed reforms that were applicable or generalizable to all applicants, medical schools, or residency programs. An inductive approach to qualitative content analysis was used to generate codes and higher-order categories.

RESULTS

Of 10 407 unique references screened, 116 met our inclusion criteria. Qualitative analysis generated 34 codes that were grouped into 14 categories according to the broad stages of resident selection: application submission, application review, interviews, and the Match. The most commonly proposed reforms were implementation of an application cap (n = 28), creation of a standardized program database (n = 21), utilization of standardized letters of evaluation (n = 20), and pre-interview screening (n = 13).

CONCLUSIONS

This scoping review collated and categorized proposed reforms to the resident selection process, developing a common language and framework to facilitate national conversations and change.

Evaluating Urology Residency Applications: What Matters Most and What Comes Next?

PURPOSE OF REVIEW

In light of the announcement that the United States Medical Licensing Examination Step 1 exam will transition to pass/fail reporting, we reviewed recent literature on evaluating residency applicants with a focus on identifying objective measurements of applicant potential.

RECENT FINDINGS

References from attending urologists, Step 1 scores, overall academic performance, and research publications are among the most important criteria used to assess applicants. There has been a substantial increase in the average number of applications submitted per applicant, with both applicants and residency directors indicating support for a cap on the number of applications that may be submitted. Additionally, there are increasing efforts to promote diversity with the goal of improving care and representation in urology. Despite progress in standardizing interview protocols, inappropriate questioning remains an issue. Opportunities to improve residency application include promoting diversity, enforcing prohibitions of illegal practices, limiting application numbers, and finding more transparent and equitable screening measures to replace Step 1.