An Examination of Standardized Letters of Recommendation Rating Scales Among Neurosurgical Residency Candidates During the 2020-2021 Application Cycle

"Home Field Advantage": The Presence of a "Home" Neurosurgical Residency Training Program is Associated With Improved Match Outcomes

BACKGROUND AND OBJECTIVES

Competition for neurosurgical residency training positions remains fierce. The support provided by applicants' home neurosurgery residency training programs (NRTP) is suspected to play a key role in the National Resident Matching Program (NRMP) process. We sought to evaluate the impact of the presence of an Accreditation Council for Graduate Medical Education-accredited NRTP at medical students' home institutions has on NRMP match outcomes.

METHODS

Our cross-sectional observational study examined all US allopathic senior student Electronic Residency Application Service applications to a single NRTP from 2016 to 2022.

RESULTS

We analyzed a total of 1650 Electronic Residency Application Service applications to a single NRTP, of which 1432 (86.8%) were from schools with an Accreditation Council for Graduate Medical Education-accredited NRTP (NRTP+) and 218 (13.2%) were from schools without a residency (NRTP-). NRTP+ applicants matched a higher rate on both pooled analysis (80.8% vs 71.6%, P = .002) and paired analysis ( P = .02) over the seven-cycle study period. This difference was present before (82.4% vs 73.9%, P = .01) and after (77.2% vs 65.6%, P = .046) the COVID-19 pandemic. Cohorts were overall similar; however, NRTP+ applicants had more publication experiences (19.6 ± 19.0 vs 13.1 ± 10.2, P < .001) and were more likely to complete a research gap year (RGY) (25.8% vs 17.0%, P = .004). Completing a RGY was associated with an increased likelihood of matching for NRTP+ applicants but not for NRTP- applicants: NRTP+: 84.9% vs 78.1% ( P = .0056); NRTP-: 70.3% vs 70.9% ( P = .94).

CONCLUSION

The presence of a NRTP at a medical student's home institution is associated with improved NRMP match outcomes. This held true both before and after the COVID-19 pandemic. Applicants from schools with a NRTP had more publication experiences and were more likely to complete a RGY. Completion of a RGY is associated with an increased likelihood of matching only for students with an affiliated NRTP.

Navigating the Neurosurgery Match Process: Insights from the National Resident Matching Program - Program Director Surveys

OBJECTIVE

Neurosurgery is one of the most competitive specialties, and navigating the match process is often challenging for aspiring applicants. Here, we analyze insights from the National Resident Matching Program Director Surveys, illustrating evolving trends in applicant selection for interviews and for the ranking process, and providing a comparison with other specialties.

METHODS

We evaluated 7 surveys administered from 2012 to 2022. Six biennial surveys reported on factors influencing interview and ranking processes, while all 7 surveys included data about the program director (PD)'s attitude toward United States Medical Licensing Examination (USMLE) test scores.

RESULTS

The response rate of PDs decreased over the years. The most cited factor for interviews included specialty-specific recommendation letters (95%), USMLE Step 1 scores (91%), and interest in research (78%). A recent decline in emphasis on USMLE Step 1 scores coincided with a growing reliance on USMLE Step 2 scores. Award in basic science held significant esteem to a subset of programs. Personal characteristics dominated for ranking, with faculty interaction (89%), interpersonal skills (89%), and house staff interaction (85%) being the most important. Yet, PDs reported a difficulty in assessing interpersonal skills through virtual interviews.

CONCLUSIONS

Our analysis revealed the pervasive importance of specialized endorsements and academic achievements when screening applicants for the interview process. A shift in emphasis toward the USMLE Step 2 became apparent. Personal characteristics, on the other hand, seemed crucial to make a match and rank high among the pool of interviewed applicants. We uncovered difficulties in assessing these characteristics through virtual interviews.

How Prominent Are Gender Bias, Racial Bias, and Score Inflation in Orthopaedic Surgery Residency Recommendation Letters? A Systematic Review

INTRODUCTION

Letters of recommendation are considered an important factor in the holistic ranking of orthopaedic residency applications. The standardized letter of recommendation (SLOR) was introduced in 2017 in response to the inherent bias and limited comparative value of traditional LORs. It includes standardized questions about an applicant's orthopaedic qualifications, designed to enable fair comparisons among candidates. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies on the SLOR and narrative letter of recommendation.

QUESTION/PURPOSES

In this systematic review we asked: (1) What is the distribution of applicant SLOR rating among nine domains and summative scores? (2) Are there applicant characteristics associated with SLOR ratings? (3) Is there gender and racial bias in the SLOR and narrative letter of recommendation?

METHODS

PubMed, EBSCO, and Google Scholar electronic databases were queried on March 20, 2023, to identify all studies evaluating SLOR and narrative letter of recommendations between January 1, 2017, and March 20, 2023. Articles that evaluated orthopaedic SLOR or narrative LORs were included. Systematic reviews, case reports, duplicate studies between databases, and grey literature such as abstracts and articles on preprint servers were excluded. Eight studies including 6179 applicants and 18,987 letters of recommendation were included in the final analysis. The applicant classes ranged from years 2014 to 2020. Two reviewers independently evaluated the quality of the included studies using the Joanna Briggs Institute (JBI) tool for cross-sectional studies. The mean JBI score of included studies was 7.4 out of a maximum of 8, with higher scores indicating better quality. The primary outcome was to determine the distribution of applicant SLOR rankings in their summative score. Summative scores were separated into ranked to match, top one-third, middle one-third, lower one-third, and not a fit. Four studies reported the summative SLOR scores of applicants. Our secondary study goal was to assess applicant characteristics associated with SLOR summative scores and assess any bias present in the SLOR or narrative recommendation. Five studies compared SLOR ratings across applicant characteristics including gender, race, USMLE Step 1 score, USMLE Step 2 score, Alpha Omega Alpha (AΩA) membership, clerkship grades, and publications. Gender and racial bias were also assessed across five studies. Four studies utilized a linguistic analysis software to search letters of recommendation for differences in word category use by gender and race.

RESULTS

Studies consistently found that a higher percentage of candidates were identified in the top percentile blocks than is mathematically possible. For example, the two studies that tallied the proportion of candidates ranking in the top 10% of applicants found that 36% and 42% were rated as being in the top 10%. Similarly, articles found more than 87% of applicants scored in the top one-third. Studies had mixed results, but in general, they suggested that AΩA status, higher Step 1 scores, and more research publications have a slightly positive association with increased SLOR scores. We found no evidence of gender bias against women, and in fact, most studies evaluating word choices found word choices for women candidates generally were stronger. Similarly, no consistent disparities were identified in word categories utilized in SLORs based on applicant race.

CONCLUSION

Most notably, a mathematically impossible proportion of applicants were counted in the top percentiles in letters of recommendation. This compromises readers' abilities to differentiate candidates. Factors like AOA status and research publications displayed a modest positive correlation with SLOR scores. Gender bias against women or candidates from racial minority groups was not evident.

CLINICAL RELEVANCE

We suggest that group SLOR authorship, with a consensus assessment by clerkship or residency directors, should be considered. We also recommend the incorporation of mean and SD scores for each letter writer (or group) alongside their letters. Furthermore, studies indicate that submitting all four SLOR letters can substantially improve an applicant's chances of success. We advise students to choose strategically and submit the maximum allowable number of SLORs.

Measuring and Predicting Faculty Consensus Rankings of Standardized Letters of Evaluation

Background Standardized letters of evaluation (SLOE) are becoming more widely incorporated into the residency application process to make the letter of recommendation, an already critical component in a residency application packet, more objective. However, it is not currently known if the reviewers of these letters share consensus regarding the strength of an applicant determined by their SLOE. Objective We measured the level of faculty agreement regarding applicant competitiveness as determined by SLOEs and the ability of 2 algorithms to predict faculty consensus rankings. Methods Using data from the 2021-2022 Match cycle from the Council of Residency Directors in Emergency Medicine SLOE Database as a blueprint, authors created 50 fictional SLOEs representative of the national data. Seven faculty then rated these SLOEs in order of applicant competitiveness, defined as suggested rank position. Consensus was evaluated using cutoffs established a priori, and 2 prediction models, a point-based system and a linear regression model, were tested to determine their ability to predict consensus rankings. Results There was strong faculty consensus regarding the interpretation of SLOEs. Within narrow windows of agreement, faculty demonstrated similar ranking patterns with 83% and 93% agreement for "close" and "loose" agreement, respectively. Predictive models yielded a strong correlation with the consensus ranking (point-based system r=0.97, linear regression r=0.97). Conclusions Faculty displayed strong consensus regarding the competitiveness of applicants via SLOEs, adding further support to the use of SLOEs for selection and advising. Two models predicted consensus competitiveness rankings with a high degree of accuracy.

The Utility of the Plastic Surgery Standardized Letter of Recommendation Form in Predicting Residency Match Outcomes

BACKGROUND

Letters of recommendation play an important role in resident selection. While plastic surgery's Standardized Letter of Recommendation (SLOR) form most commonly serves as an adjunct to narrative letters, the SLOR provides objective data in the review process and could eventually replace narrative letters. The utility of the SLOR in predicting Match outcomes has not been studied.

METHODS

Applicant data from 225 first-time residency applicants in 2020-21 were collected. Logistic regression modeling was used to predict Match outcomes. This model was validated using 100 randomly selected applicants from 2019-20.

RESULTS

Rank placement (SLOR Question 6) was the most important factor when predicting Match outcomes (p<0.0001). All other SLOR questions were not predictive and subject to notable score inflation. No SLOR score differences were noted based on race; female applicants were rated higher in two of ten domains (p<0.05).

CONCLUSIONS

One question on the plastic surgery SLOR was highly predictive of an applicant matching. However, the remaining SLOR questions had little utility and were subject to gross score inflation. Further work should be done to optimize the utility of the SLOR in differentiating applicants. This has important implications in ensuring the selection of professional, competent residents according to the aims of the Accreditation Council of Graduate Medical Education.

Words matter: using natural language processing to predict neurosurgical residency match outcomes

OBJECTIVE

Narrative letters of recommendation (NLORs) are considered by neurosurgical program directors to be among the most important parts of the residency application. However, the utility of these NLORs in predicting match outcomes compared to objective measures has not been determined. In this study, the authors compare the performance of machine learning models trained on applicant NLORs and demographic data to predict match outcomes and investigate whether narrative language is predictive of standardized letter of recommendation (SLOR) rankings.

METHODS

This study analyzed 1498 NLORs from 391 applications submitted to a single neurosurgery residency program over the 2020-2021 cycle. Applicant demographics and match outcomes were extracted from Electronic Residency Application Service applications and training program websites. Logistic regression models using least absolute shrinkage and selection operator were trained to predict match outcomes using applicant NLOR text and demographics. Another model was trained on NLOR text to predict SLOR rankings. Model performance was estimated using area under the curve (AUC).

RESULTS

Both the NLOR and demographics models were able to discriminate similarly between match outcomes (AUCs 0.75 and 0.80; p = 0.13). Words including "outstanding," "seamlessly," and "AOA" (Alpha Omega Alpha) were predictive of match success. This model was able to predict SLORs ranked in the top 5%. Words including "highest," "outstanding," and "best" were predictive of the top 5% SLORs.

CONCLUSIONS

NLORs and demographic data similarly discriminate whether applicants will or will not match into a neurosurgical residency program. However, NLORs potentially provide further insight regarding applicant fit. Because words used in NLORs are predictive of both match outcomes and SLOR rankings, continuing to include narrative evaluations may be invaluable to the match process.

Trends in utilization of standardized letters of recommendation in the 2021-2022 neurosurgery application process

OBJECTIVE

Standardized letters of recommendation (SLORs) were introduced during the 2020-2021 neurosurgery residency application cycle, but suffered from poor interrater reliability and grade inflation. Changes were made to the SLOR template and utilization patterns in response to these shortcomings. The authors examined the second year of SLOR utilization. They hypothesized that grade inflation and interrater reliability would be improved from the first iteration. They also hypothesized that increased numbers of letters by single writers would correlate with broader rating distributions.

METHODS

This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2021-2022 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of the letter writer. Interrater reliability was evaluated using Krippendorff's alpha. The frequency of letters written was evaluated using the Kruskal-Wallis H test.

RESULTS

Ninety percent of SLORs rated applicants among the top 25%, but there was a significant decrease in the usage of the top 1% and top 2%-5% ratings. Interrater reliability was poor across all competencies. Writers who completed 1 SLOR rated applicants higher and had a narrower range than those who completed multiple SLORs.

CONCLUSIONS

Changes in the format and subsequent utilization patterns of SLORs have slightly decreased grade inflation; however, interrater reliability remains poor. The most wide-ranging evaluators submitted the highest number of SLORs, suggesting that future evaluation and usage of SLORs should emphasize letter-writer characteristics and numbers of SLORs written. Overall, SLORs have been well and broadly accepted with subtle improvements in the second year of utilization.

Home Program Matching in Neurosurgical Residency Programs: A 7-Year Study

OBJECTIVE

To determine home program matching percentage (staying in a program affiliated with one's medical school) for each neurosurgical residency program in the United States. Secondarily, to elucidate both program-level and resident characteristics associated with home program matching.

METHODS

Demographic and bibliometric characteristics were collected for 1572 residents in US-based and Accreditation Council for Graduate Medical Education (ACGME)-accredited neurosurgery programs over the 2014 to 2020 match period using publicly available websites. Program characteristics were collected, including number of clinical faculty, top 20 Doximity research ranking, top 10 Doximity reputation ranking, top 10 U.S. News department ranking, affiliation with a U.S. News top 10 medical school, and geographic region. Programs were ranked according to home program matching percentage and associations were statistically evaluated.

RESULTS

The average home program matching percentage per residency was 18.6%. New York Presbyterian/Columbia retained the largest percentage of its own medical students with a home program matching percentage of 57.14%. From the resident frame of reference, only a higher preresidency H-index (3.7 ± 4.0 vs. 3.2 ± 3.7, p=0.033) was significantly associated with home program matching. From a program perspective, program size (standardized β=0.234, p=0.006), Doximity research (standardized β=0.206, p=0.031), Doximity reputation (standardized β=0.196, p=0.040), and U.S. News program rankings (standardized β=0.200, p=0.036) were all significantly associated with home program matching. Overall home program matching percentage remained relatively constant over the 2014-2020 time period.

CONCLUSIONS

The results of this study delineate home program matching patterns on a program-by-program level for U.S. neurosurgical residency programs.