Evaluating the Standardized Letter of Recommendation Form in Applicants to Orthopaedic Surgery Residency

Decoding competitiveness: Exploring how emergency medicine faculty interpret standardized letters of evaluation

Background

Given the importance of the standardized letter of evaluation (SLOE) for application to emergency medicine (EM) residency, it is important that SLOE developers and authors understand how reviewers determine SLOE competitiveness. To inform SLOE design and authorship, the authors set out to build a novel theory to explain how faculty holistically interpret SLOE competitiveness.

Methods

The authors used constructivist grounded theory to explore how EM faculty determine SLOE competitiveness. They used purposive sampling to recruit EM faculty participants with at least 1 year of experience in scoring SLOEs. One author conducted hour-long, semistructured interviews over Zoom between August 2023 and March 2024. Two authors iteratively coded the data to develop the initial codebook, organize codes into categories, and build connections to construct the resulting theory.

Results

The authors interviewed 11 EM faculty from throughout the United States. Participants described a complex process to determine SLOE competitiveness. They began by contextualizing the SLOE to determine its trustworthiness and value before using various components of the SLOE to stratify and refine their understanding of competitiveness. Finally, when participants noted the inconsistency between different aspects of the SLOE, they used various methods to reconcile discordances and determine competitiveness.

Conclusions

This study illuminates the framework used by EM faculty to determine applicant competitiveness based on the SLOE and highlights several factors that SLOE authors should consider to ensure the accurate and efficient transfer of information.

How Prominent Are Gender Bias, Racial Bias, and Score Inflation in Orthopaedic Surgery Residency Recommendation Letters? A Systematic Review

INTRODUCTION

Letters of recommendation are considered an important factor in the holistic ranking of orthopaedic residency applications. The standardized letter of recommendation (SLOR) was introduced in 2017 in response to the inherent bias and limited comparative value of traditional LORs. It includes standardized questions about an applicant's orthopaedic qualifications, designed to enable fair comparisons among candidates. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies on the SLOR and narrative letter of recommendation.

QUESTION/PURPOSES

In this systematic review we asked: (1) What is the distribution of applicant SLOR rating among nine domains and summative scores? (2) Are there applicant characteristics associated with SLOR ratings? (3) Is there gender and racial bias in the SLOR and narrative letter of recommendation?

METHODS

PubMed, EBSCO, and Google Scholar electronic databases were queried on March 20, 2023, to identify all studies evaluating SLOR and narrative letter of recommendations between January 1, 2017, and March 20, 2023. Articles that evaluated orthopaedic SLOR or narrative LORs were included. Systematic reviews, case reports, duplicate studies between databases, and grey literature such as abstracts and articles on preprint servers were excluded. Eight studies including 6179 applicants and 18,987 letters of recommendation were included in the final analysis. The applicant classes ranged from years 2014 to 2020. Two reviewers independently evaluated the quality of the included studies using the Joanna Briggs Institute (JBI) tool for cross-sectional studies. The mean JBI score of included studies was 7.4 out of a maximum of 8, with higher scores indicating better quality. The primary outcome was to determine the distribution of applicant SLOR rankings in their summative score. Summative scores were separated into ranked to match, top one-third, middle one-third, lower one-third, and not a fit. Four studies reported the summative SLOR scores of applicants. Our secondary study goal was to assess applicant characteristics associated with SLOR summative scores and assess any bias present in the SLOR or narrative recommendation. Five studies compared SLOR ratings across applicant characteristics including gender, race, USMLE Step 1 score, USMLE Step 2 score, Alpha Omega Alpha (AΩA) membership, clerkship grades, and publications. Gender and racial bias were also assessed across five studies. Four studies utilized a linguistic analysis software to search letters of recommendation for differences in word category use by gender and race.

RESULTS

Studies consistently found that a higher percentage of candidates were identified in the top percentile blocks than is mathematically possible. For example, the two studies that tallied the proportion of candidates ranking in the top 10% of applicants found that 36% and 42% were rated as being in the top 10%. Similarly, articles found more than 87% of applicants scored in the top one-third. Studies had mixed results, but in general, they suggested that AΩA status, higher Step 1 scores, and more research publications have a slightly positive association with increased SLOR scores. We found no evidence of gender bias against women, and in fact, most studies evaluating word choices found word choices for women candidates generally were stronger. Similarly, no consistent disparities were identified in word categories utilized in SLORs based on applicant race.

CONCLUSION

Most notably, a mathematically impossible proportion of applicants were counted in the top percentiles in letters of recommendation. This compromises readers' abilities to differentiate candidates. Factors like AOA status and research publications displayed a modest positive correlation with SLOR scores. Gender bias against women or candidates from racial minority groups was not evident.

CLINICAL RELEVANCE

We suggest that group SLOR authorship, with a consensus assessment by clerkship or residency directors, should be considered. We also recommend the incorporation of mean and SD scores for each letter writer (or group) alongside their letters. Furthermore, studies indicate that submitting all four SLOR letters can substantially improve an applicant's chances of success. We advise students to choose strategically and submit the maximum allowable number of SLORs.

The Utility of the Standardized Letter of Recommendation for Orthopedic Surgery Residency Applicants: A Systematic Review

Despite widespread adoption for evaluating residency candidates, few studies have evaluated the orthopedic standardized letter of recommendation (SLOR). A systematic review using PubMed, Embase, and Web of Science was performed in June 2022. Study design and results from SLOR investigations were compiled. Common outcomes studied were summative rank statement scores and SLOR individual domains. Applicants were rated ranked to match or in the top one-third of rank lists in non-normally distributed frequencies. The association of summative rank statement score with match outcome was rarely studied. Applicants' ratings skew positively, the utility is reportedly limited, and influence on match outcome has been inadequately studied. [Orthopedics. 2024;47(1):e1-e5.].

Confidence of Faculty in Writing Letters of Recommendation for Pediatric Fellowship Applicants

BACKGROUND

The assessment of pediatric residents applying to subspecialty fellowship programs relies on faculty letters of recommendation (LOR). However, it is unclear if pediatric faculty are confident that their LOR are effective.

OBJECTIVE

This study aims to assess the confidence of pediatric faculty in writing an effective LOR for pediatric residents applying to subspecialty fellowship programs.

METHODS

Survey development was conducted using evidence-based best practices. Surveys were distributed via email in 2021 to all full-time pediatric faculty members who taught pediatric residents in a large academic medical center. Categorical values were compared by chi-square test.

RESULTS

Eighty-five out of 150 (57%) faculty members completed the survey. Forty-one percent of participants were very confident that their LOR provided adequate content to assess residents during the application process. Confidence was associated with higher academic rank (p=0.02), frequent contact with residents (p=0.01), and writing >2 LOR in the last five years (p=0.0002). Confident LOR writers were more likely to describe their own background, details about the resident's scholarly activity, and the resident's ability to work as part of a team. Thirty-five percent of respondents reported never considering gender bias when writing LOR, whereas 28% reported always considering gender bias. Eighty-seven percent of respondents reported an interest in receiving LOR writing guidelines.

CONCLUSION

Half of the faculty respondents were not very confident in their ability to write an effective LOR for pediatric residents applying for a fellowship. Faculty development and standardized instructions on writing effective LOR may be helpful both at the institutional and national levels, including the importance of considering gender bias when writing LOR.

Standardized Letter of Recommendation: Can Everyone Be Awesome?

OBJECTIVE

Standardized letters of recommendation (SLOR) are hypothesized to decrease bias and provide consistent domains for evaluation. However, their ability to differentiate among applicants is unknown. The utilization and functionality of SLOR and the impact of SLOR domain rating on matching for colon and rectal surgery (CRS) residency applicants have yet to be assessed.

DESIGN

Descriptive statistics and bivariate analysis were employed. Applicants were categorized into 3 groups; Top-tier(TT): applicants rated 100% Excellent/Very Good; Mid-Tier(MT) applicants rated 80-99% Excellent/Very Good; and Non-Top Tier(NTT) applicants rated <80% Excellent/Very Good.

SETTING

University of Hospitals Cleveland Medical Center.

PARTICIPANTS

SLORs submitted to a single colorectal surgery residency in 2019 were analyzed RESULTS: A total of 101 applicants were included, 54 (53.5%) of the applicants were male. 75 (74.2%) applicants who applied to our residency matched into a CRS residency, compared to the national rate of 66%. Of the 101 applicants with SLOR, 54 (53.5%) were categorized as TT, 26 (25.7%) as MT, and 21(20.8%) as NTT. The univariable analysis demonstrated a statistically significant difference in research experience (p=0.029) and match status (p=0.01) between applicant tiers. There were no statistically significant differences between applicant-tier and demographics, foreign medical graduates (FMG), H-indices, ABSITE scores, type of residency, preliminary year, completing an unaccredited CRS, and applicants with an additional degree. On multivariable analysis age (OR=0.65; CI=0.48-0.87) and FMG applicants (OR=0.05; CI=0.01-0.44) were inversely associated with successfully matching. Compared to TT applicants, MT (OR=0.07; CI=0.01-0.57) and NTT (OR=0.04; CI=0.01-0.34) applicants were inversely associated with a successful match. Individuals who completed research prior to residency but after medical school was associated with successfully matching (p=0.009).

CONCLUSIONS

The presence of MT and NTT ratings is associated with failure to match and may represent an area of concern for CRS programs rather than a tool to discern differences between candidates.

Development and Assessment of Early Utilization of the Standardized Letter of Recommendation for Use in the Radiology Residency Match

RATIONALE AND OBJECTIVES

Letters of recommendation are essential for residency applications. Traditionally, narrative letters have been used. Standardized letters of recommendation (SLOR) have been developed and have benefits to the writer and reader. The goal was to develop an informative, meaningful, and efficient SLOR for the radiology residency Match and to assess its early use.

MATERIALS AND METHODS

An Association of University Radiologists (AUR) Research and Education Venture Fund Grant was awarded for the development of a SLOR for use in the radiology residency Match. Grant recipients developed the letter and modifications were based on feedback from the Alliance of Medical Student Educators in Radiology (AMSER) SLOR task force and a larger task force including additional AUR and Association of Program Directors in Radiology (APDR) members. AUR and APDR members were surveyed in 2020 to assess the radiology SLOR.

RESULTS

The radiology SLOR became available for use in September 2018. It highlights the top six traits or abilities deemed most valued in a resident candidate and guides the writer to develop a concrete narrative. Top perceived benefits of the Radiology SLOR are ease of reading and interpretation. Top perceived drawbacks are an impersonal format and concerns regarding program directors' perceptions of the new SLOR. SLOR utilization increased in the second year of availability.

CONCLUSION

The radiology SLOR was developed and first used in 2018. Its use increased over two years and will likely continue to increase given the perceived benefits and increasing awareness of the SLOR.

Trends in utilization of standardized letters of recommendation in the 2021-2022 neurosurgery application process

OBJECTIVE

Standardized letters of recommendation (SLORs) were introduced during the 2020-2021 neurosurgery residency application cycle, but suffered from poor interrater reliability and grade inflation. Changes were made to the SLOR template and utilization patterns in response to these shortcomings. The authors examined the second year of SLOR utilization. They hypothesized that grade inflation and interrater reliability would be improved from the first iteration. They also hypothesized that increased numbers of letters by single writers would correlate with broader rating distributions.

METHODS

This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2021-2022 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of the letter writer. Interrater reliability was evaluated using Krippendorff's alpha. The frequency of letters written was evaluated using the Kruskal-Wallis H test.

RESULTS

Ninety percent of SLORs rated applicants among the top 25%, but there was a significant decrease in the usage of the top 1% and top 2%-5% ratings. Interrater reliability was poor across all competencies. Writers who completed 1 SLOR rated applicants higher and had a narrower range than those who completed multiple SLORs.

CONCLUSIONS

Changes in the format and subsequent utilization patterns of SLORs have slightly decreased grade inflation; however, interrater reliability remains poor. The most wide-ranging evaluators submitted the highest number of SLORs, suggesting that future evaluation and usage of SLORs should emphasize letter-writer characteristics and numbers of SLORs written. Overall, SLORs have been well and broadly accepted with subtle improvements in the second year of utilization.

Reshaping Residency Recruitment: Achieving Alignment Between Applicants and Programs in Surgery

OBJECTIVE

The residency recruitment process has become increasingly challenging for both applicants and program directors, in part, due to the inflation in the number of applications per student. As a result, it has become more daunting for programs to design processes that evaluate applicants holistically. Furthermore, the existing methods used to evaluate and select applicants do not necessarily predict success in residency and may inadvertently lend to gender, racial, and ethnic bias. This narrative review aims to identify innovative tools used in residency recruitment that will allow programs and applicants to better determine concordance of interests and achieve value alignment while supporting improved, objective evaluation of an applicant's unique attributes and experiences.

DESIGN

PubMed was used to conduct a narrative review of recruitment strategies in admission processes of undergraduate and graduate medical education between 1975 and June 2021, using the designated Medical Subject Heading (MeSH0 terms. Inclusion criteria were established surrounding innovative tools to better objectively screen, evaluate, or select applicants. Strategies relying primarily on traditional metrics (United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, and clerkship grades) were excluded.

RESULTS

Forty-two articles met specific inclusion criteria. Using these articles, a framework was created with two specific aims: (1) to allow applicants and programs to express or assess interest and (2) to foster objective review of unique applicant attributes, skills, experiences, and competencies that align with program mission and values. The following five innovative tools for recruitment were identified: preference signaling, secondary applications, standardized letters of recommendation, situational judgment testing, and surgical simulation.

CONCLUSIONS

As the number of applications continues to rise, strategies must be implemented to allow applicants and institutions to achieve better alignment or "fit," while also giving balanced consideration to all of an applicant's unique characteristics. A more holistic approach to applicant selection is a necessary tool in order to increase diversity and inclusion within the field of surgery.

How Did Coronavirus-19 Impact the Expenses for Medical Students Applying to an Orthopaedic Surgery Residency in 2020 to 2021?

BACKGROUND

Orthopaedic surgery is one of the most competitive specialties for residency applicants. For the 2021 residency match, the coronavirus-19 pandemic introduced complexity for programs and applicants because away rotations were limited and in-person interviews were cancelled. This may have changed the landscape in terms of expenses for candidates in important ways, but this topic has been insufficiently studied.

QUESTIONS/PURPOSES

Given that in 2021, students did not attend away rotations and all interviews were held virtually, we asked (1) What were the financial savings associated with this change? (2) Was medical school geographic region associated with differences in expenses when applying to residency?

METHODS

A retrospective, cross-sectional analysis of the 2020 and 2021 Texas Seeking Transparency in Application to Residency Dashboard database was performed. The data were derived from an online survey of a nationwide pool of applicants from 87% (123 of 141) of US allopathic medical schools upon conclusion of the match. The response percentage was 29% (521 of 1794). We believe this nationwide dataset represents the largest and most current data for this applicant group. Responses from applicants applying to orthopaedic surgery residency in the year before the COVID-19 pandemic application changes (2020) and during COVID-19 (2021) were queried and compared. After the orthopaedic surgery match, the database was evaluated for individual (application costs, away rotation expenses, and interview expenses) and total expenses for medical school seniors applying to orthopaedic surgery residency. Applicant characteristics were compared between application cycles. The 2020 to 2021 Texas Seeking Transparency in Application to Residency Dashboard database had 521 responses (n = 263 in 2020 and n = 258 in 2021) from applicants applying to orthopaedic surgery residency. Demographic and applicant characteristics were comparable between application cycles. Median expenses are reported with percentile distributions and geographic comparisons. A Mann-Whitney U test or Kruskal-Wallis H test was used to determine whether there were statistically significant differences in expenses between years and between medical school regions at a p value threshold of < 0.05.

RESULTS

For all applicants, the median total expenses (USD 7250 versus USD 2250), application costs (USD 2250 versus USD 1750), away rotation expenses (USD 2750 versus USD 250), and interview expenses (USD 2250 versus USD 75) declined in 2021 compared with 2020 (all p < 0.001). The median total savings in expenses for all applicants in 2021 compared with 2020 was USD 5000. In 2021, median total expenses were lower in all geographic regions with the greatest savings from applicants in the West (USD 6000); in addition, the difference in median total expenses between the geographic region with the highest total expenses and the lowest total expenses was lower in the pandemic year than it was in the year prior (USD 1000 versus USD 1500; p < 0.001). In 2021, there were differences in total expenses between the Northeast (USD 1750), West (USD 1750), and Central (USD 2750) regions (p < 0.001). From 2020 to 2021, only application fees from Northeast applicants differed (USD 2250 versus USD 1250; p < 0.001). In 2020, interview expenses were not different between all regions (USD 2250 Northeast and West versus USD 2750 Central and South; p = 0.19); similarly in 2021, interview expenses were similar between all regions (USD 75 versus USD 75; p = 0.82). Finally, in 2020, Northeast (USD 3250) and Western (USD 3250) applicants spent more for away rotations than Southern (USD 2750) and Central (USD 2250) applicants (p = 0.01). In 2021, applicants from schools in the South (USD 250) and Central (USD 250) regions spent more than their counterparts (USD 0; p = 0.028).

CONCLUSION

In the COVID-19 application cycle, the median expenditures of orthopaedic residency candidates were USD 5000 lower than they were in the previous year; the difference can be attributed to the use of virtual interviews and the lack of away rotations. There are geographic implications, with applicants from Western United States medical schools potentially saving the most. Despite the financial savings during the 2021 match, further study related to the long-term success of the current application process (both for applicants and programs) is needed. The recommendation in May 2020 by the AOA Council of Orthopaedic Residency Directors (CORD) to limit the number of applications submitted by candidates with USMLE Step 1 scores greater than 235 did not result in any considerable decline in applications submitted or expenses. A better understanding of how differences in these expenses may influence our specialty's ability to attract socioeconomically diverse candidates would be important, and we need to explore perceived and actual financial obstacles to obtaining this diversity in the application process. Finally, avenues should be explored by program directors and chairpersons to reduce the expenses of the traditional application process while maintaining recruitment of top candidates.

LEVEL OF EVIDENCE

Level IV, economic analysis.

The Orthopaedic Resident Selection Process: Proposed Reforms and Lessons From Other Specialties

INTRODUCTION

Proposals for substantive reforms to the orthopaedic resident selection process are growing, given increasing applicant competitiveness, burgeoning inefficiencies and inequities of the current system, and impending transition of Step 1 to pass/fail. The COVID-19 pandemic has further catalyzed the need for reforms, offering unprecedented opportunities to pilot novel changes. However, a comprehensive collation of all proposed and implemented orthopaedic reforms is currently lacking. Thus, we aimed to characterize proposed orthopaedic-specific resident selection reforms in the context of reforms implemented by other specialties.

METHODS

EMBASE, MEDLINE, Scopus, and Web of Science databases were searched for references proposing reforms to the orthopaedic resident selection process published from 2005 to 2020. An inductive approach to qualitative content analysis was used to categorize reforms.

RESULTS

Twenty-six articles proposing 13 unique reforms to the orthopaedic resident selection process were identified. The most commonly proposed reforms included noncognitive assessments (n = 8), application caps (n = 7), standardized letters of recommendation (n = 5), program-specific supplemental applications (n = 5), creation of a centralized database of standardized program information (n = 4), use of a standardized applicant composite score (n = 4), and a moratorium on postinterview communication (n = 4). Importantly, nearly all of these reforms have also been proposed or implemented by other specialties.

DISCUSSION

Numerous reforms to the orthopaedic resident selection process have been suggested over the past 15 years, several of which have been implemented on a program-specific basis, including noncognitive assessments, supplemental applications, and standardized letters of recommendation. Careful examination of applicant and program experiences and Match outcomes after these reforms is imperative to inform future directions.

Compliance with CDIM-APDIM Guidelines for Department of Medicine Letters: an Opportunity to Improve Communication Across the Continuum

BACKGROUND

With rising applications to internal medicine programs and pending changes in United States Medical Licensing Examination Step 1 score reporting, program directors desire transparent data for comparing applicants. The Department of Medicine Letters of Recommendation (DOM LORs) are frequently used to assess applicants and have the potential to provide clearly defined data on performance including stratification of a medical school class. Despite published guidelines on the expected content of the DOM LOR, these LORs do not always meet that need.

OBJECTIVES

To better understand the degree to which DOM LORs comply with published guidelines.

METHODS

We reviewed DOM LORs from 146 of 155 LCME-accredited medical schools in the 2019 Match cycle, assessing for compliance with published guidelines.

RESULTS

Adherence to the recommendation for DOM LORs to provide a final characterization of performance relative to peers was low (68/146, 47%). Of those that provided a final characterization, 19/68 (28%) provided a quantitative measure, and 49/68 (72%) provided a qualitative descriptor. Only 17/49 (35%) with qualitative terms described those terms, and thirteen distinct qualitative scales were identified. Ranking systems varied, with seven different titles given to highest performers. Explanations about determination of ranking groups were provided in 12% of cases.

CONCLUSIONS

Adherence to published guidelines for DOM LORs varies but is generally low. For program directors desiring transparent data to use in application review, clearly defined data on student performance, stratification groupings, and common language across schools could improve the utility of DOM LORs.

How Does Geographic Region Affect the Total and Individual Costs for Medical Students Applying to the Competitive Surgical Residencies?

OBJECTIVE

Studies have individually assessed the expenses for medical students desiring a surgical residency in neurosurgery, orthopaedic surgery, otolaryngology, plastic surgery, and urology. A detailed comparison of expenses by geographic region have not been reported. The aim was to geographically compare the expenses of United States medical students applying to the 5 most competitive surgical residencies.

SETTING/PARTICIPANTS

Anonymous nationwide survey of plastic surgery, orthopaedic surgery, otolaryngology, urology, and neurological surgery residency applicants.

DESIGN

A cross sectional, retrospective analysis of the 2019-2020 Texas STAR Dashboard database, an online tool generated from a nationwide survey of students. Individual (application, away rotation, interview) and total costs for medical school seniors were recorded in addition to applicant characteristics. Mean and median costs were reported for each specialty with percentile distributions and geographic comparisons. A Kruskal-Wallis H test was performed to compare differences in costs between surgical specialties and medical school region.

RESULTS

In total, 1136 applicants to surgical residency were included. The number of applicants to orthopaedic surgery (OS) (n = 459), neurological surgery (NS) (n = 121), urology (UR) (n = 191), plastic surgery (PS) (n = 117), and otolaryngology (OTO) (n = 248) were reported. Mean total costs were (OS; $8,205), (NS; $11,882), (UR; $8,207), (PS; $10,845), and (OTO; $7,516) (p ≤ 0.029). Application fees were only significantly different in the northeast and southern applicants between the different specialties, notably orthopaedic surgery applicants spent the most. In all geographic regions excluding the west, neurosurgery and plastic surgery applicants spent significantly more than other specialties for interview costs and away rotation costs. In all geographic regions neurosurgery and plastic surgery applicants spent significantly more than other specialties in total costs.

CONCLUSIONS

Orthopaedic surgery applicants spend the most on application fees in select geographic regions. Neurosurgery and plastic surgery applicants spend more on interviews, away rotations, and total costs in nearly all geographic regions.

An Examination of Standardized Letters of Recommendation Rating Scales Among Neurosurgical Residency Candidates During the 2020-2021 Application Cycle

BACKGROUND

Letters of recommendation (LORs) are historically an important, though subjective, component of the neurosurgery residency application process. Standardized LORs (SLORs) were introduced during the 2020 to 2021 application cycle. The intent of SLORs is to allow objective comparison of applicants and to reduce bias.

OBJECTIVE

To examine the utility of SLORs during this application cycle. We hypothesized that "grade inflation" and poor inter-rater reliability, as described by other specialties using SLORs, would limit the utility of SLORs in their current form.

METHODS

This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2020 to 2021 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of letter writer. Inter-rater reliability was evaluated using Krippendorff's alpha.

RESULTS

One or more SLORs was submitted as part of 298 of 393 applications (76%). Approximately 58.3% of letters written by neurosurgery chairpersons rated a given applicant as being within the top 5% across all competencies. Approximately 44.4% of program director letters similarly rated applicants as amongst the top 5%, while 73.2% and 81.4% of letters by other neurosurgeons and general surgery evaluators, respectively, rated applicants in the top 5%. Inter-rater reliability was poor (<0.33) in all rating categories, including overall (α = 0.18).

CONCLUSION

The utility of the first iteration of SLORs in neurosurgery applications is undermined by significant "grade inflation" and poor inter-rater reliability. Improvements are necessary for SLORs if they are to provide meaningful information in future application cycles.

Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation

BACKGROUND

The standardized letter of evaluation (SLOE) is the application component that program directors value most when evaluating candidates to interview and rank for emergency medicine (EM) residency. Given its successful implementation, other specialties, including otolaryngology, dermatology, and orthopedics, have adopted similar SLOEs of their own, and more specialties are considering creating one. Unfortunately, for such a significant assessment tool, no study to date has comprehensively examined the validity evidence for the EM SLOE.

OBJECTIVE

We summarized the published evidence for validity for the EM SLOE using Messick's framework for validity evidence.

METHODS

A scoping review of the validity evidence of the EM SLOE was performed in 2020. A scoping review was chosen to identify gaps and future directions, and because the heterogeneity of the literature makes a systematic review difficult. Included articles were assigned to an aspect of Messick's framework and determined to provide evidence for or against validity.

RESULTS

There have been 22 articles published relating to validity evidence for the EM SLOE. There is evidence for content validity; however, there is a lack of evidence for internal structure, relation to other variables, and consequences. Additionally, the literature regarding response process demonstrates evidence against validity.

CONCLUSIONS

Overall, there is little published evidence in support of validity for the EM SLOE. Stakeholders need to consider changing the ranking system, improving standardization of clerkships, and further studying relation to other variables to improve validity. This will be important across GME as more specialties adopt a standardized letter.

Are Narrative Letters of Recommendation for Medical Students Interpreted as Intended by Orthopaedic Surgery Residency Programs?

BACKGROUND

Narrative letters of recommendation are an important component of the residency application process. However, because narrative letters of recommendation are almost always positive, it is unclear whether those reviewing the letters understand the writer's intended strength of support for a given applicant.

QUESTIONS/PURPOSES

(1) Is the perception of letter readers for narrative letters of recommendation consistent with the intention of the letter's author? (2) Is there inter-reviewer consistency in selection committee members' perceptions of the narrative letters of recommendation?

METHODS

Letter writers who wrote two or more narrative letters of recommendation for applicants to one university-based orthopaedic residency program for the 2014 to 2015 application cycle were sent a survey linked to a specific letter of recommendation they authored to assess the intended meaning regarding the strength of an applicant. A total of 247 unstructured letters of recommendation and accompanying surveys were sent to their authors, and 157 surveys were returned and form the basis of this study (response percentage 64%). The seven core members of the admissions committee (of 22 total reviewers) at a university-based residency program were sent a similar survey regarding their perception of the letter. To answer our research question about whether letter readers' perceptions about a candidate were consistent with the letter writer's intention, we used kappa values to determine agreement for survey questions involving discrete variables and Spearman correlation coefficients (SCCs) to determine agreement for survey questions involving continuous variables. To answer our research question regarding inter-reviewer consistency among the seven faculty members, we compared the letter readers' responses to each survey question using intraclass correlation coefficients (ICCs).

RESULTS

There was a negligible to moderate correlation between the intended and perceived strength of the letters (SCC 0.26 to 0.57), with only one of seven letter readers scoring in the moderate correlation category. When stratifying the applicants into thirds, there was only slight agreement (kappa 0.07 to 0.19) between the writers and reviewers. There were similarly low kappa values for agreement about how the writers and readers felt regarding the candidate matching into their program (kappa 0.14 to 0.30). The ICC for each question among the seven faculty reviewers ranged from poor to moderate (ICC 0.42 to 0.52).

CONCLUSION

Our results demonstrate that the reader's perception of narrative letters of recommendation did not correlate well with the letter writer's intended meaning and was not consistent between letter readers at a single university-based urban orthopaedic surgery residency program.

CLINICAL RELEVANCE

Given the low correlation between the intended strength of the letter writers and the perceived strength of those letters, we believe that other options such as a slider bar or agreed-upon wording as is used in many dean's letters may be helpful.

Systems-Level Reforms to the US Resident Selection Process: A Scoping Review

BACKGROUND

Calls to reform the US resident selection process are growing, given increasing competition and inefficiencies of the current system. Though numerous reforms have been proposed, they have not been comprehensively cataloged.

OBJECTIVE

This scoping review was conducted to characterize and categorize literature proposing systems-level reforms to the resident selection process.

METHODS

Following Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches of Embase, MEDLINE, Scopus, and Web of Science databases were performed for references published from January 2005 to February 2020. Articles were included if they proposed reforms that were applicable or generalizable to all applicants, medical schools, or residency programs. An inductive approach to qualitative content analysis was used to generate codes and higher-order categories.

RESULTS

Of 10 407 unique references screened, 116 met our inclusion criteria. Qualitative analysis generated 34 codes that were grouped into 14 categories according to the broad stages of resident selection: application submission, application review, interviews, and the Match. The most commonly proposed reforms were implementation of an application cap (n = 28), creation of a standardized program database (n = 21), utilization of standardized letters of evaluation (n = 20), and pre-interview screening (n = 13).

CONCLUSIONS

This scoping review collated and categorized proposed reforms to the resident selection process, developing a common language and framework to facilitate national conversations and change.

Cost Analysis of Medical Students Applying to Orthopaedic Surgery Residency: Implications for the 2020 to 2021 Application Cycle During COVID-19

Although studies have evaluated the economic burden to medical students desiring an orthopaedic residency broadly, no study has evaluated in detail the application costs, away rotation costs, interview costs, and total costs. Given that the Association of American Medical Colleges and the American Orthopaedic Association's Council of Orthopaedic Residency Directors have recommended orthopaedic surgery residency programs for the 2021 residency application cycle cancel away rotations and in-person interviews, our objective was to evaluate the cost savings to medical students applying during this time and the potential implications.

Race- and Gender-Based Differences in Descriptions of Applicants in the Letters of Recommendation for Orthopaedic Surgery Residency

Background

Letters of recommendation (LOR) are an important component of trainee advancement and assessment. Examination of word use in LOR has demonstrated significant differences in how letter writers describe female and male applicants. Given the emphasis on increasing both gender and racial diversity among orthopaedic surgeons, we aimed to study gender and racial differences in LOR for applicants to orthopaedic surgery residencies.

Methods

All applications submitted to a single, academic orthopaedic residency program in 2018 were included. Self-identified gender and race were recorded. The LOR were analyzed via a text analysis software program using previously described categories of communal, agentic, grindstone, ability, and standout words. We examined the relative frequency of word use in letters for (1) male and female applicants and (2) white and underrepresented in orthopaedics (UiO) applicants, with the subgroup analysis based on whether standardized (using the American Orthopaedic Association template) or traditional (narrative) LOR were used.

Results

Two thousand six hundred twenty-five LOR were submitted for 730 applicants (79% men). Fifty-nine percent of applicants were self-identified as white, and 34% were self-identified as UiO. In traditional LOR, standout words (odds ratio [OR] 1.07; p = 0.01) were more likely to be used in letters for women compared with men, with no difference in any other word-use category. In standardized LOR, there were no gender-based differences in any word category. In traditional LOR, grindstone words (OR = 0.96; p = 0.02) were more likely to be used in letters for UiO than white applicants, whereas standout words (OR = 1.05; p = 0.04) were more likely to be used in letters for white candidates. In standardized LOR, there were no race-based differences in any word category use.

Conclusions

Small differences were found in the categories of words used to describe male and female candidates and white and UiO candidates. These differences were not present in the standardized LOR compared with traditional LOR. It is possible that the use of standardized LOR may reduce gender- and race-based bias in the narrative assessment of applicants.

The Impact of COVID-19 on the Orthopaedic Surgery Residency Application Process

Over recent months, coronavirus disease 2019 (COVID-19) has swept the world as a global pandemic, largely changing the practice of medicine as it was previously known. Physician trainees have not been immune to these changes-uncertainty during this time is undeniable for medical students at all levels of training. Of particular importance is the potential impact of COVID-19 on the upcoming residency application process for rising fourth-year students; a further source of added complexity in light of the newly integrated allopathic and osteopathic match in the 2020 to 2021 cycle. Owing to the impact COVID-19 could have on the residency match, insight regarding inevitable alterations to the application process and how medical students can adapt is in high demand. Furthermore, it is very possible that programs will inquire about how applicants spent their time while not in the hospital because of COVID-19, and applicants should be prepared to provide a meaningful answer. Although competitive at a basal level, the complexity of COVID-19 now presents an unforeseen, superimposed development in the quest to match. In this article, we aim to discuss and provide potential strategies for navigating the impact of COVID-19 on the residency application process for orthopaedic surgery.