Machine Learning for Predictive Analysis of Otolaryngology Residency Letters of Recommendation

INTRODUCTION

Letters of recommendation (LORs) are a highly influential yet subjective and often enigmatic aspect of the residency application process. This study hypothesizes that LORs do contain valuable insights into applicants and can be used to predict outcomes. This pilot study utilizes natural language processing and machine learning (ML) models using LOR text to predict interview invitations for otolaryngology residency applicants.

METHODS

A total of 1642 LORs from the 2022-2023 application cycle were retrospectively retrieved from a single institution. LORs were preprocessed and vectorized using three different techniques to represent the text in a way that an ML model can understand written prose: CountVectorizer (CV), Term Frequency-Inverse Document Frequency (TF-IDF), and Word2Vec (WV). Then, the LORs were trained and tested on five ML models: Logistic Regression (LR), Naive Bayes (NB), Decision Tree (DT), Random Forest (RF), and Support Vector Machine (SVM).

RESULTS

Of the 337 applicants, 67 were interviewed and 270 were not interviewed. In total, 1642 LORs (26.7% interviewed) were analyzed. The two best-performing ML models in predicting interview invitations were the TF-IDF vectorized DT and CV vectorized DT models.

CONCLUSION

This preliminary study revealed that ML models and vectorization combinations can provide better-than-chance predictions for interview invitations for otolaryngology residency applicants. The high-performing ML models were able to classify meaningful information from the LORs to predict applicant interview invitation. The potential of an automated process to help predict an applicant's likelihood of obtaining an interview invitation could be a valuable tool for training programs in the future.

LEVEL OF EVIDENCE

N/A Laryngoscope, 2024.

Gender bias in colorectal surgery fellowship letters of recommendation

BACKGROUND

As leaders strive to create equitable surgical pipelines, one process under scrutiny is letters of recommendation (LORs). We sought to review the Colon and Rectal Surgery (CRS) Resident Candidate Assessment questionnaire and LORs for gendered differences.

METHODS

This retrospective observational study of letters of recommendation to CRS fellowship during the 2018-2019 application cycle utilized linguistic Inquiry and Word Count (LIWC2015) software to assess letter length and themes comparing differences by applicant and referee gender.

RESULTS

103 applicants (35 ​% women) with 363 LORs (16 ​% written by women) were included. Short answer responses were longer for women applicants, while LORs were longer for men applicants (368 vs 325 words p ​= ​0.03). Men applicants' strengths had more technical skill descriptors, while women applicants' strengths had more emotional language and cognitive and perceptual words.

CONCLUSIONS

This study found significant differences between LORs written for CRS fellowship applicants based on gender.

Do gender differences exist in letters of recommendation for reproductive endocrinology and infertility fellowship?

OBJECTIVE(S)

To determine if gender differences exist in letters of recommendation (LORs) for reproductive endocrinology and infertility fellowship.

DESIGN

Cohort study.

SETTING

Academic medical center.

PATIENTS

A cohort of applicants to a reproductive endocrinology and infertility fellowship program at a single Midwestern academic institution in a single year.

INTERVENTIONS

None.

MAIN OUTCOME MEASURES

Quantitative measures included linguistics inquiry and word count analysis, with 4 summary variables and 25 word categories, to quantify tone and themes present by gender. Performing qualitative analysis in parallel to linguistic analysis allowed for the exploration of themes not conveyed in quantitative methods alone. Qualitative measures included the frequency of code themes in LOR by applicant gender.

RESULTS

There were 272 letters from 72 applicants, 54 (76%) of which were women and 17 (24%) were men. One applicant was excluded because gender information was not specified; 269 letters were included in the quantitative linguistics and qualitative coding analysis. One hundred 10 letters (41%) were written by women and 159 (59%) by men. LOR written for men had a higher mean word count than those written for women (537 vs. 474, P=.04). Linguistics Inquiry and Word Count analysis exhibited more risk words used to describe men applicants (P=.01). When comparing word categories by letter writer gender, women letter writers more frequently used communal, affect, and home word categories whereas men writers more frequently used affiliation-related words. Qualitative analysis revealed that letters for men applicants described leadership, altruism, rapport with patients, and strong endorsements more frequently, whereas women applicants' letters more often mentioned doubt raisers and disclosures of personal life. In addition, letters for women applicants more often described candidates as drama free, not easy to "fluster," and not complainers. Furthermore, letters for women more frequently disclosed parenthood status.

CONCLUSIONS

Gender differences exist among both letter writers and applicants. Qualitative analysis revealed that women applicants were more likely to be described with doubt raisers and personal life disclosures, whereas men were more likely to be described by their altruism and with a strong endorsement. Increasing bias awareness and implementing mitigation strategies would improve gender equity in LOR.

Key factors for residency interview selection from the National Resident Matching Program: analysis of residency Program Director surveys, 2016-2020

CONTEXT

As the number of medical school graduates continues to outpace the available residency training positions, applying for residency in the United States has become a highly competitive process, often associated with a low rate of selection and invitation for interview. The National Resident Matching Program (NRMP) Program Director survey provides data assessing factors considered by Program Directors (PD) in selecting and inviting candidates for interview. Assessing the evolution of these factors over time is efficacious to inform and guide prospective applicants toward improving preparation for residency application.

OBJECTIVES

We aim to synthesize NRMP data showing factors that PDs reported and rated as important in their decision to select and invite applicants for interview.

METHODS

Data from residency PD surveys from 2008 to 2021 were accessed, but after applying inclusion/exclusion criteria, only the data from 2016 to 2020 were reviewed and analyzed. The NRMP survey reports provided two metrics that characterized PDs' evaluation of the residency factors for interview, namely, "percent citing factor" and "average rating" on a 0 to 5 Likert-type scale. These two metrics were combined into an aggregate measure of importance (AI), and another measure of relative importance (RI) was constructed from normalizing the AI of each individual factor to the sum of the AI within each survey year.

RESULTS

The top ranked factors were United States Medical Licensing Examination (USMLE) Step 1/Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1, Letter of Recommendation (LOR) in the specialty, Medical Student Performance Evaluation (MSPE/Dean's Letter), and USMLE Step 2 Clinical Knowledge (CK)/COMLEX Level 2 Cognitive Exam (CE) score, any failed attempt in USMLE/COMLEX, and perceived commitment to specialty. Factors rising in importance were Audition Elective/Rotation Within Your Department, Personal Statement (PS), Perceived Commitment to Specialty, Perceived Interest in Program, LOR in the Specialty, Other Life Experience, and Personal Prior Knowledge of the Applicant. Factors with declining importance were Interest in Academic Career, Awards or Special Honors in Basic Sciences, Graduate of Highly Regarded US Medical School, Awards or Special Honors in Clinical Clerkships, Lack of Gaps in Medical Education, Awards or Special Honors in Clerkship in Desired Specialty, and Consistency of Grades. Compared to the 2021 PD survey, our findings show continued predictive consistency, particularly related to specialty and program commitment.

CONCLUSIONS

The factors identified for the selection of medical school graduates for interview into a residency program reveal that PDs move toward a more integrated approach. Specifically, PDs are placing increasing emphasis on factors that border on subjective qualities more so than the more traditional, quantitative, and objective metrics. Medical students and educators need to continually apprise themselves of the NRMP data to inform students' preparation endeavors throughout medical school to strengthen their application portfolios and enhance their competitiveness for the matching process.

Vascular surgery integrated resident selection criteria in the pass or fail era

OBJECTIVES

Vascular surgery integrated residency (VSIR) programs are highly competitive; however, criteria for resident selection remain opaque and non-standardized. The already unclear selection criteria will be further impacted by the impending transition of the United States Medical Licensing Examination (USMLE) Step 1 from numeric scores to a binary pass/fail outcome. The purpose of this study was to investigate the historical and anticipated selection criteria of VSIR applicants.

METHODS

This was a cross-sectional, nationwide, 59-item survey that was sent to all VSIR program directors (PDs). Data was analyzed using the Fisher exact test if categorical and the Mann-Whitney U test and the Kruskal-Wallis test if ordinal.

RESULTS

Forty of 69 PDs (58%) responded to the survey. University-based programs constituted 85% of responders. Most VSIR PDs (65%) reported reviewing between 101 to 150 applications for 1 to 2 positions annually. Forty-two percent of the responding PDs reported sole responsibility for inviting applicants to interview, whereas 50% had a team of faculty responsible for reviewing applications. On a five-point Likert scale, letters of recommendation (LOR) from vascular surgeons or colleagues (a person the PD knows) were the most important objective criteria. Work within a team structure was rated highest among subjective criteria. The majority of respondents (72%) currently use the Step 1 score as a primary method to screen applicants. Regional differences in use of Step 1 score as a primary screening method were: Midwest (100%), Northeast (76%), South (43%), and West (40%) (P = .01). PDs responded that that they will use USMLE Step 2 score (42%) and LOR (10%) to replace USMLE Step 1 score. The current top ranked selection criteria are letters from a vascular surgeon, USMLE Step 1 score and overall LOR. The proposed top ranked selection criteria after transition of USMLE Step 1 to pass/fail include LOR overall followed by Step 2 score.

CONCLUSIONS

This is the first study to evaluate the selection criteria used by PDs for VSIR. The landscape of VSIR selection criteria is shifting and increasing transparency is essential to applicants' understanding of the selection process. The transition of USMLE Step 1 to a pass/fail report will shift the attention to Step 2 scores and elevate the importance of other relatively more subjective criteria. Defining VSIR program selection criteria is an important first step toward establishing holistic review processes that are transparent and equitable.

Which Application Factors Are Associated With Outstanding Performance in Orthopaedic Surgery Residency?

BACKGROUND

Identifying ideal candidates for orthopaedic surgery residency is difficult. Data available for applicant selection are evolving; preclinical grades and the Alpha Omega Alpha (AOA) honors society are being phased out at some medical schools. Similarly, three-digit United States Medical Licensing Examination (USMLE) Step 1 scores have been eliminated. There is renewed interest in improving resident selection to provide a diverse, comprehensive educational opportunity that produces orthopaedic surgeons who are prepared for practice.

QUESTIONS/PURPOSES

We sought to identify whether (1) academic achievements, (2) letters of recommendation, (3) research activity, and (4) miscellaneous factors available on Electronic Residency Application Service (ERAS) applications were associated with outstanding residency performance.

METHODS

Ten faculty members (22% of all full-time faculty) with extensive educational involvement for at least 7 years, whose expertise covered all subspecialty departments at an urban, academic orthopaedic surgery residency program, were given an anonymous survey on the performance of the four most recent classes of residency graduates (24 residents). This survey was developed due to the lack of a validated residency outcomes tool or objective metrics for residency performance. The evaluated criteria were decided upon after discussion by a relatively large group of academic orthopaedic surgeons considering the factors most important for graduating orthopaedic residents. The faculty were selected based on their long-term knowledge of the residency, along with their diversity of specialty and backgrounds; there were no nonresponders. Faculty graded each resident on a scale from 1 to 10 (higher is better) on six criteria: surgical technical skills, research productivity, clinical knowledge, professionalism, personality, and fellowship match. The mean of the faculty ratings made by all faculty for all six criteria was calculated, producing the overall residency performance score. Factors available on each resident's ERAS application were then correlated with their overall residency performance score. Categorical ERAS factors, including AOA status, five or more honors in core clerkships, at least three exceptional letters of recommendation, collegiate athletics participation, expertise with a musical instrument, and research (6-year) track residents, were correlated with overall residency performance score via point biserial analysis. Continuous ERAS factors including USMLE Step 1 and Step 2 scores, number of publications before residency, number of research years before residency, medical school ranking, and number of volunteer experiences were correlated with overall residency performance score via Pearson correlation. USMLE Step 1 three-digit scores were evaluated despite their recent elimination because of their historic importance as a screening tool for residency interviews and for comparison to USMLE Step 2, which retains a three-digit score. Application factors with a p < 0.2 on univariate analysis (five or more honors in core clerkships, at least three exceptional letters of recommendation, research track residents) were included in a stepwise linear regression model with "overall residency performance score" as the outcome variable. All p values < 0.05 were considered significant.

RESULTS

The mean overall residency performance score was 7.9 ± 1.2. Applicants with at least five honors grades in core clerkships had overall residency performance scores 1.2 points greater than those of their peers (95% confidence interval (CI) 0.3 to 2.0; p = 0.01, Cohen ƒ 2 = 0.2, representing a small effect size). ERAS applications including at least three exceptional letters of recommendation were associated with a 0.9-point increase in residency performance (95% CI 0.02 to 1.7; p = 0.046, Cohen ƒ 2 = 0.1, representing a small effect size). Participation in the residency research (6-year) track was associated with a 1-point improvement in residency performance (95% CI 0.1 to 1.9; p = 0.03, Cohen ƒ 2 = 0.2, again, representing a small effect size). Together, these three factors accounted for 53% of the variance in overall residency performance score observed in this study.

CONCLUSION

Past clinical excellence, measured by core clerkship grades and exceptional letters of recommendation, is associated with slightly improved overall orthopaedic residency performance scores. Applicants meeting both criteria who also complete a research track residency may perform substantially better in residency than their counterparts, as these three factors accounted for half of all the variance observed in the current study. Although minimum requirements are necessary, traditionally used screening factors (such as USMLE scores, AOA status, medical school rank, and number of publications) may be of less utility in identifying successful future residents than previously thought.

LEVEL OF EVIDENCE

Level III, therapeutic study.

Standardized Letter of Recommendation: Can Everyone Be Awesome?

OBJECTIVE

Standardized letters of recommendation (SLOR) are hypothesized to decrease bias and provide consistent domains for evaluation. However, their ability to differentiate among applicants is unknown. The utilization and functionality of SLOR and the impact of SLOR domain rating on matching for colon and rectal surgery (CRS) residency applicants have yet to be assessed.

DESIGN

Descriptive statistics and bivariate analysis were employed. Applicants were categorized into 3 groups; Top-tier(TT): applicants rated 100% Excellent/Very Good; Mid-Tier(MT) applicants rated 80-99% Excellent/Very Good; and Non-Top Tier(NTT) applicants rated <80% Excellent/Very Good.

SETTING

University of Hospitals Cleveland Medical Center.

PARTICIPANTS

SLORs submitted to a single colorectal surgery residency in 2019 were analyzed RESULTS: A total of 101 applicants were included, 54 (53.5%) of the applicants were male. 75 (74.2%) applicants who applied to our residency matched into a CRS residency, compared to the national rate of 66%. Of the 101 applicants with SLOR, 54 (53.5%) were categorized as TT, 26 (25.7%) as MT, and 21(20.8%) as NTT. The univariable analysis demonstrated a statistically significant difference in research experience (p=0.029) and match status (p=0.01) between applicant tiers. There were no statistically significant differences between applicant-tier and demographics, foreign medical graduates (FMG), H-indices, ABSITE scores, type of residency, preliminary year, completing an unaccredited CRS, and applicants with an additional degree. On multivariable analysis age (OR=0.65; CI=0.48-0.87) and FMG applicants (OR=0.05; CI=0.01-0.44) were inversely associated with successfully matching. Compared to TT applicants, MT (OR=0.07; CI=0.01-0.57) and NTT (OR=0.04; CI=0.01-0.34) applicants were inversely associated with a successful match. Individuals who completed research prior to residency but after medical school was associated with successfully matching (p=0.009).

CONCLUSIONS

The presence of MT and NTT ratings is associated with failure to match and may represent an area of concern for CRS programs rather than a tool to discern differences between candidates.

Sex bias in letters of recommendation and personal statements for otolaryngology residency

Objective

Application for otolaryngology residency is highly competitive, with letters of recommendation (LORs) and applicant personal statements (PSs) representing important components of the application process. However, their inherently subjective nature predisposes them to potential implicit bias. Otolaryngology has historically been predominated by male physicians and while implicit sex bias has been demonstrated in LORs for application to residency of multiple specialties, data is limited for otolaryngology.

Methods

LORs and PSs for all otolaryngology applicants to an academic medical center during the 2019-20 and 2020-21 cycles were abstracted. Quantitative analysis was performed using Linguistic Inquiry and Word Count 2015 (LIWC2015), a validated software application designed to analyze various emotional, cognitive, and structural components of written text.

Results

LORs written for females were found to be written from a perspective of higher expertise and confidence while LORs written for males were associated with a more honest, personal, and disclosing tone. Moreover, LORs written for female applicants were found to reference achievement and "grindstone" terminology more than those written for men. No differences were observed in any word category between PSs written by male and female applicants.

Conclusion

Minor linguistic differences exist in multiple domains between LORs written for male and female applicants for otolaryngology residency. These tended to favor female applicants, with their letters demonstrating higher clout, achievement, and grindstone scores. This trend was unexpected in this historically predominantly male specialty. While differences were statistically significant, the overall difference in an entire letter of recommendation is likely subtle.

Level of Evidence

3.

Trends in utilization of standardized letters of recommendation in the 2021-2022 neurosurgery application process

OBJECTIVE

Standardized letters of recommendation (SLORs) were introduced during the 2020-2021 neurosurgery residency application cycle, but suffered from poor interrater reliability and grade inflation. Changes were made to the SLOR template and utilization patterns in response to these shortcomings. The authors examined the second year of SLOR utilization. They hypothesized that grade inflation and interrater reliability would be improved from the first iteration. They also hypothesized that increased numbers of letters by single writers would correlate with broader rating distributions.

METHODS

This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2021-2022 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of the letter writer. Interrater reliability was evaluated using Krippendorff's alpha. The frequency of letters written was evaluated using the Kruskal-Wallis H test.

RESULTS

Ninety percent of SLORs rated applicants among the top 25%, but there was a significant decrease in the usage of the top 1% and top 2%-5% ratings. Interrater reliability was poor across all competencies. Writers who completed 1 SLOR rated applicants higher and had a narrower range than those who completed multiple SLORs.

CONCLUSIONS

Changes in the format and subsequent utilization patterns of SLORs have slightly decreased grade inflation; however, interrater reliability remains poor. The most wide-ranging evaluators submitted the highest number of SLORs, suggesting that future evaluation and usage of SLORs should emphasize letter-writer characteristics and numbers of SLORs written. Overall, SLORs have been well and broadly accepted with subtle improvements in the second year of utilization.

Descriptors in Letters of Recommendation for Otolaryngology Residency Across Gender, Race, and Time

OBJECTIVE

To determine whether differences exist in the descriptors used in letters of recommendations for Otolaryngology - Head and Neck Surgery (OHNS) residency candidates, comparing race and gender as depicted in visual letters of recommendation (VLORs) from the 2014 and 2019 application cycles.

DESIGN

Four hundred thirty-three LORS (284 narrative LORs and 149 standardized LORs) and 63 medical student performance evaluations for 104 candidates who interviewed at the University of Cincinnati OHNS residency program in 2014 and 2019 were analyzed. Descriptors from LORs and medical student performance evaluations were collected by two reviewers and QSR NVivo 12 was used to generate a word cloud that grouped words by synonym and weighted them by frequency. Reviewers coded these synonyms into one of eight descriptor categories. Race and gender were self-reported from residency applications. The average of each category frequency for each race and gender were compared using student t-tests.

SETTING

University of Cincinnati OHNS Residency Program.

PARTICIPANTS

104 OHNS applicants that interviewed at the University of Cincinnati in 2014 and 2019.

RESULTS

Of the 104 candidates reviewed, 39 were female (37.5%). 66/104 (63%) of applicants identified as white, 31/104 (30%) as non-white, and 7/104 (7%) preferred not to say. No significant differences were found between male vs female descriptors. However, white applicants had more "leadership" descriptors (1.3% vs 0.5%, p = 0.01) and fewer "intelligence" words (6.6% vs 4.8%, p = 0.02) than non-white applicants. Applicants in 2019 were described with more team player (14.2% vs 9.6%, p < 0.0001), leadership (1.6% vs 0.7%, p = 0.047), and reserved (1.7% vs 0.7%, p = 0.02) words, but fewer grindstone (25.2% vs 32.1%, p < 0.0001) and ability/agentic (27.9% vs 32.6%, p < 0.0001) words than applicants in 2014.

CONCLUSIONS

In VLORs for OHNS residency, male and female applicants are described similarly, but white applicants are described as leaders more frequently and as intelligent less frequently than non-white counterparts. Regardless, it is encouraging to see applicants described with a more humanistic vocabulary in 2019.

Standardized Letters of Recommendation in Plastic Surgery: The Impact of Gender and Race

BACKGROUND

In the last few decades, standardized letters of recommendation have seen increasing popularity among various medical specialties and have been shown to demonstrate score inflation and gender bias in some select specialties. Through analysis of standardized letter of recommendation scoring patterns, this study evaluated the impact of race and gender on performance in standardized letters of recommendation and provides suggestions for programs to optimally interpret standardized letters of recommendation and minimize bias when writing letters of recommendation.

METHODS

Available standardized letters of recommendation were those from interviewed applicants to the integrated plastic surgery program between 2015 and 2018 (82 total applicants) and all applicants in the year of 2019 (241 total applicants). Various standardized letters of recommendation component scores were analyzed by demographics of both applicants and letter writers, including descriptive statistics, analysis of variance, t testing, and Fleiss kappa for interrater reliability using SPSS software (IBM, Armonk, N.Y.).

RESULTS

No scoring category had an average score below the eighty-fifth percentile. Male letter writers scored male applicants significantly higher in Overall, Conscientiousness, Self-Initiative, and Academic Performance components, whereas female letter writers scored female applicants higher in Communication and Overall components. In the 2019 cohort, applicants of a minority race received lower scores on average in nine out of 10 categories, a difference that was significant in the Team Player category.

CONCLUSIONS

Score inflation pervades standardized letters of recommendation scoring in plastic surgery. Work must be done by letter writers to help mitigate racial and gender biases that influence recommendations for resident selection in plastic surgery.

The ADVICER Template for Faculty Reviewer Letters for Promotion and Appointment

RATIONALE AND OBJECTIVES

Promotion is an important milestone in the career of academic radiologists. Appointments, Promotion and Tenure (APT) committees require multiple letters of support from both internal and external referees. Traditional narrative letters are highly subjective, have high inter-reader variability, are time-intensive, and vulnerable to gender and other biases. The Alliance of Directors and Vice Chairs of Education in Radiology (ADVICER) recognized the need for a standardized template to assist academic faculty, letter writers, and APT committees.

MATERIALS AND METHODS

An ADVICER ad hoc committee of six educators with experience serving as external referees was convened to create a standardized template. Committee members performed a search of the relevant literature and internet sites, spoke with stakeholders such as APT chairs, and ultimately developed a template for faculty reviewer letters using the common clinician-educator pathway as a focal point.

RESULTS

An open source, modifiable, standardized, template was produced. The template has been made available to ADVICER members and is available on the Association of University Radiologists (AUR) website at: https://www.aur.org/resources/Template-for-Faculty-Reviewer-Letters-for-Promotion-and-Appointment CONCLUSION: This external referee template has the potential to reduce subjectivity, eliminate bias, and provide a flexible, modifiable, comprehensive faculty review letter template which will be useful for academic faculty, letter writers, and promotions committees.

Compliance with CDIM-APDIM Guidelines for Department of Medicine Letters: an Opportunity to Improve Communication Across the Continuum

BACKGROUND

With rising applications to internal medicine programs and pending changes in United States Medical Licensing Examination Step 1 score reporting, program directors desire transparent data for comparing applicants. The Department of Medicine Letters of Recommendation (DOM LORs) are frequently used to assess applicants and have the potential to provide clearly defined data on performance including stratification of a medical school class. Despite published guidelines on the expected content of the DOM LOR, these LORs do not always meet that need.

OBJECTIVES

To better understand the degree to which DOM LORs comply with published guidelines.

METHODS

We reviewed DOM LORs from 146 of 155 LCME-accredited medical schools in the 2019 Match cycle, assessing for compliance with published guidelines.

RESULTS

Adherence to the recommendation for DOM LORs to provide a final characterization of performance relative to peers was low (68/146, 47%). Of those that provided a final characterization, 19/68 (28%) provided a quantitative measure, and 49/68 (72%) provided a qualitative descriptor. Only 17/49 (35%) with qualitative terms described those terms, and thirteen distinct qualitative scales were identified. Ranking systems varied, with seven different titles given to highest performers. Explanations about determination of ranking groups were provided in 12% of cases.

CONCLUSIONS

Adherence to published guidelines for DOM LORs varies but is generally low. For program directors desiring transparent data to use in application review, clearly defined data on student performance, stratification groupings, and common language across schools could improve the utility of DOM LORs.

Attitudes towards and impact of letters of recommendation for anesthesiology residency applicants

Background: This survey aims to identify the relative value and the critical components of anesthesiology letters of recommendation(LORs) from the perspective of Program Directors (PDs) and Associate/Assistant Program Directors (APDs). Knowledge and insights originating from this survey might add to the understanding of the anesthesiology residency selection process and mitigate unintended linguistic biases.Methodology: Anonymous online surveys were sent to anesthesiology PDs/APDs from the Accreditation Council for Graduate Medical Education (ACGME) accredited anesthesiology residency Programs in the USA (US), as listed on the ACGME website and the American Medical Association Fellowship and Residency Electronic Interactive Database (AMA FREIDA) Residency Program Database. The survey authors were blinded to the identity of the respondents.Results: 62 out of 183 (33.8%) invited anesthesiology PDs/APDs completed the survey anonymously. In our survey, LORs are reported as more important in granting an interview than in making the rank list. 64% of respondents prefer narrative LORs. 77.4% of respondents look for specific keywords in LORs. Keywords such as 'top % of students' and 'we are recruiting this candidate' indicate a strong letter of recommendation while keywords such as 'I recommend to your program' or non-superlative descriptions indicate a weak letter of recommendation. Other key components of LORs include the specialty of the letter-writer, according to 84% of respondents, with anesthesiology as the most valuable specialty. Although narrative LORs are preferred, 55.1% of respondents are not satisfied with the content of narrative LORs.Conclusion: LORs containing specific keywords play an important role in the application to anesthesiology residency, particularly when submitted by an anesthesiologist. While narrative LORs are still the preferred format, most of our respondents feel they need improvements. The authors suggest specific LOR improvements including creating formalized LOR training, adding a style guide, and applying comparative scales, with standardized vocabulary in the narrative LOR.

An Examination of Standardized Letters of Recommendation Rating Scales Among Neurosurgical Residency Candidates During the 2020-2021 Application Cycle

BACKGROUND

Letters of recommendation (LORs) are historically an important, though subjective, component of the neurosurgery residency application process. Standardized LORs (SLORs) were introduced during the 2020 to 2021 application cycle. The intent of SLORs is to allow objective comparison of applicants and to reduce bias.

OBJECTIVE

To examine the utility of SLORs during this application cycle. We hypothesized that "grade inflation" and poor inter-rater reliability, as described by other specialties using SLORs, would limit the utility of SLORs in their current form.

METHODS

This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2020 to 2021 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of letter writer. Inter-rater reliability was evaluated using Krippendorff's alpha.

RESULTS

One or more SLORs was submitted as part of 298 of 393 applications (76%). Approximately 58.3% of letters written by neurosurgery chairpersons rated a given applicant as being within the top 5% across all competencies. Approximately 44.4% of program director letters similarly rated applicants as amongst the top 5%, while 73.2% and 81.4% of letters by other neurosurgeons and general surgery evaluators, respectively, rated applicants in the top 5%. Inter-rater reliability was poor (<0.33) in all rating categories, including overall (α = 0.18).

CONCLUSION

The utility of the first iteration of SLORs in neurosurgery applications is undermined by significant "grade inflation" and poor inter-rater reliability. Improvements are necessary for SLORs if they are to provide meaningful information in future application cycles.

Characterizing Standardized Letters of Recommendation in Urology Residency Applications

OBJECTIVE

To identify the current formats of standardized letters of recommendation (SLORs) and evaluate their characteristics, the distribution of applicants' ratings, correlation between SLOR domain ratings and conventional application metrics, and potential biases.

METHODS

We evaluated all applications submitted to our residency program for the 2020-2021 urology match. Two main formats of SLOR were identified. We extracted application characteristics and SLOR domain ratings.

RESULTS

Ninety SLORs from 82 applicants were reviewed. Applicants were highly rated among top tiers in both formats. Some correlations were observed between domain ratings and application metrics such as Step 1 and Step 2 Clinical Knowledge scores, and percentage of Honors in core clinical clerkships. No statistically significant differences were found between female and male applicants in terms of domain ratings. Alpha Omega Alpha members received higher ratings in "urology resident potential," "academic urologist potential," and "performance as a sub-intern" domains. Applicants from top 40 US medical schools performed better as sub-interns, and were more likely to be ranked higher. Letters from home institutions were associated with higher ratings in several domains. In-person vs virtual interactions received similar ratings except for "communication".

CONCLUSION

While it is promising to observe such number of SLORs submitted for the first time in urology, the current formats could benefit from further refinement in their structures and domains to distinguish between highly qualified urology applicants more efficiently. Given the transition in Step 1 score reporting to pass/fail outcome, the need for a reliable urology-specific SLOR will be critical.

Characterization of the 2019 Micrographic Surgery and Dermatologic Oncology Standardized Letter of Recommendation

BACKGROUND

Traditional letters of recommendation used for postgraduate medical training applications have multiple limitations, including a lack of clarity, inflated and overly flattering assessments, and low reliability between interpreting faculty. A micrographic surgery and dermatologic oncology (MSDO) standardized letter of recommendation (SLOR) was created to improve the efficiency, validity, and stratification of applicants to dermatology fellowship training programs.

OBJECTIVE

To analyze the MSDO SLOR for trends in grading based on letter-writer and applicant characteristics and to evaluate its ability to demonstrate differences between applicants.

METHODS

Standardized letter of recommendations received by 4 fellowship programs from the 2019 San Francisco Match application cycle were reviewed retrospectively.

RESULTS

Two hundred forty-nine SLORs were analyzed from 140 applicants. Grade inflation and limited variability in scores were evident. Higher scores correlated with the length of the relationships between letter-writers and applicants and with female letter-writer gender. There was no applicant gender or ethnicity bias detected.

CONCLUSION

Despite score inflation, the MSDO SLOR allows for differentiation between fellowship applicants. Future studies are needed to further evaluate the reliability of the SLOR and find ways to improve its content.

Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation

BACKGROUND

The standardized letter of evaluation (SLOE) is the application component that program directors value most when evaluating candidates to interview and rank for emergency medicine (EM) residency. Given its successful implementation, other specialties, including otolaryngology, dermatology, and orthopedics, have adopted similar SLOEs of their own, and more specialties are considering creating one. Unfortunately, for such a significant assessment tool, no study to date has comprehensively examined the validity evidence for the EM SLOE.

OBJECTIVE

We summarized the published evidence for validity for the EM SLOE using Messick's framework for validity evidence.

METHODS

A scoping review of the validity evidence of the EM SLOE was performed in 2020. A scoping review was chosen to identify gaps and future directions, and because the heterogeneity of the literature makes a systematic review difficult. Included articles were assigned to an aspect of Messick's framework and determined to provide evidence for or against validity.

RESULTS

There have been 22 articles published relating to validity evidence for the EM SLOE. There is evidence for content validity; however, there is a lack of evidence for internal structure, relation to other variables, and consequences. Additionally, the literature regarding response process demonstrates evidence against validity.

CONCLUSIONS

Overall, there is little published evidence in support of validity for the EM SLOE. Stakeholders need to consider changing the ranking system, improving standardization of clerkships, and further studying relation to other variables to improve validity. This will be important across GME as more specialties adopt a standardized letter.

Gender bias in reference letters for residency and academic medicine: a systematic review

Reference letters play an important role for both postgraduate residency applications and medical faculty hiring processes. This study seeks to characterise the ways in which gender bias may manifest in the language of reference letters in academic medicine. In particular, we conducted a systematic review in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We searched Embase, MEDLINE and PsycINFO from database inception to July 2020 for original studies that assessed gendered language in medical reference letters for residency applications and medical faculty hiring. A total of 16 studies, involving 12 738 letters of recommendation written for 7074 applicants, were included. A total of 32% of applicants were women. There were significant differences in how women were described in reference letters. A total of 64% (7/11) studies found a significant difference in gendered adjectives between men and women. Among the 7 studies, a total of 86% (6/7) noted that women applicants were more likely to be described using communal adjectives, such as "delightful" or "compassionate", while men applicants were more likely to be described using agentic adjectives, such as "leader" or "exceptional". Several studies noted that reference letters for women applicants had more frequent use of doubt raisers and mentions of applicant personal life and/or physical appearance. Only one study assessed the outcome of gendered language on application success, noting a higher residency match rate for men applicants. Reference letters within medicine and medical education exhibit language discrepancies between men and women applicants, which may contribute to gender bias against women in medicine.

Systems-Level Reforms to the US Resident Selection Process: A Scoping Review

BACKGROUND

Calls to reform the US resident selection process are growing, given increasing competition and inefficiencies of the current system. Though numerous reforms have been proposed, they have not been comprehensively cataloged.

OBJECTIVE

This scoping review was conducted to characterize and categorize literature proposing systems-level reforms to the resident selection process.

METHODS

Following Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches of Embase, MEDLINE, Scopus, and Web of Science databases were performed for references published from January 2005 to February 2020. Articles were included if they proposed reforms that were applicable or generalizable to all applicants, medical schools, or residency programs. An inductive approach to qualitative content analysis was used to generate codes and higher-order categories.

RESULTS

Of 10 407 unique references screened, 116 met our inclusion criteria. Qualitative analysis generated 34 codes that were grouped into 14 categories according to the broad stages of resident selection: application submission, application review, interviews, and the Match. The most commonly proposed reforms were implementation of an application cap (n = 28), creation of a standardized program database (n = 21), utilization of standardized letters of evaluation (n = 20), and pre-interview screening (n = 13).

CONCLUSIONS

This scoping review collated and categorized proposed reforms to the resident selection process, developing a common language and framework to facilitate national conversations and change.

Improving the Standardized Letter of Recommendation in the Plastic Surgery Resident Selection Process

OBJECTIVE

The American Council of Academic Plastic Surgeons (ACAPS) Plastic Surgery Recommendation Form is a standardized letter of recommendation (SLOR) used in the plastic surgery resident selection process since 2012. This study aims to evaluate this form and use a multistep survey to optimize this SLOR.

DESIGN

A 16 question survey was sent via Survey Monkey to 510 ACAPS faculty. The results of this survey were followed up with a live-response survey at the ACAPS Spring Retreat. Consensus was used to modify and create a more optimal SLOR.

SETTING

This study was performed by the Division of Plastic and Reconstructive Surgery at the University of California, San Diego.

PARTICIPANTS

Survey respondents represented over five hundred plastic surgery faculty from a variety of subspecialties and academic programs nation-wide.

RESULTS

The response rate to the online survey was 36%. On a scale of 1 (not useful) to 5 (very useful), letter readers and letter writers found the SLOR 3.88 and 3.90 respectively. Most respondents believed that the SLOR should be used as an adjunct to the narrative LOR. Forty-two faculties participated as respondents to the live audience questions, although over a 100 faculty were present for the discussion. Most participants agreed that independent program applicants should have a separate SLOR form, with at least 1 filled out by the Program Director of the residents' primary board. The majority (61%) of respondents supported the addition of 1-line characteristics to each applicant quality, and the change to a percentile format for assessment.

CONCLUSIONS

Letters of recommendation are vital to the resident selection process, and thus it is incumbent upon fields of medicine to consciously work on improving how these letters are written. This study outlines a 2-step process by which a SLOR was evaluated and optimized for faculty in the field of plastic surgery.

Ideas and Innovations to Improve the Otolaryngology–Head and Neck Surgery Residency Application and Selection Process

Objective

To (1) summarize strategies proposed in the recent otolaryngology–head and neck surgery (OTO-HNS) literature for improving the residency application and selection process, (2) evaluate the effects of recently implemented changes to the OTO-HNS match, and (3) discuss recommendations for future changes to the OTO-HNS residency application and selection process.

Data Sources

PubMed, Medline Ovid database, and article reference lists.

Review Methods

A structured literature search was performed to identify current English language articles relating to the objectives of this study using the aforementioned data sources, in accordance with the Preferred Reporting Items for Systematic Reviews and Meta Analyses (PRISMA) guidelines. The search was limited to submissions published between January 1, 2015, and January 1, 2020.

Conclusions

Numerous proposals have been made for improving the otolaryngology residency application and selection process and addressing the competitive nature of the Match. These proposals include but are not limited to mandating a secondary essay, implementing consortia and early match processes, using a signaling system, conducting regional and web-based interviews, offering early engagement with interest groups, instituting a hard cap on applications, increasing costs of applying, counseling self-restraint to prospective applicants, and creating application filters.

Implications for Practice

As the volume of literature surrounding the OTO-HNS Match continues to increase, this review aims to provide a summary of past proposals and serve as a guide for possible future innovations. We propose 3 initiatives that may improve the residency application and selection process for both program and resident, with minimal impact to the current National Resident Matching Program (NRMP) Match structure.

Evaluating the Standardized Letter of Recommendation Form in Applicants to Orthopaedic Surgery Residency

INTRODUCTION

A standardized letter of recommendation (SLOR) form for orthopaedic surgery residency programs has recently been adopted for use, but it has not been scientifically evaluated. The purpose of this study is to investigate the usefulness of the SLOR form in the selection process.

METHODS

All SLOR forms submitted to our institution over a single application cycle were extracted and analyzed. The United States Medical Licensing Examination Step 1 scores, grades in clinical rotations, Alpha Omega Alpha status, and the number of publications were recorded for each applicant. Correlations were calculated with Spearman rho, and inter-rater reliability was evaluated by calculating intraclass correlation coefficients.

RESULTS

One thousand one hundred thirty-seven SLOR forms were analyzed for 513 applicants. There was substantial rank inflation with the SLOR form; the majority (92%) of applicants were rated as either ranked to match or in the top one-third of their rank list. Objective applicant factors such as grades and Step 1 scores demonstrated a very weak to nonexistent correlation with the summative rank (rho 0.07 to 0.13, P ≤ 0.012). Poor inter-rater reliability was observed with the intraclass correlation coefficient ranging from 0.22 to 0.33 (P < 0.001).

CONCLUSIONS

The usefulness of the SLOR form is limited by the very high ratings observed for all questions, and in particular, the final summative rank. Measures to reduce rank inflation must be implemented to improve the discriminant ability of the SLOR form, and if this cannot be accomplished, perhaps the form should be abandoned.

LEVEL OF EVIDENCE

Level III Retrospective.

When (Almost) Everyone is Above Average: A Critical Analysis of American Orthopaedic Association Committee of Residency Directors Standardized Letters of Recommendation

The American Orthopaedic Association introduced standardized letters of recommendations (SLORs) to improve on traditional letters of recommendations by “providing a global prospective on an applicant.” However, no study has defined the utilization of SLORs, the distribution of applicant ratings in SLORs, or the impact of sex, race, or degree of involvement between the letter writer and applicant on SLOR domain ratings.

Race- and Gender-Based Differences in Descriptions of Applicants in the Letters of Recommendation for Orthopaedic Surgery Residency

Background

Letters of recommendation (LOR) are an important component of trainee advancement and assessment. Examination of word use in LOR has demonstrated significant differences in how letter writers describe female and male applicants. Given the emphasis on increasing both gender and racial diversity among orthopaedic surgeons, we aimed to study gender and racial differences in LOR for applicants to orthopaedic surgery residencies.

Methods

All applications submitted to a single, academic orthopaedic residency program in 2018 were included. Self-identified gender and race were recorded. The LOR were analyzed via a text analysis software program using previously described categories of communal, agentic, grindstone, ability, and standout words. We examined the relative frequency of word use in letters for (1) male and female applicants and (2) white and underrepresented in orthopaedics (UiO) applicants, with the subgroup analysis based on whether standardized (using the American Orthopaedic Association template) or traditional (narrative) LOR were used.

Results

Two thousand six hundred twenty-five LOR were submitted for 730 applicants (79% men). Fifty-nine percent of applicants were self-identified as white, and 34% were self-identified as UiO. In traditional LOR, standout words (odds ratio [OR] 1.07; p = 0.01) were more likely to be used in letters for women compared with men, with no difference in any other word-use category. In standardized LOR, there were no gender-based differences in any word category. In traditional LOR, grindstone words (OR = 0.96; p = 0.02) were more likely to be used in letters for UiO than white applicants, whereas standout words (OR = 1.05; p = 0.04) were more likely to be used in letters for white candidates. In standardized LOR, there were no race-based differences in any word category use.

Conclusions

Small differences were found in the categories of words used to describe male and female candidates and white and UiO candidates. These differences were not present in the standardized LOR compared with traditional LOR. It is possible that the use of standardized LOR may reduce gender- and race-based bias in the narrative assessment of applicants.

The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail

Introduction

Interest is growing in specialty-specific assessments of student candidates based on clinical clerkship performance to assist in the selection process for postgraduate training. The most established and extensively used is the emergency medicine (EM) Standardized Letter of Evaluation (SLOE), serving as a substitute for the letter of recommendation. Typically developed by a program’s leadership, the group SLOE strives to provide a unified institutional perspective on performance. The group SLOE lacks guidelines to direct its development raising questions regarding the assessments, processes, and standardization programs employ. This study surveys EM programs to gather validity evidence regarding the inputs and processes involved in developing group SLOEs.

Methods

A structured telephone interview was administered to assess the input data and processes employed by United States EM programs when generating group SLOEs.

Results

With 156/178 (87.6%) of Accreditation Council of Graduate Medical Education-approved programs responding, 146 (93.6%) reported developing group SLOEs. Issues identified in development include the following: (1) 84.9% (124/146) of programs limit the consensus process by not employing rigorous methodology; (2) several stakeholder groups (nurses, patients) do not participate in candidate assessment placing final decisions at risk for construct under-representation; and (3) clinical shift assessments don’t reflect the task-specific expertise of each stakeholder group nor has the validity of each been assessed.

Conclusion

Success of the group SLOE in its role as a summative workplace-based assessment is dependent upon valid input data and appropriate processes. This study of current program practices provides specific recommendations that would strengthen the validity arguments for the group SLOE.

Current Trends in Orthopaedic Surgery Residency Applications and Match Rates

BACKGROUND

The current U.S. orthopaedic residency application process is becoming increasingly impersonal in the wake of an increasing number of applications. Through an analysis of orthopaedic surgery residency application statistics, we have highlighted the effect that the number of orthopaedic applications has on match rate, and we have suggested methods for a more personalized application process.

METHODS

Data from the Association of American Medical Colleges (AAMC) and the National Resident Matching Program (NRMP) for United States orthopaedic residency applicants from 2008 to 2018 were collected. These data included the average number of applications submitted per applicant, the average number of applications received per program, the total number of residency positions offered in the U.S., the total number of U.S. applicants, and the total number of U.S. applicants who matched to a U.S. orthopaedic surgery residency program. U.S. applicant match rates and the average number of applications received per residency position offered were calculated. Linear regression models were used to determine the rate at which these variables changed over time.

RESULTS

The average number of applications submitted by an applicant increased from 46.5 in 2008 to 74.9 in 2018. The average number of applications received per residency position offered increased from 54.1 in 2008 to 85.7 in 2018. The number of U.S. applicants was 740 in 2008 and 849 in 2018. The number of U.S. orthopaedic residency programs only slightly increased from 160 in 2008 to 171 in 2018. The match rate for U.S. medical school applicants has remained stable from 2008 to 2018 at a mean of 76.9% and a standard deviation of 2.3%.

CONCLUSIONS

The match rate has remained stable from 2008 to 2018 despite an increase in the number of applications per position. This discrepancy suggests that increasing the number of submitted applications may not correlate with applicant success. We address this discrepancy and suggest methods that can potentially allow for a more targeted orthopaedic application experience.

What Are We REALLY Looking for in a Letter of Recommendation?

OBJECTIVES

The letter of recommendation (LOR) is an important component of a residency application. There is great subjectivity in the interpretation of a letter writer's narrative and many residencies have moved toward standardized LORs (sLOR). We aimed to scrutinize the importance afforded to specific content and applicant attributes in an LOR in hopes of decoding this time-honored process.

DESIGN

A 35-question survey comprised of nonidentifying general questions, and participant evaluation of applicant characteristics and LOR phrases were administered (cross-sectional design). Evaluations were scored both objectively on a 10-point Likert scale and subjectively using a relative ranking.

SETTING

Academic, tertiary care center with a large general surgery residency program (Mayo Clinic, Rochester, MN).

PARTICIPANTS

Surgery attendings and general surgery residents (n = 122).

RESULTS

Seventy-two attendings and 50 general surgery residents completed the survey. Faculty ranked strong work ethic/hard working (median rank = 1) and inquisitive/hungry learner (median rank = 3) as the top 2 attributes. "We will plan to recruit this candidate" (median rank = 1.5) and "I give my highest recommendation" (median rank = 2) were the top 2 phrases. Residents valued strong work ethic and collaborative/team player as the top 2 applicant attributes. Seventy-three percent of attendings and 82% of residents agreed that LOR allows the applicant pool to be divided into upper and lower halves. Only 17% of faculty and 18% of residents agreed that an LOR allowed candidate stratification into quartiles.

CONCLUSIONS

Elaborating the most favorable applicant characteristics and highly regarded content in an LOR will help truly exceptional candidates obtain letters that make them stand out in the eyes of their evaluators. Since LORs are mostly considered to be able to stratify only upper and lower halves of the applicant pool, it is imperative to move toward LORs which portray superior applicant qualities, and can provide more objective evaluation of a candidate.

The USMLE Step 1 Pass/Fail Reporting Proposal: The APDR Position

BACKGROUND

The National Board of Medical Examiners (NBME) and the United States Medical Licensing Examination (USMLE) has convened a conference of "key stakeholders" on March 11-12, 2019 to consider reporting the results of the USMLE Step 1 as pass/fail.

DISCUSSION

While the original purpose of the USMLE Step 1 was to provide an objective basis for medical licensing, the score is increasingly used in residency applicant screening and selection because it is an objective, nationally recognized metric allowing comparison across medical schools in and outside the United States. Excessive reliance on the Step 1 score in the matching process has led to "Step 1 Culture" that drives medical schools to "teach to the test," increases medical student anxiety, and disadvantages minorities that have been shown to score lower on the USMLE Step 1 examination. The outsize role of the USMLE Step 1 score in resident selection is due to lack of standardization in medical school transcripts, grade inflation, and the lack of class standing in many summative assessments. Furthermore, the numeric score allows initial Electronic Residency Application Service filtering, commonly used by programs to limit the number of residency applications to review.

CONCLUSION

The Association of Program Directors in Radiology (APDR) is concerned that pass/fail reporting of the USMLE Step 1 score would take away an objective measure of medical student's knowledge and the incentive to acquire as much of it as possible. Although the APDR is not in favor of the Step 1 exam being used as a screening tool, in the absence of an equal or better metric for applicant comparison the APDR opposes the change in Step 1 reporting from the numeric score to pass/fail.

Single Day Observational Experience at High Volume Ablation Programs: What is the Impact to Practicing Electrophysiologists?

Introduction

Significant improvements in catheter technology, electro-anatomic (EA) mapping and techniques to reduce fluoroscopy during radiofrequency ablation (RFA) of atrial fibrillation (AF) are on-going.However, few educational opportunities are available post fellowship for Electrophysiologists (EPs) to understand and integrate them into their practice, preventing wide spread adoption. The impact of observational learning for adoption of new technologies and techniques in the field of cardiac electrophysiology has not been studied. We sought to report the impact of a visit to a high-volume center with experience in new technologies and fluoroscopy reductionto the clinical practice of EPs.

Methods

Between 8/2014 and 10/2017 a total of 150 EPs visited 3 hospitals that perform a high volume of AF RFAs. EPs observed a minimum of 4 RFAs, primarily AF. AF RFAs were performed without fluoroscopy, using Carto 3 Version 4 (Biosense Webster) and intracardiac Echocardiography. There was ample interaction and discussion between hosting and visiting EP.

Results

73 EPs (48.6% of visitors) completed an electronic survey after the visit. The majority reported a significant reduction in fluoroscopy (>50%) and procedure (>20%).68% adopted a rigorous workflow and reported an increase in their confidence level with intracardiac echo (79%), continuous mapping (52%) and the Visitag module (61%).

Conclusions

Observational experience can have an immense impact on the clinical practice of EPs. Further effort should be devoted to such programs and to study in a more systematic way their ultimate impact on patient care.

Are Standardized Letters of Recommendation in Residency Applications Correlated With Objective Data?

OBJECTIVE

To evaluate whether the standardized letter of recommendation (SLOR), commonly used for medical students applying to otolaryngology residency, correlates with objective data in the application.

BACKGROUND

Standardized letters of recommendation using Likert-type scales for different attributes are commonly used by evaluators because of their high interrater reliability and efficiency in preparation and interpretation. Given that these are subjectively scored, it is unknown how well these correlate with objective data.

METHODS

Applications to the University of Cincinnati otolaryngology residency were evaluated in the academic cycle of 2017-2018. Standardized letters of recommendation were scored to determine whether certain attributes were correlated with objective data (United States Medical Licensing Examination [USMLE] scores and number of presentations/publications) provided in their application. Spearman correlations were used to evaluate the strength of the relationship between the subjective score in certain attributes with objective data.

RESULTS

There were 217 applications to the University of Cincinnati that contained SLORs. Of these applications, 474 standardized letters were scored in categories of medical knowledge, research, and commitment to academic medicine. Total publications and presentations were weakly correlated with commitment to academic medicine (0.35, P < .0001, n = 369) and with research (0.44, P < .0001, n = 355). Medical knowledge was weakly correlated with Step 1 scores (0.20, P < .0001) and Step 2 scores (0.18, P = .0002).

CONCLUSIONS

Subjective research and commitment to academic medicine rating scores were weakly correlated with greater academic productivity. Similarly, medical knowledge scores were weakly correlated with Step 1/2 scores. Further research may be needed to assess how to interpret SLOR scores in addition to the information available in an otolaryngology application.

LEVEL OF EVIDENCE

4.

Using word clouds to Re-envision letters of recommendation for residency applicants

OBJECTIVES/HYPOTHESIS

To develop a "word cloud"-based visual letter of recommendation (VLOR) and to evaluate its efficiency in discerning applicant quality compared to narrative letters of recommendation (NLORs).

STUDY DESIGN

Cross-sectional cohort study.

METHODS

NLORs for 48 otolaryngology residency applicants interviewed from the 2016 application cycle were identified and mined for descriptive terms to generate a word cloud, referred to as a VLOR. Eight individuals reviewed and rated a total of 187 blinded NLORs and 48 VLORs on a four-point scale (negative to exceptional). Median VLOR and NLOR scores and the time to review for each candidate were compared using the Wilcoxon signed rank test.

RESULTS

It took significantly more time to review the NLORs in comparison to the VLORs (67 seconds, interquartile range [IQR]: 41-98 seconds vs. 17 seconds, IQR: 11-26 seconds, P < .001). There was no significant difference between median scores for VLORs and NLORs (P = .136). Review time and score correlated positively for VLORs and was statistically significant (ρ = 0.459, P = .001), indicating that more time spent reviewing equates to higher scores. The same relationship appeared with NLORs, but was not statistically significant (ρ = 0.276, P = .058).

CONCLUSIONS

VLORs are a novel and efficient additive tool for screening candidates for otolaryngology residency interview slots. Their scores do not significantly vary from NLOR scores and are significantly faster to evaluate.

LEVEL OF EVIDENCE

2b Laryngoscope, 129:2026-2030, 2019.

Use of Standard Guidelines for Department of Medicine Summary Letters

Phenomenon: Fourth-year medical students obtain Department of Medicine (DOM) letters ("Chair" letters) to support their residency applications. Writing and interpreting DOM letters are challenging. There is heterogeneity in the letters that makes it difficult to both write and read these letters.

APPROACH

The purpose of this study is to determine the value of new guidelines developed by a task force of clerkship directors and program directors in internal medicine and assess the implementation of these guidelines. The Clerkship Directors in Internal Medicine performed a cross-sectional survey of internal medicine clerkship directors at U.S. and Canadian medical schools in 2014. In addition, the primary author's institution reviewed 1,347 DOM letters between 2012 and 2014 to assess the implementation of these guidelines.

FINDINGS

The survey response rate was 78%. DOM letter writers reported the guidelines were better, easier to implement, and more compatible with the purpose of DOM letters than previously. Most letter readers reported that letters using the guidelines were more credible. Writers of DOM letters in lower academic ranks rated the letters with guidelines higher in several domains than those in higher academic ranks. Readers of DOM letters in higher academic ranks rated the letters with guidelines higher in several domains than those in lower academic ranks. In the DOM letters examined, the odds of meeting each guideline increased with each additional year. However, for 3 guidelines there was an initial decline in adherence from 2012 to 2013 before increasing again in 2014-the recommended length, clerkship description, and detailed narrative guidelines. Letters solely written by a chair were less likely to incorporate the guidelines. Insights: Clerkship directors often write the DOM letters and identify with the purpose of the guidelines. As writers, lower ranking academic faculty value the guidelines more than higher ranking academic faculty. As readers of DOM letters, higher academic ranking faculty value letters that incorporate the guidelines more than lower academic ranking faculty. DOM letters implemented more guideline criteria since the guidelines were released. If implementing the guidelines, chairs should solicit the help of their clerkship director or educational representative when writing DOM letters. Although many clerkship directors read letters for their residency programs, additional program directors' opinions are needed.

Letters of Recommendation: Association with Interviewers' Perceptions and Preferences

Objective Letters of recommendation (LORs) are an inescapable aspect of the application process. Standardized LORs (SLORs) have been developed and compared with traditional narrative LORs (NLORs). This study investigated whether there was a difference in degree of association between LOR types and face-to-face interviews. Interviewer preference for LOR was anonymously surveyed. Study Design Survey. Setting Single-institution otolaryngology residency program, 1 year before and 4 years after introduction of SLORs. Subjects and Methods Residency interviewers indicated on a visual analog scale how well their impression of an applicant compared between the LOR and the face-to-face interview. Interviewers assessed each applicant, each year, based on LOR type. Results Of 2573 assessments, 964 were collected (37.5% response rate), including 927 responses related to NLORs, 561 to SLORs, and 316 to medical student performance evaluations (ie, dean's letters). The average association of VAS scores between LORs and interviews ranged from 72 to 81 across years and LOR types. Sixty-one percent of interviewers preferred NLORs, and 13% preferred SLORs. Reasons for these preferences included more information provided in NLORs versus faster read time with SLORs. Conclusions Interviewers' perceptions of applicants based on LORs and face-to-face interviews were comparable across LOR types over the last 5 years. For many reasons, the general utility of the LOR remains questionable, and the continued need for it should be critically assessed.

Improving Otolaryngology Residency Selection Using Principles from Personnel Psychology

There has been a heightened focus on improving the resident selection process, particularly within highly competitive specialties. Previous research, however, has generally lacked a theoretical background, leading to inconsistent and biased results. Our recently published systematic review examining applicant characteristics and performance in residency can provide historical insight into the predictors (ie, constructs) and outcomes (ie, criteria) previously deemed pertinent by the otolaryngology community. Personnel psychology uses evidence-based practices to identify the most qualified candidates for employment using a variety of selection methods. Extensive research in this discipline has shown that integrity tests, structured interviews, work samples, and conscientiousness offer the greatest increase in validity when combined with general cognitive ability. Blending past research knowledge with the principles of personnel selection can provide the necessary foundation with which to engage in theory-driven, longitudinal studies on otolaryngology resident selection moving forward.

Surgical Residency Recruitment-Opportunities for Improvement

INTRODUCTION

The Association of Program Directors in Surgery convened a panel during Surgical Education Week 2016 to discuss the current state of the general surgery residency application process and to review alternative ways to evaluate the suitability of each applicant to a residency program.

METHODS/RESULTS

Over 40,000 applicants registered for the National Resident Matching Program's 2016 Main Residency Match. General Surgery had 2345 applicants for 1241 categorical postgraduate year (PGY)-1 positions, and 1239 of those positions were filled when the matching algorithm was processed. Program Directors reported that only 33% of applications received an in-depth review, and 62% were rejected with minimal review. Eventually (after all applications had been reviewed), only 13% of applicants were invited to interview.

CONCLUSIONS

There are several opportunities for improvement within the current application process. These included standardized letter of recommendation and personal statements, refinement of the interview process, and recalibration of the Medical Student Performance Evaluation.

Improving the Medical Student Performance Evaluation to Facilitate Resident Selection

The Medical Student Performance Evaluation (MSPE) was introduced as a refinement of the prior "dean's letter" to provide residency program directors with a standardized comprehensive assessment of a medical student's performance throughout medical school. The author argues that, although the MSPE was created with good intentions, many have questioned its efficacy in predicting performance during residency. The author asserts that, despite decades of use and some acknowledged improvement, the MSPE remains a suboptimal tool for informing program directors' decisions about which applicants to interview and rank. In the current approach to MSPEs, there may even be some inherent conflicts of interest that cannot be overcome. In January 2015, an MSPE Task Force was created to review the MSPE over three years and recommend changes to its next iteration. The author believes, however, that expanding this collaborative effort between undergraduate and graduate medical education and other stakeholders could optimize the MSPE's standardization and transparency. The author offers six recommendations for achieving this goal: developing a truly standardized MSPE template; improving faculty accountability in student assessment; enhancing transparency in the MSPE; reconsidering the authorship responsibility of the MSPE; including assessment of compliance with administrative tasks and peer assessments in student evaluations; and embracing milestones for evaluation of medical student performance.