Attitudes towards and impact of letters of recommendation for anesthesiology residency applicants

Machine Learning for Predictive Analysis of Otolaryngology Residency Letters of Recommendation

INTRODUCTION

Letters of recommendation (LORs) are a highly influential yet subjective and often enigmatic aspect of the residency application process. This study hypothesizes that LORs do contain valuable insights into applicants and can be used to predict outcomes. This pilot study utilizes natural language processing and machine learning (ML) models using LOR text to predict interview invitations for otolaryngology residency applicants.

METHODS

A total of 1642 LORs from the 2022-2023 application cycle were retrospectively retrieved from a single institution. LORs were preprocessed and vectorized using three different techniques to represent the text in a way that an ML model can understand written prose: CountVectorizer (CV), Term Frequency-Inverse Document Frequency (TF-IDF), and Word2Vec (WV). Then, the LORs were trained and tested on five ML models: Logistic Regression (LR), Naive Bayes (NB), Decision Tree (DT), Random Forest (RF), and Support Vector Machine (SVM).

RESULTS

Of the 337 applicants, 67 were interviewed and 270 were not interviewed. In total, 1642 LORs (26.7% interviewed) were analyzed. The two best-performing ML models in predicting interview invitations were the TF-IDF vectorized DT and CV vectorized DT models.

CONCLUSION

This preliminary study revealed that ML models and vectorization combinations can provide better-than-chance predictions for interview invitations for otolaryngology residency applicants. The high-performing ML models were able to classify meaningful information from the LORs to predict applicant interview invitation. The potential of an automated process to help predict an applicant's likelihood of obtaining an interview invitation could be a valuable tool for training programs in the future.

LEVEL OF EVIDENCE

N/A Laryngoscope, 134:4016-4022, 2024.

Do certain phrases in letters of recommendation predict success in fellowship performance?

BACKGROUND

Letters of recommendation (LORs) are a valued, yet imperfect tool. Program directors (PDs) score phrases such as give my highest recommendation and top 5 to 10% of students as positive. Although positive phrases are valued by PDs, there is no evidence that these phrases predict performance. We attempt to identify whether 12 specific phrases found in letters of recommendation predict future performance of fellows.

METHODS

LORs were evaluated for 12 select phrases and statements. Alpha Omega Alpha (AOA) status, Step 2 Clinical Knowledge (CK) score, and whether the letter writer was personally known to our admission's committee were also categorized. Logistic regressions were performed to evaluate the relationship of the independent variables with fellow performance.

RESULTS

Using multivariate logistic regression, one of the best residents (OR = 4.02, 95% CI (1.0, 15.9), p < 0.05), exceeds expectations (OR = 4.74, 95% CI (1.4, 16.3), p = 0.01), and give my highest recommendation (OR = 3.87, 95% CI (1.3, 11.7), p = 0.02) predicted positive performance. Highly recommend (OR = 0.31, 95% CI (0.1, 1.0), p < 0.05) and top 5 to 10% (OR = 0.05, 95% CI (0.0, 0.6), p = 0.02) predicted negative performance. The remaining phrases did not correlate to fellowship performance.

CONCLUSION

The current LOR evaluation process may place undo importance on phrases that have limited bearing on a candidate's success in training. Training both letter readers and writers to avoid using coded language or avoid assigning improper importance to select phrases may help improve the candidate selection process.

Do Program Directors of Anesthesiology Residency Programs Interpret Narrative Letters of Recommendation as Intended?

Background Letters of recommendation (LORs) are an important part of the application process for medical residency programs with most specialties preferring a narrative format. Given the inherent subjectivity of narrative LORs, the current study sought to determine whether the intended messages of narrative LORs written for applicants to anesthesiology residency programs are accurately interpreted by readers. Methodology Anonymous online surveys were sent via the Qualtrics platform to program directors (PDs) of the Accreditation Council for Graduate Medical Education-accredited anesthesiology residency programs in the Mid-Atlantic region as designated by the Electronic Residency Application Service, which consists of the states of New York, Pennsylvania, and New Jersey. Each PD participant received five surveys, each of which was attached to a de-identified LOR that was written by another PD located at an institution in the same region. Both the letter writer and study participants were asked to score LORs on a Likert-like scale. Participants were additionally asked whether the LORs, if received, would influence their decision to either offer an interview to the applicant or to rank the applicant. Finally, participants were asked to note any specific words or phrases within the LORs that they found to be particularly impactful. Results Overall, 10 of 34, 29.41%, PDs responded to the survey. There was a high correlation between the LOR intent and the respondents' assigned rating (Spearman's rho = 0.7973, p < 0.001). Responses were more accurate for "outstanding and excellent" LORs compared to the lower three categories. Results were unaffected after adjusting for respondents' years of experience as PDs. Additionally, 71.6% indicated that the LORs would influence the decision about offering an interview, and 56.5% stated that the LORs would influence a ranking decision. Conclusions Our results indicate that respondents' perception of LORs correlated strongly with the intent of the writer. Additionally, respondents seemed to value LORs for interview and ranking decisions.

Identification, Characterization, and Ranking of Candidate Metrics for Selection to Anesthesiology Residency: An Iterative Survey of Program Directors

Background

The primary aim of this study was to identify and stratify candidate metrics used by anesthesiology residency program directors (PDs) to develop their residency rank lists through the National Resident Matching Program.

Methods

Sixteen PDs comprised the participants, selected for diversity in geography and program size. We used a 3-round iterative survey to identify and stratify candidate metrics. In the first round, participants listed metrics they planned to use to evaluate candidates. In the second round, metrics from the first round were ranked by importance, and criteria were solicited to define an exceptional, strong, average, marginal, and uncompetitive candidate for each metric. In the third round, aggregated results were presented and participants refined their rankings.

Results

Of the 16 PDs selected, 15 participated in the first and second survey rounds, and 10 in the third. Eighteen candidate metrics were indicated by 8 or more PDs for residency selection. All 10 PDs from the final round identified passing Step 1 of the United States Medical Licensing Exam (USMLE) and the absence of "red flags" like a failed rotation as key selection metrics, both averaging an importance score of 4.9 out of 5. Other metrics identified by all PDs included clerkship evaluation comments, USMLE Step 2 scores, class rank, letters of recommendation, personal statement, and program and geographical signals.

Conclusions

The study reveals key metrics anesthesiology residency PDs use for candidate ranking, which may offer candidates insights into their competitiveness for anesthesiology residency.

Gender and Language in Letters of Recommendation for Obstetrics and Gynecology Fellowship Applications

OBJECTIVE

To delineate the use of gender-biased language in letters of recommendation for Obstetrics and Gynecology fellowships and its impact on applicants.

DESIGN

Fellowship letters of recommendation from 4 Obstetrics and Gynecology specialties at a single institution in 2020 were included.

PRIMARY OUTCOME

frequency of agentic and communal language in letters of recommendation using Linguistics Inquiry Word Count software.

SECONDARY OUTCOMES

letter of recommendation length and language utilization by author gender and applicant success measured by interviews and match success. Marginal models were fit to determine if language varied by applicant and writer gender and subspecialty. Modified Poisson regression models were used to determine associations between language and interview receipt.

SETTING

Single academic institution (Duke University); 2020 OB/GYN fellowship application cycle.

PARTICIPANTS

A total of 1216 letters of recommendation submitted by 326 unique applicants for OB/GYN subspecialty fellowships at our institution.

RESULTS

Rates of gender-biased language were low (Agentic:1.3%; communal: 0.8%). Agentic term use did not vary by applicant or author gender (p = 0.78 and 0.16) Male authors utilized 19% fewer communal terms than females (p < 0.001). Each 0.25% increase in agentic language was associated with an 18% reduction in the probability of interview invitation at our institution (p = 0.004). Percentage of agentic or communal language was not associated with successful matching into any subspecialty.

CONCLUSIONS

No differences in agentic vs communal language based on applicant gender were found in this cohort, though female letter writers wrote longer letters with more communal terms. Increasing agentic terms negatively impacted interview invitation but did not affect successful matching.

Augmented Web-based applicant experiences for anesthesiology away audition rotations: A systematic review of the current literature

Objective

The objective of this paper is to evaluate Augmented Web-based Applicant Experiences in the field of Anesthesiology. We hope to advocate for Augmented Web-based Applicant Experiences rotations and address their limitations in case in-person rotations become restricted again.

Methods

A thorough search of articles from the years 1978 to 2023 was completed using the search phrases "Away Rotations Anesthesiology," "Away Rotations Anesthesia," "Audition Rotation Anesthesia" and "Audition Rotation Anesthesiology." These search phrases were used on both the PubMed (Medline) and Excerpta medica database (EMBASE) databases, and the number of total articles that appeared was 73. These articles were then filtered to gather relevant articles for our study.

Results

After the articles were filtered, there was one remaining article that was used for our study. To supplement this article, 29 additional sources were added using the worldwide web. However, these did not address rotations in Anesthesiology. Although these other sources did not involve Anesthesiology residencies, they still provide significant perspectives that can be applied to online rotations. A total of 30 sources were used for our manuscript.

Conclusion

Although in-person rotations are preferred, Augmented Web-based Applicant Experiences rotations are valuable as they give residency programs an opportunity to evaluate students. Augmented Web-based Applicant Experiences rotations also provide students with learning opportunities and also help familiarize them with the residency programs. There are limitations that come with Augmented Web-based Applicant Experiences rotations. However, addressing these shortcomings can help Augmented Web-based Applicant Experiences rotations become a proper substitution for in-person rotations if they become restricted again.

Predictors of Matching into Anesthesiology and Surgery: Analysis of One Program's Results

PURPOSE

The objectives of this study were to use a multivariable regression model to determine what application factors made anesthesiology and surgery applicants more or less likely to match into an anesthesiology or surgery residency program.

METHODS

Surgery and Anesthesiology applicants listed on the final National Resident Matching Program (NRMP) Rank Order Lists from WMC in the 2020-2021 application cycle were included in analysis. All applicant data were collected through the Electronic Residency Application Service (ERAS). All ERAS and letters of recommendation (LOR) data were deidentified and LOR were subsequently inputted into a linguistics software to analyze the language use in LOR. Descriptive analyses were conducted to compare variables between applicants that matched to a specific residency program and those who matched elsewhere. A multivariable regression model was then used to determine characteristics of anesthesiology and surgery applicants that were indicative of matching to a specific rank of residency program.

RESULTS

A total of 116 anesthesiology and 78 surgery applicants were included in final analysis. Analysis of anesthesiology applicants yielded four significant application characteristics that influenced matching to a higher or lower ranked residency program: USMLE Step 2 CK scores, medical school attended, insight category words in LOR, and anger category words in LOR. Similarly, analysis of surgery applicants yielded four significant characteristics: Race, USMLE Step 1 scores, insight category words, and see category words.

CONCLUSION

Our results demonstrated that specialties of anesthesiology and surgery considered different metrics regarding the residency application process. Among the many factors that were analyzed, USMLE scores and language in LOR were considered significant in both specialties. As the application process continues to evolve, we may see a shift in what application factors are considered more important than others.

Virtual Residency Interviews- A Survival Guide and Lessons Learnt

With the outbreak of Coronavirus Disease 2019 (COVID-19) pandemic, the residency programs were required to conduct virtual/online interviews for recruitment of candidates for their residency programs. While both programs and the candidates had challenges, with the abrupt transition of interviews to online platform, there were some perceived benefits by the applicants. This paper will further review the pros and cons, the challenges and the changes brought about by the online transformation of residency interviews and conclude with tips to the residency applicants and lessons learnt from this transition. Although, residency programs are considering going back to in-person interviews, they may continue to offer virtual interview as well to the candidates in the future.

The Utility of the Plastic Surgery Standardized Letter of Recommendation Form in Predicting Residency Match Outcomes

BACKGROUND

Letters of recommendation play an important role in resident selection. While plastic surgery's Standardized Letter of Recommendation (SLOR) form most commonly serves as an adjunct to narrative letters, the SLOR provides objective data in the review process and could eventually replace narrative letters. The utility of the SLOR in predicting Match outcomes has not been studied.

METHODS

Applicant data from 225 first-time residency applicants in 2020-21 were collected. Logistic regression modeling was used to predict Match outcomes. This model was validated using 100 randomly selected applicants from 2019-20.

RESULTS

Rank placement (SLOR Question 6) was the most important factor when predicting Match outcomes (p<0.0001). All other SLOR questions were not predictive and subject to notable score inflation. No SLOR score differences were noted based on race; female applicants were rated higher in two of ten domains (p<0.05).

CONCLUSIONS

One question on the plastic surgery SLOR was highly predictive of an applicant matching. However, the remaining SLOR questions had little utility and were subject to gross score inflation. Further work should be done to optimize the utility of the SLOR in differentiating applicants. This has important implications in ensuring the selection of professional, competent residents according to the aims of the Accreditation Council of Graduate Medical Education.

How to Write Powerful Letters of Recommendation

Nurses at all levels may be asked to write a letter of recommendation to support colleagues' and students' career and academic advancement. This article outlines key steps and considerations in the letter-writing process.

Applicant Selection for Anesthesiology Residency Programs in Saudi Arabia

Background

Anesthesiology is a medical specialty that involves pre, intra, and postoperative surgical and medical procedures; it is a profession shaped by the clinician’s medical knowledge and manual dexterity. To date, very few studies have addressed the selection criteria and factors associated with the applicant selection process for anesthesia residency programs in Saudi Arabia.

Objectives

We aimed to define the criteria, factors, and guidelines for candidate selection in anesthesia residency programs in Saudi Arabia.

Methodology

This was a cross-sectional study conducted using electronic questionnaires that were distributed to anesthesiology program directors in Saudi Arabia via email. The questionnaire was divided into six sections, and each section included various parameters such as demographic data, cognitive/academic activities, non-cognitive/non-academic activities, individual qualities, and red flags or negative factors of the individual. The participants rated each parameter, and the collected data were analyzed for statistical significance (p≤0.05).

Results

A total of 28 programs were included in this survey. All 14 parameters associated with individual qualities were found to be significantly important for applicant selection (p≤0.05). Except for delayed entry into residency after graduation, all 12 parameters associated with red flags or negative characteristics of individuals were significant for candidate selection (p≤0.05).

Conclusion

The results showed that academic/cognitive factors and non-academic/non-cognitive factors, along with the individual characteristics of the applicant, were given priority when selecting candidates for anesthesiology residency programs in Saudi Arabia.

Increasing Clinician-Scientist Workforce Diversity through the National Institute of General Medical Sciences’ Medical Scientist Training Program

The National Institute of General Medical Sciences Medical Scientist Training Program (MSTP) has been successful in producing clinician-scientists, with a majority of graduates pursuing research-related careers. However, there are a number of areas of continuing concern for the program. In particular, women and individuals from certain racial and ethnic backgrounds remain persistently underrepresented in MSTPs relative to the average college-aged U.S. population and to students receiving life sciences bachelor’s degrees. The authors, who include leaders of NIGMS, identify a number of challenges and opportunities for enhancing diversity, equity and inclusion in the MSTPs and suggest strategies for addressing them.

Relative Importance of Applicant Characteristics in Ophthalmology Residency Interview Selection: A Survey of Program Directors

Introduction  Ophthalmology residency positions remain competitive. A lack of clarity regarding which residency selection criteria are prioritized by program directors can heighten the stress associated with the match process. While surveys of program directors in several other medical specialties have been conducted to identify the most important residency selection criteria, there is limited data on selection criteria used by ophthalmology residency program directors. The purpose of our study was to survey ophthalmology residency program directors to identify the current state of interview selection decisions-the factors currently considered most important in determining whether to extend an interview invitation to residency applicants. Methods  We developed and distributed a Web-based questionnaire to all U.S. ophthalmology residency program directors. Questions evaluated program demographics and the relative importance of 23 different selection criteria used by ophthalmology residency program directors when evaluating applicants for residency interviews (Likert scale 1-5, with 1 being "not important" and 5 being "very important"). Program directors were also asked to identify the one factor they felt was most important. Results  The overall residency program director response rate was 56.5% (70/124). The selection criteria with the highest average importance scores were core clinical clerkship grades (4.26/5) followed by letters of recommendation (4.06/5), and United States Medical Licensing Examination (USMLE) Step 1 score (4.03/5). The most frequently cited single most important factor for interview selection was core clinical clerkship grades (18/70, 25.7%), with USMLE Step 1 score (9/70, 12.9%) and rotations at the program director's department (6/70, 8.6%) also commonly reported. Conclusion  Our results suggest that core clinical clerkship grades, letters of recommendation, and USMLE Step 1 scores are deemed the most important selection criteria by ophthalmology residency program directors as of a 2021 survey. With changes in clerkship grading for many medical schools and changes in national USMLE Step 1 score reporting, programs will face challenges in evaluating applicants and the relative importance of other selection criteria will likely increase.

New Heuristics to Stratify Applicants: Predictors of General Surgery Residency Applicant Step 1 Scores

OBJECTIVE

In 2022, United States Medical Licensing Examination (USMLE) Step 1 scores will become pass/fail. This may be problematic, as residency programs heavily rely on USMLE Step 1 scores as a metric when determining interview invitations. This study aimed to assess candidate application metrics associated with USMLE Step 1 scores to offer programs new cues for stratifying applicants.

DESIGN

Retrospective cohort study analyzing interviewed applicants to one general surgery residency program in 2019 and 2020. Applicant data analyzed included USMLE Step 1 scores, number of publications, clerkship scores, letter of recommendation scores (out of 2, scored by 0.25 interval), interview overall score (out of 5, scored by integer level), and standardized question score (out of 10). Each year, applicant's answers to one standardized behavioral question during their interview were scored by interviewers.

SETTING

Tertiary medical center, academic general surgery residency program.

PARTICIPANTS

Interviewed applicants at one general surgery residency program whose applications contained complete demographic information (203 out of 247).

RESULTS

Multiple Linear Regression revealed that higher surgical clerkship (β = 0.19, p = 0.006) and higher standardized interview question (β = 0.32, p < 0.001) scores were positively associated with applicant USMLE Step 1 score (F[7, 195] = 6.61, p < 0.001, R2 = 0.19). Letter of recommendation score, number of peer reviewed publications, gender, race, and applicant type (preliminary/categorical) were not associated with USMLE Step 1 scores.

CONCLUSIONS

With USMLE Step 1 scores transitioning to pass/fail, surgical residency programs need new selection heuristics. Surgery clerkship scores and standardized behavioral questions answered by applicants prior to the interview could provide a holistic view of applicants and help programs better stratify candidates without USMLE Step 1 scores.