The Standardized Letter of Evaluation Narrative: Differences in Language Use by Gender

Decoding the Reference Letter: Strategies to Reduce Unintentional Gender Bias in Letters of Recommendation

Introduction

There is a growing body of literature on gender bias in letters of recommendation (LORs) in academic medicine and the negative effect of bias on promotion and career advancement. Thus, increasing knowledge about gender bias and developing skills to mitigate it is important for advancing gender equity in medicine. This workshop aims to provide participants with knowledge about linguistic bias (focused on gender), how to recognize it, and strategies to apply to mitigate it when writing LORs.

Methods

We developed an interactive 60-minute workshop for faculty and graduate medical education program directors consisting of didactics, reflection exercises, and group activities. We used a postworkshop survey to evaluate the effectiveness of the workshop. Descriptive statistics were used to analyze Likert-scale questions and a thematic content analysis for open-ended prompts.

Results

We presented the workshop four times (two local and two national conferences) with one in-person and one virtual format for each. There were 50 participants who completed a postworkshop survey out of 74 total participants (68% response rate). Ninety-nine percent of participants felt the workshop met its educational objectives, and 100% felt it was a valuable use of their time. Major themes described for intended behavior change included utilization of the gender bias calculator, mindful use and balance of agentic versus communal traits, closer attention to letter length, and dissemination of this knowledge to colleagues.

Discussion

This workshop was an effective method for helping participants recognize gender bias when writing LORs and learn strategies to mitigate it.

Identifying Gender and Racial Bias in Pediatric Fellowship Letters of Recommendation: Do Word Choices Influence Interview Decisions?

OBJECTIVES

To describe linguistic differences in letters of recommendation (LORs) for pediatric fellowship candidates based on applicant and letter writer demographics and to examine if these differences influenced the decision to interview a candidate for a fellowship position.

STUDY DESIGN

LORs for applicants to 8 pediatric subspecialty fellowships at a single academic center from the 2020 Match were analyzed in this cross-sectional study. Frequency of validated agentic and communal terms in each letter were determined by a language processing web application. Bias was determined as having a >5% surplus of agentic or communal terms.

RESULTS

We analyzed 1521 LORs from 409 applicants: 69% were women, 28% were under-represented minorities in medicine (URM), and 50% were invited to interview. Overall, 66% of LORs were agentic biased, 16% communal biased, and 19% neutral. There was no difference in bias in LORs by an applicant's gender (woman 67% agentic vs man 62% agentic; P = .058), race, or ethnicity (non-URM 65% agentic vs URM 67% agentic; P = .660). Despite a lower frequency of agentic terms in LORs for applicants invited for interviews, when accounting for other components of an application and applicant demographics, no significant association was made between language bias in LORs and fellowship interview status.

CONCLUSIONS

The frequency of agentic and communal terms in LORs for pediatric subspecialty fellowship candidates were not found to influence the decision to invite a candidate to interview. However, raising awareness of potential areas of bias within the pediatric fellowship selection process might lead to a more equitable and holistic approach to application review.

Gender bias in colorectal surgery fellowship letters of recommendation

BACKGROUND

As leaders strive to create equitable surgical pipelines, one process under scrutiny is letters of recommendation (LORs). We sought to review the Colon and Rectal Surgery (CRS) Resident Candidate Assessment questionnaire and LORs for gendered differences.

METHODS

This retrospective observational study of letters of recommendation to CRS fellowship during the 2018-2019 application cycle utilized linguistic Inquiry and Word Count (LIWC2015) software to assess letter length and themes comparing differences by applicant and referee gender.

RESULTS

103 applicants (35 ​% women) with 363 LORs (16 ​% written by women) were included. Short answer responses were longer for women applicants, while LORs were longer for men applicants (368 vs 325 words p ​= ​0.03). Men applicants' strengths had more technical skill descriptors, while women applicants' strengths had more emotional language and cognitive and perceptual words.

CONCLUSIONS

This study found significant differences between LORs written for CRS fellowship applicants based on gender.

Do gender differences exist in letters of recommendation for reproductive endocrinology and infertility fellowship?

OBJECTIVE(S)

To determine if gender differences exist in letters of recommendation (LORs) for reproductive endocrinology and infertility fellowship.

DESIGN

Cohort study.

SETTING

Academic medical center.

PATIENTS

A cohort of applicants to a reproductive endocrinology and infertility fellowship program at a single Midwestern academic institution in a single year.

INTERVENTIONS

None.

MAIN OUTCOME MEASURES

Quantitative measures included linguistics inquiry and word count analysis, with 4 summary variables and 25 word categories, to quantify tone and themes present by gender. Performing qualitative analysis in parallel to linguistic analysis allowed for the exploration of themes not conveyed in quantitative methods alone. Qualitative measures included the frequency of code themes in LOR by applicant gender.

RESULTS

There were 272 letters from 72 applicants, 54 (76%) of which were women and 17 (24%) were men. One applicant was excluded because gender information was not specified; 269 letters were included in the quantitative linguistics and qualitative coding analysis. One hundred 10 letters (41%) were written by women and 159 (59%) by men. LOR written for men had a higher mean word count than those written for women (537 vs. 474, P=.04). Linguistics Inquiry and Word Count analysis exhibited more risk words used to describe men applicants (P=.01). When comparing word categories by letter writer gender, women letter writers more frequently used communal, affect, and home word categories whereas men writers more frequently used affiliation-related words. Qualitative analysis revealed that letters for men applicants described leadership, altruism, rapport with patients, and strong endorsements more frequently, whereas women applicants' letters more often mentioned doubt raisers and disclosures of personal life. In addition, letters for women applicants more often described candidates as drama free, not easy to "fluster," and not complainers. Furthermore, letters for women more frequently disclosed parenthood status.

CONCLUSIONS

Gender differences exist among both letter writers and applicants. Qualitative analysis revealed that women applicants were more likely to be described with doubt raisers and personal life disclosures, whereas men were more likely to be described by their altruism and with a strong endorsement. Increasing bias awareness and implementing mitigation strategies would improve gender equity in LOR.

Examination of residency program websites for the use of gendered language and imagery

BACKGROUND

Significant disparity in gender distribution exists among medical specialties. Residency program websites are a main source of preliminary program information for candidates, and website content may influence a prospective applicant's sense of belongingness within a particular program. Given the importance of the residency program website as a recruiting tool, this study sought to examine and compare the presence of gendered language and imagery on residency program websites across various specialties.

METHODS

A list of words considered masculine or feminine was used to evaluate residency program websites of the two most male-dominated specialties (orthopedic and thoracic surgery), female-dominated specialties (pediatrics and obstetrics and gynecology), and gender-balanced specialties (dermatology and family medicine) in the United States in 2022. Forty-five residency programs were randomly selected from each specialty across different regions of the US, with the exception of thoracic surgery of which there are only 33 programs. Masculine and feminine words were evaluated using a parsing and scraping program. Representation of female and male-presenting team members in photos on program websites was also evaluated.

RESULTS

Masculine wording occurred more frequently in male-dominated specialties compared to gender-balanced (p = 0.0030), but not female-dominated specialties (p = 0.2199). Feminine language was used more frequently in female-dominated compared to male dominated fields (p = 0.0022), but not gender balanced (p = 0.0909). The ratio of masculine-to-feminine words used was significantly higher in male-dominated specialties compared to both gender-balanced (p < 0.0001) and female-dominated specialties. (p < 0.0001). There was an average of 1, 7, and 10 female-presenting residency team members pictured on each male-dominated, gender balanced, and female-dominated specialty RPW respectively, with significantly more female-presenting team members pictured in the photographs on female-dominated specialty websites when compared to male-dominated and gender-balanced specialty websites (p < 0.0001, p = 0.014).

CONCLUSIONS

The use of gendered language and female representation in photographs varies significantly across specialties and is directly correlated with gender representation within the specialty. Given that students' perceptions of specialty programs may be affected by the use of language and photos on residency program websites, programs should carefully consider the language and pictures depicted on their program websites.

Can AI Mitigate Bias in Writing Letters of Recommendation?

Letters of recommendation play a significant role in higher education and career progression, particularly for women and underrepresented groups in medicine and science. Already, there is evidence to suggest that written letters of recommendation contain language that expresses implicit biases, or unconscious biases, and that these biases occur for all recommenders regardless of the recommender's sex. Given that all individuals have implicit biases that may influence language use, there may be opportunities to apply contemporary technologies, such as large language models or other forms of generative artificial intelligence (AI), to augment and potentially reduce implicit biases in the written language of letters of recommendation. In this editorial, we provide a brief overview of existing literature on the manifestations of implicit bias in letters of recommendation, with a focus on academia and medical education. We then highlight potential opportunities and drawbacks of applying this emerging technology in augmenting the focused, professional task of writing letters of recommendation. We also offer best practices for integrating their use into the routine writing of letters of recommendation and conclude with our outlook for the future of generative AI applications in supporting this task.

Using a Consensus-Driven Approach to Incorporate Holistic Review Into an Internal Medicine Residency Program

Background

Despite growing interest in incorporating holistic review within residency admissions, implementation by residency programs remains challenging.

Objective

To incorporate holistic review into the internal medicine residency program at the University of Wisconsin and to report initial feasibility and acceptability data.

Methods

During the 2020-2021 application cycle, residency stakeholders performed a consensus-driven process to identify highly valued applicant attributes. We used a holistic review process to identify the presence of these attributes among applicants and updated our rank list algorithm to incorporate these attributes. We modified our interview screening criteria and rank list algorithm to de-emphasize other metrics. We surveyed stakeholders to assess time required for this process and compared our final rank list to what it would have been using our prior system.

Results

The final list of 10 prioritized applicant attributes included extraordinary leadership, community service, and grit, among others. Among 25 matched residents, 8 (32%) were recognized to have exceptional achievement within one of these 10 attributes. Four members of the incoming intern class (16%) would have been in a rank position lower than our historical matched resident cutoff had they not received additional points for these attributes. Faculty reported that holistic review of applications took an additional 3.8 minutes on average. It was felt that current application materials limit the ability to implement a fully holistic review.

Conclusions

The addition of holistic review to our residency admissions process was achieved using a consensus-driven approach and showed favorable feasibility and acceptability data.

Bias, Fairness, and Validity in Graduate-School Admissions: A Psychometric Perspective

As many schools and departments are considering the removal of the Graduate Record Examination (GRE) from their graduate-school admission processes to enhance equity and diversity in higher education, controversies arise. From a psychometric perspective, we see a critical need for clarifying the meanings of measurement "bias" and "fairness" to create common ground for constructive discussions within the field of psychology, higher education, and beyond. We critically evaluate six major sources of information that are widely used to help inform graduate-school admissions decisions: grade point average, personal statements, resumes/curriculum vitae, letters of recommendation, interviews, and GRE. We review empirical research evidence available to date on the validity, bias, and fairness issues associated with each of these admission measures and identify potential issues that have been overlooked in the literature. We conclude by suggesting several directions for practical steps to improve the current admissions decisions and highlighting areas in which future research would be beneficial.

An Analysis of Diversity Statements and Support of Special Interest Societies by General Surgery Residency Programs

OBJECTIVE

The purpose of this study was to characterize the nondiscrimination and diversity, equity, and inclusion (DEI) statements found on the websites of general surgery residency programs, as well as to measure programmatic commitment to diversity through their involvement with special interest surgical societies (SISS).

DESIGN

The authors evaluated the relationship between DEI statements and SISS participation, and performed a natural language processing analysis of general surgery residency DEI statements.

SETTING

The residency program websites from 319 non-military general surgery residency programs within the United States were analyzed.

PARTICIPANTS

This study evaluated the DEI statement and SISS participation in general surgery residency programs.

RESULTS

Of the 319 general surgery residency websites reviewed, 127 (39.8%) featured an identifiable statement of nondiscrimination or commitment to diversity. Compared to programs without diversity statements, programs with statements were more likely to be involved with special interest surgical societies (53.5% vs 30.7%, p < 0.001). Natural language processing analysis revealed that the diversity statements of programs with SISS involvement had higher word counts (p = 0.001), higher clout scores (measure of confidence conveyed, p = 0.001), and higher positive tone scores (p = 0.006) compared with the statements of those without special interest society involvement.

CONCLUSIONS

In the era of virtual interviewing, applicants are forced to rely heavily on surgery residency websites as their main source of information. Less than 40% of programs participating in the Match in 2022 feature diversity statements on their websites. Programs with some degree of involvement with special interest societies were more likely to have statements that score higher in confidence and positivity in natural language processing analysis, which may potentially reflect a more earnest commitment to diversity, equity, and inclusion. Residency programs should continue to improve the visibility of their DEI efforts to recruit a diverse resident class.

Ethnic and Racial Differences in Ratings in the Medical Student Standardized Letters of Evaluation (SLOE)

BACKGROUND

The Standardized Letter of Evaluation (SLOE) stratifies the assessment of emergency medicine (EM) bound medical applicants. However, bias in SLOE, particularly regarding race and ethnicity, is an underexplored area.

OBJECTIVE

This study aims to assess whether underrepresented in medicine (UIM) and non-UIM applicants are rated differently in SLOE components.

METHODS

This was a cross-section study of EM-bound applicants across 3 geographically distinct US training programs during the 2019-2020 application cycle. Using descriptive and regression analyses, we examine the differences between UIM applicants and non-UIM applicants for each of the SLOE components: 7 qualifications of an EM physician (7QEM), global assessment (GA) rating, and projected rank list (RL) position.

RESULTS

Out of a combined total of 3759, 2002 (53.3%) unique EM-bound applicants were included. UIM applicants had lower ratings for each of the 7QEM questions, GA, and RL positions. Compared to non-UIM applicants, only some of the 7QEM components: "Work ethic and ability to assume responsibility," "Ability to work in a team, and "Ability to communicate a caring nature," were associated with their SLOE. "Commitment to EM" correlated more with GA for UIM than for non-UIM applicants.

CONCLUSIONS

This study shows a difference in SLOE rating, with UIM applicants receiving lower ratings than non-UIM applicants.

Building a RAFFT: Impact of a professional development program for women faculty and residents in emergency medicine

BACKGROUND

Women comprise 28% of faculty in academic departments of emergency medicine (EM) and 11% of academic chairs. Professional development programs for women are key to career success and to prevent pipeline attrition. Within emergency medicine, there is a paucity of outcomes-level data for such programs.

OBJECTIVES

We aim to measure the impact of a novel structured professional development curriculum and mentorship group (Resident and Faculty Female Tribe, or RAFFT) within an academic department of EM.

METHODS

This prospective single-center curriculum implementation and evaluation was conducted in the academic year 2020-2021. A planning group identified potential curricular topics using an iterative Delphi process. We developed a 10-session longitudinal curriculum; a postcurriculum survey was conducted to assess the perceived benefit of the program in four domains.

RESULTS

A total of 76% of 51 eligible women attended at least one session; for this project we analyzed the 24 participants (47%) who attended at least one session and completed both the pre- and the postsurvey. The majority of participants reported a positive benefit, which aligned with their expectations in the following areas: professional development (79.2%), job satisfaction (83.3%), professional well-being (70.8%), and personal well-being (79.2%). Resident physicians more often reported less benefit than expected compared to fellow/faculty physicians. Median perceived impact on career choice and trajectory was positive for all respondents.

CONCLUSIONS

Success of this professional development program was measured through a perceived benefit aligning with participant expectations, a positive impact on career choice and career trajectory for participants in each career stage, and a high level of engagement in this voluntary program. Recommendations for the successful implementation of professional development programs include early engagement of stakeholders, the application of data from a program-specific needs assessment, early dissemination of session dates to allow for protected time off, and structured discussions with appropriate identification of presession resources.

Gender differences in emergency medicine standardized letters of evaluation

OBJECTIVES

The Standardized Letter of Evaluation (SLOE) is a vital portion of any medical student's emergency medicine (EM) residency application. Prior literature suggests gender bias in EM SLOE comparative ranking, but there is limited understanding of the impact of gender on other SLOE components. The study objective was to evaluate the presence of gender differences in the 7 Qualifications for EM (7QEM), Global Assessment (GA), and anticipated Rank List (RL) position. A secondary objective was to evaluate the gender differences in 7QEM scores and their link to GA and anticipated RL position.

METHODS

We performed a cross-sectional study using SLOEs from a subset of United States applicants to three EM residency programs during the 2019-2020 application cycle. We collected self-reported demographics, 7QEM scores, GA, and anticipated RL position. We utilized linear regression analyses and repeated measures ANOVA to evaluate if the relationship between the 7QEM scores, GA score, and anticipated RL position was different for men and women.

RESULTS

2103 unique applicants were included (38.6% women, 61.4% men), with 4952 SLOEs meeting inclusion criteria. The average QEM (2.51 vs. 2.39; p < 0.001), GA (2.68 vs. 2.48; p < 0.001), and RL (2.68 vs. 2.47; p < 0.001) scores were statistically higher for women than men. When exploring the relationship between the 7QEM and GA, Ability to communicate a caring nature to patients was not found to be a statistically significant predictor for men, but it was for women. When exploring the relationship between 7QEM and RL, Commitment to EM was not a significant predictor for men, but it was for women.

CONCLUSIONS

Women scored higher than men on the 7QEM, GA, and anticipated RL position on SLOEs. The 7QEM scores factored differently for men and women.

Gender-Based Linguistic Analysis of Pediatric Clinical Faculty Evaluations

BACKGROUND

Gendered stereotypes are embedded in the culture of medicine. Women are stereotypically expected to act collaboratively and less assertively, while men are expected to act with authority and power. Whether gender-biased language is expressed in academic pediatric teaching evaluations is unknown.

OBJECTIVE

Determine whether stereotypic gender-based linguistic differences exist in resident evaluations of pediatric faculty.

METHODS

We performed a retrospective cross-sectional study of clinical faculty evaluations by pediatric residents in a single program from July 2016 to June 2019. Using Linguistic Inquiry and Word Count, responses to 2 open-ended questions were analyzed for stereotypic language. Categories were reported as a percent of total words written. Comparisons between gender groups were conducted using nonparametric Wilcoxon rank sum tests. Rates of word use within each category were analyzed using logistic regression where faculty and resident gender were included as predictor variables.

RESULTS

A total of 6436 free-text responses from 3218 unique evaluations were included. As hypothesized, evaluations of women faculty were less likely than those of men to include certain agentic language like power (odds ratio [OR] 0.9, P < .001) and insight (OR 0.9, P < .001), and research words (OR 0.6, P = .003). As expected, evaluations of women were more likely to include grindstone words, like "hardworking" (OR 1.2, P = .012). Contrary to our hypothesis, women received fewer teaching words like "mentor" (OR 0.9, P = .048) and communal words like "friendly" (OR 0.6, P = .001).

CONCLUSION

Certain stereotypic language was demonstrated in clinical teaching evaluations of pediatric faculty. These findings should be further examined to improve gender inequities in academic pediatrics.

Lower-Third Standardized Letters of Evaluation in Emergency Medicine: Does Gender Make a Difference in Match Outcome?

Objective The purpose of this study was to determine whether gender influences the likelihood of receiving a lower-third global assessment (GA) on the standardized letter of evaluation (SLOE) submitted as part of the emergency medicine (EM) application process as well as the impact of gender on ultimate match outcomes for applicants receiving a lower-third GA ranking. Our hypothesis was that female applicants with a lower-third GA ranking have a higher risk of not matching. Methods We conducted a retrospective cohort study evaluating U.S.-based allopathic applicants to a single EM residency program in the Mid-Atlantic region during the 2017-2018 and 2018-2019 match cycles. GA SLOE rankings and gender for all applicants were extracted and compared to the National Resident Matching Program (NRMP) data for each applicant on match outcome. Comparative analyses were conducted between gender and SLOE GA rankings in order to obtain an odds ratio (OR) of gender and match outcomes. Results A total of 2,017 SLOEs were reviewed from 798 applicants in the 2018 and 2019 EM match cycles. Overall, 716 (90%) applicants successfully matched in EM, with 82 (10%) applicants failing to match into EM; 277 students had at least one lower-third GA ranking. For all applicants, having at least one lower-third GA ranking was associated with a significant risk of not matching (OR: 0.20; 95% CI: 0.12-0.34). Of the 277 students with at least one lower-third GA ranking, 85 (31%) were female and 192 (69%) were male. Of the female applicants with a lower-third GA ranking, 15 (18%) failed to match in EM, and 39 (20%) of the males failed to match in EM. For applicants with a lower-third GA ranking, female gender alone was not associated with a significantly increased risk of not matching (OR: 1.18; 95% CI: 0.61-2.21). Conclusions Female applicants receive a lower-third GA ranking less frequently than their male counterparts. One or more lower-third rankings on the GA significantly reduced an applicant's chances of matching into an EM program. For those with a lower-third GA ranking, female gender alone does not significantly increase the risk of not matching into EM.

Representation by Gender of Recognition Award Recipients from Gastroenterology and Hepatology Professional Societies

Background: Recognition awards from professional medical societies play an important role in physicians' career advancement. Our aim was to evaluate the gender representation of award recipients from gastroenterology and hepatology societies. Methods: We analyzed the lists of award recipients from the American Association for the Study of Liver Diseases, American College of Gastroenterology, American Gastroenterological Association, and American Society for Gastrointestinal Endoscopy and determined the gender of these award recipients. The primary outcome was the overall representation of women physician award recipients as compared with the representation of women in the specialty. Results: Between 1941 and 2019, there were 921 awards, of which 77 (8.4%) were given to women and 844 (91.6%) to men. There was a significant increase in the proportion of women recipients over time, from 0% in 1970-1984 to 22.0% in 2015-2019 (p for trend <0.0001). Compared with the concurrent representation of women in the specialty, women physician recipients were underrepresented from 1970 to 1981 and 1984 to 2015, equitably represented from 1981 to 1984, and overrepresented from 2015 to 2019. Of the total number of awards in each category, women received 13.8% of teaching/mentorship awards and 3.8% of the highest achievement awards (p = 0.002). The proportion of women recipients varied among the societies, from 6.8% to 14.5%. Conclusions: The representation of women physician recipients of gastroenterology and hepatology society recognition awards has generally been low until most recently, when it has surpassed the proportion of women in the specialty. Because award recognition is important to career development, professional societies should have transparent processes that aim to identify and reduce various forms of bias, including gender-related bias, in all phases of award recognition.

Linguistic Analysis of Letters of Recommendation for Vascular Surgery and Obstetrics and Gynecology Applicants Detects Differences in Attributable Strengths Based on Gender

OBJECTIVE

We asked whether letters of recommendation (LOR) written for applicants to vascular surgery (VS), a field where men have traditionally predominated, differentially highlight attributes based on applicant gender. For comparison, LOR for applicants to Obstetrics and Gynecology (Ob/Gyn), a surgical field where women are highly represented were evaluated.

DESIGN

We performed a cross-sectional review of LORs for students applying to VS or Ob/Gyn at our institution from 2017 to2018. Blinded to the gender of both the applicant and the letter author, investigators assessed word count per letter and used published rubrics to quantify how many words in the following categories: communal ("friendly"), able ("competent"), standout ("exceptional"), and grindstone ("hardworking"). Frequencies were reported as a function of specialty and gender.

SETTING

The study was performed at the University of Pittsburgh Medical Center and included letters written for applicants only to the stated residency programs at University of Pittsburgh Medical Center.

PARTICIPANTS

LOR written for self-identified women and men applying to both residencies from US-based allopathic medical schools were de-identified and evaluated by blinded reviewers.

RESULTS

One hundred and ninety-eight letters were reviewed for vascular surgery applicants. Two hundred letters were randomly selected from applications to Ob/Gyn as a comparison. Fifty-four vascular (27.8% women) and 63 Ob/Gyn (77.8% women) applicants were reviewed (p < 0.001 for gender). Licensing exam scores were higher for women than men applying to Ob/Gyn. Honor status was similar across fields and gender. Letters were shorter for VS applicants (p = 0.04). Gender-specific words (i.e., "lady" or "gentleman") were used more in VS letters (0.24 ± 0.50 vs 0.14 ± 0.42, p = 0.048). Ability words were more common (4.7 ± 2.6 vs 3.8 ± 2.1, p = 0.028) and grindstone adjectives were less common (3.4 ± 2.3 vs 4.5 ± 3.1, p = 0.024) in letters written for women compared to men VS applicants. Twenty-nine letters written for students applying to VS had honors status. While none written for women mentioned this achievement, 43% of those written for men did (p < 0.05). Letters for women applicants to Ob/Gyn contained more standout adjectives than those written for men (2.12 ± 2.2 vs 1.39 ± 1.25, p = 0.021).

CONCLUSIONS

Reference letters for both specialties highlighted attributes differently depending on the gender of the applicant. Although this likely represents an unconscious process, care should be taken to limit potential biases in LOR which are "gatekeepers" to access and advancement.

Gender Differences in Language of Standardized Letter of Evaluation Narratives in Osteopathic Emergency Medicine Residency Applicants

Background

The standardized letter of evaluation (SLOE) is used by emergency medicine (EM) faculty during the interview and match process. Data has shown that female allopathic applicants score higher in communal characteristics and have a greater number of ability words in the narrative portion of the SLOE as compared to their male counterparts. 

Objective

To determine if there is a difference in the language used to describe male and female osteopathic applicants within the SLOE. 

Methods

All applicants to a three-year EM residency within a single application cycle were eligible for inclusion. Exclusion criteria included allopathic applicants, applicants without a SLOE, or applicants with a SLOE only from the interviewing program. Data collected included applicant demographics and SLOE narratives. The previously validated Linguistic Inquiry and Word Count (LIWC; Pennebaker Conglomerates, Inc., Austin, TX) product was used to analyze word counts from the narrative portion of each SLOE. Descriptive statistics and t-tests for continuous data were used. 

Results

Of the 577 applicants to the residency program, 318 met inclusion criteria and 33% were female. Females had a higher COMLEX-2 (590 vs 559; p=0.05) as compared to males but no difference was found for the remainder of the baseline demographics. No difference was found for the number of words in the narrative portion of the SLOE between males and females (males = 122 words; females = 127 words; p=0.53). Words within the social (p=0.006), achievement (p=0.007), and standout (p<0.001) categories were more frequent in osteopathic female applicants as compared to males. No statistical differences were detected for the other 13 categories analyzed. 

Conclusion

In this sample of osteopathic applicants, little linguistic difference was noted for the narrative portion of the SLOE. SLOE authors did, however, use more social, achievement, and standout words to describe females as compared to male applicants. 

Characterization of the 2019 Micrographic Surgery and Dermatologic Oncology Standardized Letter of Recommendation

BACKGROUND

Traditional letters of recommendation used for postgraduate medical training applications have multiple limitations, including a lack of clarity, inflated and overly flattering assessments, and low reliability between interpreting faculty. A micrographic surgery and dermatologic oncology (MSDO) standardized letter of recommendation (SLOR) was created to improve the efficiency, validity, and stratification of applicants to dermatology fellowship training programs.

OBJECTIVE

To analyze the MSDO SLOR for trends in grading based on letter-writer and applicant characteristics and to evaluate its ability to demonstrate differences between applicants.

METHODS

Standardized letter of recommendations received by 4 fellowship programs from the 2019 San Francisco Match application cycle were reviewed retrospectively.

RESULTS

Two hundred forty-nine SLORs were analyzed from 140 applicants. Grade inflation and limited variability in scores were evident. Higher scores correlated with the length of the relationships between letter-writers and applicants and with female letter-writer gender. There was no applicant gender or ethnicity bias detected.

CONCLUSION

Despite score inflation, the MSDO SLOR allows for differentiation between fellowship applicants. Future studies are needed to further evaluate the reliability of the SLOR and find ways to improve its content.

Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation

BACKGROUND

The standardized letter of evaluation (SLOE) is the application component that program directors value most when evaluating candidates to interview and rank for emergency medicine (EM) residency. Given its successful implementation, other specialties, including otolaryngology, dermatology, and orthopedics, have adopted similar SLOEs of their own, and more specialties are considering creating one. Unfortunately, for such a significant assessment tool, no study to date has comprehensively examined the validity evidence for the EM SLOE.

OBJECTIVE

We summarized the published evidence for validity for the EM SLOE using Messick's framework for validity evidence.

METHODS

A scoping review of the validity evidence of the EM SLOE was performed in 2020. A scoping review was chosen to identify gaps and future directions, and because the heterogeneity of the literature makes a systematic review difficult. Included articles were assigned to an aspect of Messick's framework and determined to provide evidence for or against validity.

RESULTS

There have been 22 articles published relating to validity evidence for the EM SLOE. There is evidence for content validity; however, there is a lack of evidence for internal structure, relation to other variables, and consequences. Additionally, the literature regarding response process demonstrates evidence against validity.

CONCLUSIONS

Overall, there is little published evidence in support of validity for the EM SLOE. Stakeholders need to consider changing the ranking system, improving standardization of clerkships, and further studying relation to other variables to improve validity. This will be important across GME as more specialties adopt a standardized letter.

Letters of recommendation: Nuanced bias or useful affirmation?

Narrative letters of recommendation (NLORs) have become key elements of the application process for residency and fellowship. The potential to inadvertently admit bias into these subjective narratives has become an area of research focus across many disciplines. In the present review, we have highlighted the current data regarding bias in NLORs. We also believe that one of the most effective methods to eliminate bias from written recommendations is to first understand that it exists. Thus, the objective measures that have been taken to identify bias in NLORs are important steps in the right direction. We have presented and reflected on the accrued data on bias in NLORs pertaining to surgical training.

Gender bias in reference letters for residency and academic medicine: a systematic review

Reference letters play an important role for both postgraduate residency applications and medical faculty hiring processes. This study seeks to characterise the ways in which gender bias may manifest in the language of reference letters in academic medicine. In particular, we conducted a systematic review in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We searched Embase, MEDLINE and PsycINFO from database inception to July 2020 for original studies that assessed gendered language in medical reference letters for residency applications and medical faculty hiring. A total of 16 studies, involving 12 738 letters of recommendation written for 7074 applicants, were included. A total of 32% of applicants were women. There were significant differences in how women were described in reference letters. A total of 64% (7/11) studies found a significant difference in gendered adjectives between men and women. Among the 7 studies, a total of 86% (6/7) noted that women applicants were more likely to be described using communal adjectives, such as "delightful" or "compassionate", while men applicants were more likely to be described using agentic adjectives, such as "leader" or "exceptional". Several studies noted that reference letters for women applicants had more frequent use of doubt raisers and mentions of applicant personal life and/or physical appearance. Only one study assessed the outcome of gendered language on application success, noting a higher residency match rate for men applicants. Reference letters within medicine and medical education exhibit language discrepancies between men and women applicants, which may contribute to gender bias against women in medicine.

Assessment of gender differences in letters of recommendation for pharmacy residency applicants

Disclaimer

In an effort to expedite the publication of articles related to the COVID-19 pandemic, AJHP is posting these manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time.

Purpose

Letters of recommendation (LORs) are highly regarded components of pharmacy residency applications, as they provide insight into an applicant’s character and capabilities. In other medical fields, differences in language have been reported for letters written for female and male applicants; however, data on gender differences in LORs for pharmacy residency applications are currently lacking.

Methods

LORs for applicants to our institution’s postgraduate year 1 pharmacy residency program for the 2019-2020 academic year were extracted and processed by a natural language processing service. Words within 18 categories were identified and counted for each LOR. Total was also compared.

Results

Of the 473 LORs included for analysis, 320 (67.7%) were written for female applicants and 153 (32.3%) were written for male applicants. Approximately two-thirds of all writers were women for both female and male applicants. In comparing letters for women and men, there was a statistically significant difference in the percentage of LORs that contained terms in categories described as gendered, solitary/reserved, and desire. There was no statistically significant difference in total or in the presence of words in other categories such as grindstone, standout, agentic, or communal. When controlling for grade point average, writer gender, duration that the writer knew the applicant, and the writer’s professional position, there were no changes to the statistical findings.

Conclusion

Letters written for female and male applicants were largely similar with regard to length and word categories utilized. While no clear gender bias was found when evaluating pharmacy residency LORs, writers must continue to assess their implicit biases and how those biases might affect a candidate’s application.

Perspectives on the quality and utility of letters conforming to the AAIM guidelines

Background Standardized letters of recommendation (SLOR) have become common features of the medical school to residency transition. Research has shown many advantages over the narrative letter of recommendation including improved letter-writing efficiency, ease of interpretation, and improved reliability as performance predictors. Currently, at least four specialties require fellowship SLORs. Internal medicine adopted its SLOR in 2017. Previous research showed fellowship program directors’ satisfaction with the 2017 guidelines. Little is known about residency program directors’ acceptance and adherence to the guidelines.

Objectives The study sought to assess the adoption rate of each component, barriers to adoption, time commitment, and alignment with intended goals of the guidelines.

Methods Anonymous survey links were posted to an internal medicine discussion forum prior to the guidelines in spring 2017 and twice following the guidelines in fall 2018 and winter 2019. Two-sample tests of proportions were used to compare respondent characteristics with known survey population data. Pre- and post-survey comparisons were assessed for statistical significance with Pearson chi-squared statistic.

Results The response rate varied from 30% to 35% for each survey period. Medical knowledge, patient care, interpersonal and communication skills, professionalism, and scholarly activity were reported frequently (>96%) at baseline. Inclusion of residency program characteristics, systems-based practice, practice-based learning and improvement, and skills sought to master increased over the study period.

Conclusions The new guidelines improved uniform reporting of all core competency data. Overall, the gains were modest, as many pre-survey respondents reported high rates of including components within the guidelines.

Do male chairs of surgery have implicit gender bias in the residency application process?

INTRODUCTION

Medical students applying for general surgery residencies often require a letter of recommendation (LOR) from the Surgical Chair. However, LORs may reveal gender and ethnic bias. This study examines the presence of implicit bias in general surgery resident selection by evaluating chair LORs.

METHODS

A retrospective study of 149 LORs for categorical general surgery residents, at an academic institution, written by surgery chairs from 1980 to 2013 was performed. Two independent reviewers scored each letter for overall quality, mention of personality, academic deficiencies, technical skills, and standout adjectives. Scores were compared across gender and race and statistical analysis performed using SPSS.

RESULTS

Males comprised 85% of the applicants; racial makeup was Caucasian (90%), black (4%), Asian (4%), and Hispanic (2%). Male chairs wrote all letters. Letters for female students received higher overall scores than males (4.13 ± 0.16, 3.59 ± 0.08, p = 0.005). Discussion of personality was significantly less for Asian students (1.28 ± 0.08, 1.83 ± 0.48, 0.5 ± 0.224, 1.67 ± 0.67, p 0.050). No difference was present in mention of academic deficiencies, technical skills, or standout adjectives.

DISCUSSION

Female medical students invited to interview at a top academic general surgery residency had higher quality LORs than their male counterparts. Asian applicants had statistically significant less discussion of personality. Further examination of residency selection processes is necessary to implement changes that mitigate implicit bias in trainee selection.