Linguistic Analysis of Letters of Recommendation for Vascular Surgery and Obstetrics and Gynecology Applicants Detects Differences in Attributable Strengths Based on Gender

Assessment of Gender Differences in Letters of Recommendation for Physical Therapy Residency Applications

INTRODUCTION

Letters of recommendation (LOR) are an integral component of physical therapy residency applications. Identifying the influence of applicant and writer gender in LOR will help identify whether potential implicit gender bias exists in physical therapy residency application processes.

REVIEW OF LITERATURE

Several medical and surgical residency education programs have reported positive, neutral, or negative LOR female gender bias among applicants and writers. Little research exists on gender differences in LOR to physical therapy education programs or physical therapy residency programs.

SUBJECTS

Seven hundred sixty-eight LOR were analyzed from 256 applications to 3 physical therapy residency programs (neurologic, orthopaedic, sports) at one institution from 2014 to 2020.

METHODS

Thematic categories were developed to identify themes in a sample of LOR. Associations between writer and applicant gender were analyzed using summary statistics, word counts, thematic and psycholinguistic extraction, and rule-based and deep learning Natural Language Processing .

RESULTS

No significant difference in LOR word counts were found based on writer or applicant gender. Increased word counts were seen in sports residency LOR compared with the orthopaedic residency. Thematic analysis showed LOR gender differences with male applicants receiving more positive generalized recommendations and female applicants receiving more comments regarding interpersonal relationship skills. No thematic or psycholinguistic gender differences were seen by LOR writer. Male applicants were 1.9 times more likely to select all male LOR writers, whereas female applicants were 2.1 times more likely to choose all female LOR writers.

DISCUSSION AND CONCLUSION

Gender differences in LORs for physical therapy residencies were found using a comprehensive Natural Language Processing approach that identified both a positive recommendation male applicant gender bias and a positive interpersonal relationship skill female applicant gender bias. Applicants were not harmed nor helped by selecting LOR writers of the opposite gender. Admissions committees and LOR writers should be mindful of potential implicit gender biases in LOR submitted to physical therapy residency programs.

Maximizing Standardization While Ensuring Equity: Exploring the Role of Applicant Experiences, Attributes, and Metrics on Performance of a Surgery-Specific Situational Judgment Test

BACKGROUND

Situational judgment tests (SJT) are hypothetical but realistic scenario-based assessments that allow residency programs to measure judgment and decision-making among future trainees. A surgery-specific SJT was created to identify highly valued competencies among residency applicants. We aim to demonstrate a stepwise process for validation of this assessment for applicant screening through exploration of two often-overlooked sources of validity evidence - relations with other variables and consequences.

METHODS

This was a prospective multi-institutional study involving 7 general surgery residency programs. All applicants completed the SurgSJT, a 32-item test aimed to measure 10 core competencies: adaptability, attention to detail, communication, dependability, feedback receptivity, integrity, professionalism, resilience, self-directed learning, and team orientation. Performance on the SJT was compared to application data, including race, ethnicity, gender, medical school, and USMLE scores. Medical school rankings were determined based on the 2022 U.S. News & World Report rankings.

RESULTS

In total, 1491 applicants across seven residency programs were invited to complete the SJT. Of these, 1454 (97.5%) candidates completed the assessment. Applicants were predominantly White (57.5%), Asian (21.6%), Hispanic (9.7%), Black (7.3%), and 52% female. A total of 208 medical schools were represented, majority were allopathic (87.1%) and located in United States (98.7%). Less than a quarter of applicants (22.8%; N=337) were from a top 25 school based on U.S. News & World Report rankings for primary care, surgery, or research. Average USMLE Step 1 score was 235 (SD 37) and Step 2 score was 250 (SD 29). Sex, race, ethnicity, and medical school ranking did not significantly impact performance on the SJT. There was no relationship between SJT score and USMLE scores and medical school rankings.

CONCLUSIONS

We demonstrate the process of validity testing and importance of two specific sources of evidence-consequences and relations with other variables, in implementing future educational assessments.

Gender differences in cardiothoracic surgery letters of recommendation

OBJECTIVE

To investigate whether or not gender influences letters of recommendation for cardiothoracic surgery fellowship.

METHODS

From applications to an Accreditation Council Graduate Medical Education cardiothoracic surgery fellowship program between 2016 and 2021, applicant and author characteristics were examined with descriptive statistics, analysis of variance, and Pearson χ2 tests. Linguistic software was used to assess communication differences in letters of recommendation, stratified by author and applicant gender. An additional higher-level analysis was then performed using a generalized estimating equations model to examine linguistic differences among author-applicant gender pairs.

RESULTS

Seven hundred thirty-nine recommendation letters extracted from 196 individual applications were analyzed; 90% (n = 665) of authors were men and 55.8% (n = 412) of authors were cardiothoracic surgeons. Compared with women authors, authors who are men wrote more authentic (P = .01) and informal (P = .03) recommendation letters. When writing for women applicants, authors who are men were more likely to display their own leadership and status (P = .03) and discuss women applicants' social affiliations (P = .01), like occupation of applicant's father or husband. Women authors wrote longer letters (P = .03) and discussed applicants' work (P = .01) more often than authors who are men. They also mentioned leisure activities (P = .03) more often when writing for women applicants.

CONCLUSIONS

Our work identifies gender-specific differences in letters of recommendation. Women applicants may be disadvantaged because their recommendation letters are significantly more likely to focus on their social ties, leisure activities, and the status of the letter writer. Author and reviewer awareness of gender-biased use of language will aid in improvements to the candidate selection process.

Confidence of Faculty in Writing Letters of Recommendation for Pediatric Fellowship Applicants

BACKGROUND

The assessment of pediatric residents applying to subspecialty fellowship programs relies on faculty letters of recommendation (LOR). However, it is unclear if pediatric faculty are confident that their LOR are effective.

OBJECTIVE

This study aims to assess the confidence of pediatric faculty in writing an effective LOR for pediatric residents applying to subspecialty fellowship programs.

METHODS

Survey development was conducted using evidence-based best practices. Surveys were distributed via email in 2021 to all full-time pediatric faculty members who taught pediatric residents in a large academic medical center. Categorical values were compared by chi-square test.

RESULTS

Eighty-five out of 150 (57%) faculty members completed the survey. Forty-one percent of participants were very confident that their LOR provided adequate content to assess residents during the application process. Confidence was associated with higher academic rank (p=0.02), frequent contact with residents (p=0.01), and writing >2 LOR in the last five years (p=0.0002). Confident LOR writers were more likely to describe their own background, details about the resident's scholarly activity, and the resident's ability to work as part of a team. Thirty-five percent of respondents reported never considering gender bias when writing LOR, whereas 28% reported always considering gender bias. Eighty-seven percent of respondents reported an interest in receiving LOR writing guidelines.

CONCLUSION

Half of the faculty respondents were not very confident in their ability to write an effective LOR for pediatric residents applying for a fellowship. Faculty development and standardized instructions on writing effective LOR may be helpful both at the institutional and national levels, including the importance of considering gender bias when writing LOR.

Letters of recommendation for gynecologic oncology fellowship differ by gender and race

INTRODUCTION

Letters of recommendation (LOR) play a significant role in applicant selection for gynecologic oncology (GO) fellowship. LORs can be agentic, associated with competence, independence and strength, or communal, associated with being accommodating, cooperative and empathetic. Agentic LORs have been shown to reflect favorably on applicants, making them more likely to be selected for interview. The primary goal of this study was to determine linguistic differences in LORs for applicants applying to GO fellowship based on applicant gender and race.

METHODS

All applications to a single academic institution from the years 2018-2020 were analyzed. LORs were extracted from all applicants who self-reported their gender and race/ethnicity. Analysis was performed using Linguistic Inquiry Word Count (LIWC) in terms of agentic and communal language. All analysis was done using SAS version 9.4.

RESULTS

From 2018 to 2020, there were 239 GO fellowship applications to a single academic institution. There were 186 (78.1%) applicants who identified as female and 52 (21.8%) who identified as male. Female applicants were more likely to have more research (p = 0.047) and volunteer activities (p = 0.02) than male applicants. There were no differences between female and male applicants in terms of age, geographic location of residency, USMLE scores, AOA status, number of publications or attending a Doximitiy-rated top 20 residency program. There were 139 (60.7%) applicants who identified as White, 46 (20.1%) as Asian, 16 (7.0%) as Black, 19 (8.3%) as Latinx and 9 (3.9%) as other. On multivariable analysis, LORs written for male applicants had higher agentic scores compared to females (p < 0.05), and LORs of female applicants received higher communal scores (p < 0.05). LORs for applicants who identified as Black or Latinx were found to have lower agentic scores compared to individuals who identified as White(p < 0.05 for both).

CONCLUSION

There were demonstrated linguistic differences in LOR for GO fellowship based on applicant gender and race/ethnicity. LORs written for white, male applicants were more likely to have higher agentic scores compared to minority, female applicants. LORs written for female applicants were more likely to have higher communal scores. As LORs have been shown to be instrumental in GO fellowship interview selection and linguistic differences exist based on applicant gender and race/ethnicity, this study highlights the need for a more objective GO fellowship application process.

DISCLOSURES

Portion of this paper was presented at the Society for Gynecologic Oncology as part of a focus plenary.

Examining Implicit Bias Differences in Pediatric Surgical Fellowship Letters of Recommendation Using Natural Language Processing

OBJECTIVE

We analyzed the prevalence and type of bias in letters of recommendation (LOR) for pediatric surgical fellowship applications from 2016-2021 using natural language processing (NLP) at a quaternary care academic hospital.

DESIGN

Demographics were extracted from submitted applications. The Valence Aware Dictionary for sEntiment Reasoning (VADER) model was used to calculate polarity scores. The National Research Council dataset was used for emotion and intensity analysis.  The Kruskal-Wallis H-test was used to determine statistical significance.  SETTING: This study took place at a single, academic, free standing quaternary care children's hospital with an ACGME accredited pediatric surgery fellowship.

PARTICIPANTS

Applicants to a single pediatric surgery fellowship were selected for this study from 2016 to 2021. A total of 182 individual applicants were included and 701 letters of recommendation were analyzed.

RESULTS

Black applicants had the highest mean polarity (most positive), while Hispanic applicants had the lowest.  Overall differences between polarity distributions were not statistically significant.   The intensity of emotions showed that differences in "anger" were statistically significant (p=0.03).  Mean polarity was higher for applicants that successfully matched in pediatric surgery.

DISCUSSION

This study identified differences in LORs based on racial and gender demographics submitted as part of pediatric surgical fellowship applications to a single training program. The presence of bias in letters of recommendation can lead to inequities in demographics to a given program. While difficult to detect for humans, natural language processing is able to detect bias as well as differences in polarity and emotional intensity. While the types of emotions identified in this study are highly similar among race and gender groups, the intensity of these emotions revealed differences, with "anger" being most significant.

CONCLUSION

From this work, it can be concluded that bias in LORs, as reflected as differences in polarity, which is likely a result of the intensity of the emotions being used and not the types of emotions being expressed.   Natural language processing shows promise in identification of subtle areas of bias that may influence an individual's likelihood of successful matching.

Vascular surgery integrated resident selection criteria in the pass or fail era

OBJECTIVES

Vascular surgery integrated residency (VSIR) programs are highly competitive; however, criteria for resident selection remain opaque and non-standardized. The already unclear selection criteria will be further impacted by the impending transition of the United States Medical Licensing Examination (USMLE) Step 1 from numeric scores to a binary pass/fail outcome. The purpose of this study was to investigate the historical and anticipated selection criteria of VSIR applicants.

METHODS

This was a cross-sectional, nationwide, 59-item survey that was sent to all VSIR program directors (PDs). Data was analyzed using the Fisher exact test if categorical and the Mann-Whitney U test and the Kruskal-Wallis test if ordinal.

RESULTS

Forty of 69 PDs (58%) responded to the survey. University-based programs constituted 85% of responders. Most VSIR PDs (65%) reported reviewing between 101 to 150 applications for 1 to 2 positions annually. Forty-two percent of the responding PDs reported sole responsibility for inviting applicants to interview, whereas 50% had a team of faculty responsible for reviewing applications. On a five-point Likert scale, letters of recommendation (LOR) from vascular surgeons or colleagues (a person the PD knows) were the most important objective criteria. Work within a team structure was rated highest among subjective criteria. The majority of respondents (72%) currently use the Step 1 score as a primary method to screen applicants. Regional differences in use of Step 1 score as a primary screening method were: Midwest (100%), Northeast (76%), South (43%), and West (40%) (P = .01). PDs responded that that they will use USMLE Step 2 score (42%) and LOR (10%) to replace USMLE Step 1 score. The current top ranked selection criteria are letters from a vascular surgeon, USMLE Step 1 score and overall LOR. The proposed top ranked selection criteria after transition of USMLE Step 1 to pass/fail include LOR overall followed by Step 2 score.

CONCLUSIONS

This is the first study to evaluate the selection criteria used by PDs for VSIR. The landscape of VSIR selection criteria is shifting and increasing transparency is essential to applicants' understanding of the selection process. The transition of USMLE Step 1 to a pass/fail report will shift the attention to Step 2 scores and elevate the importance of other relatively more subjective criteria. Defining VSIR program selection criteria is an important first step toward establishing holistic review processes that are transparent and equitable.

Do gender and racial differences exist in letters of recommendation for obstetrics and gynecology residency applicants?

BACKGROUND

Studies in several fields demonstrate gender and racial differences in descriptions of applicants in letters of recommendation. However, gender and racial biases in obstetrics and gynecology residency letters of recommendation are understudied.

OBJECTIVE

This study aimed to determine whether there are differences in the letters of recommendation written for medical students applying for obstetrics and gynecology residency based on applicant gender and underrepresented in medicine status.

STUDY DESIGN

We analyzed all letters of recommendation submitted to a single obstetrics and gynecology residency program during the 2019-2020 application cycle using a computerized text analysis software to evaluate the presence of 25 word categories. Multivariable regression was used to compare differences in the frequency of word categories by gender and underrepresented in medicine status. Concurrently, we performed a qualitative content analysis of letters from a random sample of applicants balanced for gender and underrepresented in medicine status and explored emerging themes until thematic saturation was reached.

RESULTS

We analyzed 3060 letters written for 834 applicants, 721 (87%) of whom were women and 198 (24%) of whom were underrepresented in medicine. Men authored 1605 letters (53%). There was no difference in word categories used in letters written for men and women applicants on multivariable analysis. Similarly, there was no word category difference between letters written for applicants who were underrepresented in medicine compared with White and Asian applicants. However, women letter writers used more communal (relationship-oriented) words compared with men letter writers (P<.001). Moreover, on the qualitative analysis of letters from a random sample of 30 applicants, comments about personality traits were frequent in all letters. Comments on surgical skills, work ethic, and leadership were found more in letters for White and Asian applicants, and phrases that were doubt raisers were found more in letters written for underrepresented medicine applicants.

CONCLUSION

On linguistic analysis, letters of recommendation written for obstetrics and gynecology residency applicants were similar overall compared with applicant race and gender.

Don't Judge a Letter by its Title: Linguistic Analysis of Letters of Recommendation by Author's Academic Rank

OBJECTIVE

This study analyzed the linguistic differences in letters of recommendation (LORs) for general surgery residency applicants written by authors of various academic ranks. Given that many general surgery residency programs require a LOR from the Chair of surgery, this study also examined whether LORs written by the Chair demonstrate linguistic differences to support this practice.

DESIGN

A single institution, retrospective review analyzed LORs from two application cycles of general surgery residency applicants who were selected for interview at a large academic institution. Word count (WC) and linguistic characteristics of LORs were analyzed with a previously developed institution-specific dictionary using the Linguistic Inquiry and Word Count software (LIWC2015; Pennebaker Conglomerates, Inc., Austin, Texas). WC and linguistic characteristics of LORs reported as frequencies of terms within twenty-four categories were examined based on the letter authors' academic rank. Further examination compared LORs written by a Chair of surgery with those written by non-Chairs.

SETTING

A single large, Midwestern academic general surgery residency program PARTICIPANTS: Four hundred and sixty-five letters of recommendation received during two interview cycles were included for analysis.

RESULTS

A total of 465 LORs written by assistant (n = 82), associate (n = 94), and full professors (n = 289) were included in the study. No statistically significant difference was noted in the WC of LORs based on the letter writers' academic ranks (p = 0.95). Assistant professors utilized grindstone, communal, and technical skill terms with higher frequencies compared to associate professors and full professors. LORs written by assistant professors demonstrated the highest authentic variable score followed by associate professors then full professors (4.94, 3.92, 3.28, p < 0.01). LORs written by Chairs (n = 128) had lower authentic variable scores compared to LORs written by non-Chairs (n = 337; 2.71 vs. 3.91, p = 0.001). Only 50 (39%) LORs written by Chairs indicated working directly with the applicant, and sub-group analysis demonstrated a higher authentic variable score in this group compared with LORs written by Chairs who did not indicate having worked directly with the applicant (3.51 vs. 2.5, p = 0.01).

CONCLUSIONS

Linguistic analysis of LORs for general surgery residency applicants demonstrated minor yet statistically significant differences based on the author's academic rank. If applicants can obtain linguistically similar LORs from surgeons of any academic rank, but less authentic LORs from writers with higher academic ranks, these LORs may be less valuable for the residency programs when evaluating applicants. Based on the subgroup analysis, less than 40% of Chair LORs indicated that the Chair worked directly with the applicant, calling into question the utility of the Chair LORs as meaningful evaluation of applicants. Further study to compare LORs of applicants selected and not selected for interview may add additional insight into linguistic differences in LORs written by authors of different academic ranks.