Race and Gender Bias in Internal Medicine Program Director Letters of Recommendation

Racial and Ethnic Bias in Letters of Recommendation in Academic Medicine: A Systematic Review

PURPOSE

Letters of recommendations (LORs) are key components of academic medicine applications. Given that bias against students and trainees underrepresented in medicine (UIM) has been demonstrated across assessment, achievement, and advancement domains, the authors reviewed studies on LORs to assess racial, ethnic, and UIM differences in LORs. Standardized LORs (SLORs), an increasingly common form of LORs, were also assessed for racial and ethnic differences.

METHOD

A systematic review was conducted for English-language studies that assessed racial or ethnic differences in LORs in academic medicine published from database inception to July 16, 2023. Studies evaluating SLORs underwent data abstraction to evaluate their impact on the given race or ethnicity comparison and outcome variables.

RESULTS

Twenty-three studies describing 19,012 applicants and 41,925 LORs were included. Nineteen studies (82.6%) assessed LORs for residency, 4 (17.4%) assessed LORs for fellowship, and none evaluated employment or promotion. Fifteen of 17 studies (88.2%) assessing linguistic differences reported a significant difference in a particular race or ethnicity comparison. Of the 7 studies assessing agentic language (e.g., "strong," "confident"), 1 study found fewer agentic terms used for Black and Latinx applicants, and 1 study reported higher agency scores for Asian applicants and applicants of races other than White. There were mixed results for the use of communal and grindstone language in UIM and non-UIM comparisons. Among 6 studies, 4 (66.7%) reported that standout language (e.g., "exceptional," "outstanding") was less likely to be ascribed to UIM applicants. Doubt-raising language was more frequently used for UIM trainees. When SLORs and unstructured LORs were compared, fewer linguistic differences were found in SLORs.

CONCLUSIONS

There is a moderate bias against UIM candidates in the domains of linguistic differences, doubt-raising language, and topics discussed in LORs, which has implications for perceptions of competence and ability in the high-stakes residency and fellowship application process.

Letters of Reference in the Current Era

ABSTRACT

Letters of reference (LORs) are a common component of the application process for residency training programs. With the United States Medical Licensing Examination Step 1 transitioning to pass/fail grading and with the increasing use of holistic review, the potential role of LORs is rising in importance. Among some key benefits are the ability to provide a broader and more holistic view of applicants, which can include highlighting elements of experiences or skills that could be missed in their application, as well as providing a third-party assessment of the applicant external to their rotation experiences. However, LORs also face issues, including variation in quality, challenges with comparability, and risk of bias. In this article, the authors discuss the unique benefits, limitations, and best practice recommendations for LORs in academic medicine. The authors also discuss future directions, including the role of artificial intelligence, unblinded, and co-created LORs.

Decoding the Reference Letter: Strategies to Reduce Unintentional Gender Bias in Letters of Recommendation

Introduction

There is a growing body of literature on gender bias in letters of recommendation (LORs) in academic medicine and the negative effect of bias on promotion and career advancement. Thus, increasing knowledge about gender bias and developing skills to mitigate it is important for advancing gender equity in medicine. This workshop aims to provide participants with knowledge about linguistic bias (focused on gender), how to recognize it, and strategies to apply to mitigate it when writing LORs.

Methods

We developed an interactive 60-minute workshop for faculty and graduate medical education program directors consisting of didactics, reflection exercises, and group activities. We used a postworkshop survey to evaluate the effectiveness of the workshop. Descriptive statistics were used to analyze Likert-scale questions and a thematic content analysis for open-ended prompts.

Results

We presented the workshop four times (two local and two national conferences) with one in-person and one virtual format for each. There were 50 participants who completed a postworkshop survey out of 74 total participants (68% response rate). Ninety-nine percent of participants felt the workshop met its educational objectives, and 100% felt it was a valuable use of their time. Major themes described for intended behavior change included utilization of the gender bias calculator, mindful use and balance of agentic versus communal traits, closer attention to letter length, and dissemination of this knowledge to colleagues.

Discussion

This workshop was an effective method for helping participants recognize gender bias when writing LORs and learn strategies to mitigate it.

Teaching the way I wish I was taught: Design and implementation of a class on historically excluded and underrepresented scientists

As the first Inupiaq person to earn a PhD in microbiology, I learned the hard way that groups of people have been excluded from science, technology, engineering and mathematics in the United States since the first University was built by Black and Indigenous slaves. Students from historically excluded and underrepresented (HEU) backgrounds typically do not see themselves in textbooks, conferences, or classrooms, especially in science, technology, engineering, mathematics and medicine (STEMM) fields. Similarly, students from these backgrounds and non-excluded backgrounds typically do not understand the history or consequences of exclusion. Here I describe the development and implementation of a class that teaches undergraduate students about the current state of diversity in STEMM jobs in the US, the history of exclusion that resulted in a deficit of people from various backgrounds, the consequences of excluding these people from research specifically, current leaders in research from HEU backgrounds, and how to implement changes. The students are taught how to communicate their findings in oral and written communication to various audiences. Based on decades of experiences, discussions, readings, and more, I teach students the reasons there are so few people from HEU backgrounds in academia and in STEMM specifically, and what can be done at the University level to ensure that people from all backgrounds are represented in STEMM. In this way, I teach students what I wish I had been taught decades ago.

Selection and Recruitment Strategies among Competitive Pediatric Training Programs and the Impact of Diversity

OBJECTIVE

Aspects of the written application, interview and ranking may negatively impact recruitment of underrepresented in medicine (URiM) applicants. Our objectives were to explore knowledge, attitudes, and perceptions of pediatric faculty who assess potential trainees and how diversity impacts these assessments.

METHODS

We performed qualitative interviews of 20 geographically diverse faculty at large pediatric residencies and fellowships. We analyzed data using the constant comparative method to develop themes.

RESULTS

Four main themes emerged.

CONCLUSIONS

We describe ways in which bias infiltrates recruitment and strategies to promote diversity. Many strategies are variably implemented and the impact on workforce diversity in pediatric training programs remains unknown.

Language Patterns in Letters of Recommendation for Residency Applicants in Otolaryngology

OBJECTIVE

Assess for gender and race patterns in agentic and communal language used in letters of recommendation for Otolaryngology-Head and Neck Surgery (OHNS) residency applicants.

STUDY DESIGN

Retrospective content analysis.

SETTING

Applications from OHNS applications at a single training institution for the 2019 and 2020 match cycles.

METHODS

A total of 2283 letters of recommendation for 611 OHNS applicants were analyzed. Applicant and letter writer gender, applicant race and ethnicity, and applicant characteristics including United States Medical Licensing Examination® Step 1 score, research productivity, and medical school rank were extracted. Agentic and communal word use from the letters of recommendation was compared across applicant and writer characteristics using multilevel negative binomial regression modeling.

RESULTS

Letter writers use a greater rate of agentic terms when describing applicants who self-identify as Asian (incidence rate ratio [IRR] = 1.16, p < .01) or "Other/not reported" (IRR = 1.23, p < .01) as compared to white applicants. Further, standardized letters of evaluation had significantly more communal language and less agentic language. Although there was an increase in communal language in letters for female applicants compared to male applicants, these gender differences disappeared in the multivariate model.

CONCLUSION

Multivariate analysis demonstrated no significant gender-based patterns in the communal or agentic language in letters of recommendation for OHNS residency applicants. However, letters for applicants identifying as Asian or "other/not reported" had more frequent use of agentic terms. Future studies should investigate other components of residency applications to assess how gender and race bias might unfairly influence an applicant's chances at a given program.

Identifying Gender and Racial Bias in Pediatric Fellowship Letters of Recommendation: Do Word Choices Influence Interview Decisions?

OBJECTIVES

To describe linguistic differences in letters of recommendation (LORs) for pediatric fellowship candidates based on applicant and letter writer demographics and to examine if these differences influenced the decision to interview a candidate for a fellowship position.

STUDY DESIGN

LORs for applicants to 8 pediatric subspecialty fellowships at a single academic center from the 2020 Match were analyzed in this cross-sectional study. Frequency of validated agentic and communal terms in each letter were determined by a language processing web application. Bias was determined as having a >5% surplus of agentic or communal terms.

RESULTS

We analyzed 1521 LORs from 409 applicants: 69% were women, 28% were under-represented minorities in medicine (URM), and 50% were invited to interview. Overall, 66% of LORs were agentic biased, 16% communal biased, and 19% neutral. There was no difference in bias in LORs by an applicant's gender (woman 67% agentic vs man 62% agentic; P = .058), race, or ethnicity (non-URM 65% agentic vs URM 67% agentic; P = .660). Despite a lower frequency of agentic terms in LORs for applicants invited for interviews, when accounting for other components of an application and applicant demographics, no significant association was made between language bias in LORs and fellowship interview status.

CONCLUSIONS

The frequency of agentic and communal terms in LORs for pediatric subspecialty fellowship candidates were not found to influence the decision to invite a candidate to interview. However, raising awareness of potential areas of bias within the pediatric fellowship selection process might lead to a more equitable and holistic approach to application review.

Measuring and Predicting Faculty Consensus Rankings of Standardized Letters of Evaluation

Background Standardized letters of evaluation (SLOE) are becoming more widely incorporated into the residency application process to make the letter of recommendation, an already critical component in a residency application packet, more objective. However, it is not currently known if the reviewers of these letters share consensus regarding the strength of an applicant determined by their SLOE. Objective We measured the level of faculty agreement regarding applicant competitiveness as determined by SLOEs and the ability of 2 algorithms to predict faculty consensus rankings. Methods Using data from the 2021-2022 Match cycle from the Council of Residency Directors in Emergency Medicine SLOE Database as a blueprint, authors created 50 fictional SLOEs representative of the national data. Seven faculty then rated these SLOEs in order of applicant competitiveness, defined as suggested rank position. Consensus was evaluated using cutoffs established a priori, and 2 prediction models, a point-based system and a linear regression model, were tested to determine their ability to predict consensus rankings. Results There was strong faculty consensus regarding the interpretation of SLOEs. Within narrow windows of agreement, faculty demonstrated similar ranking patterns with 83% and 93% agreement for "close" and "loose" agreement, respectively. Predictive models yielded a strong correlation with the consensus ranking (point-based system r=0.97, linear regression r=0.97). Conclusions Faculty displayed strong consensus regarding the competitiveness of applicants via SLOEs, adding further support to the use of SLOEs for selection and advising. Two models predicted consensus competitiveness rankings with a high degree of accuracy.

Gender influences resident physicians' perception of an employee-to-employee recognition program: a mixed methods study

BACKGROUND

Burnout is prevalent in medical training. While some institutions have implemented employee-to-employee recognition programs to promote wellness, it is not known how such programs are perceived by resident physicians, or if the experience differs among residents of different genders.

METHODS

We used convergent mixed methods to characterize how residents in internal medicine (IM), pediatrics, and general surgery programs experience our employee-to-employee recognition ("Hi-5″) program. We collected Hi-5s received by residents in these programs from January 1, 2021-December 31, 2021 and coded them for recipient discipline, sex, and PGY level and sender discipline and professional role. We conducted virtual focus groups with residents in each training program.

MAIN MEASURES AND APPROACH

We compared Hi-5 receipt between male and female residents; overall and from individual professions. We submitted focus group transcripts to content analysis with codes generated iteratively and emergent themes identified through consensus coding.

RESULTS

Over a 12-month period, residents received 382 Hi-5s. There was no significant difference in receipt of Hi-5s by male and female residents. Five IM, 3 surgery, and 12 pediatric residents participated in focus groups. Residents felt Hi-5s were useful for interprofessional feedback and to mitigate burnout. Residents who identified as women shared concerns about differing expectations of professional behavior and communication based on gender, a fear of backlash when behavior does not align with gender stereotypes, and professional misidentification.

CONCLUSIONS

The "Hi-5" program is valuable for interprofessional feedback and promotion of well-being but is experienced differently by men and women residents. This limitation of employee-to-employee recognition should be considered when designing equitable programming to promote well-being and recognition.

Evaluation of Family Medicine Residency Programs for Diversity, Equity, and Inclusion Milestones

BACKGROUND AND OBJECTIVES

In 2020, the Association of Family Medicine Residency Directors (AFMRD) Diversity and Health Equity (DHE) Task Force developed and piloted a framework to measure diversity, equity, and inclusion (DEI) initiatives in medical residencies across five domains: curriculum, evaluation, institution, resident pathway, and faculty pathway. The objectives were (1) to measure DEI initiatives across multiple domains in family medicine residencies using the DEI milestones and (2) to obtain current national baseline data providing criteria against which to measure effectiveness of initiatives and create tailored benchmarks.

METHODS

We developed a cross-sectional survey of 12 quantitative residency characteristic items and the five DEI milestone ratings, and distributed the survey to program directors of family medicine residencies accredited by the Accreditation Council for Graduate Medical Education in the spring of 2022. We generated descriptive statistics, including item frequencies and cross-tabulations, and completed subgroup comparisons with analysis of variance.

RESULTS

We collected aggregate milestone data for 194 family medicine residencies of 588 eligible programs (33% response rate). Respondents represented 48 states and US territories: 107 community-based, university-affiliated; 48 community-based; 34 university-based; and 5 military/other programs. Overall, the curriculum milestone was rated the highest (mean=2.54, SD=1.03), whereas the faculty pathway (mean=1.94, SD=1.04) and resident pathway (mean=2.02, SD=1.06) milestones were rated lowest.

CONCLUSIONS

DEI milestone data may support residency programs as they assess their institution's developmental progress across five key domains. Additionally, aggregate data may shed light on collective strengths and areas for improvement.

Adoption of Internal Medicine Milestone Ratings and Changes in Bias Against Black, Latino, and Asian Internal Medicine Residents

BACKGROUND

The 2014 adoption of the Milestone ratings system may have affected evaluation bias against minoritized groups.

OBJECTIVE

To assess bias in internal medicine (IM) residency knowledge ratings against Black or Latino residents-who are underrepresented in medicine (URiM)-and Asian residents before versus after Milestone adoption in 2014.

DESIGN

Cross-sectional and interrupted time-series comparisons.

SETTING

U.S. IM residencies.

PARTICIPANTS

59 835 IM residents completing residencies during 2008 to 2013 and 2015 to 2020.

INTERVENTION

Adoption of the Milestone ratings system.

MEASUREMENTS

Pre-Milestone (2008 to 2013) and post-Milestone (2015 to 2020) bias was estimated as differences in standardized knowledge ratings between U.S.-born and non-U.S.-born minoritized groups versus non-Latino U.S.-born White (NLW) residents, with adjustment for performance on the American Board of Internal Medicine IM certification examination and other physician characteristics. Interrupted time-series analysis measured deviations from pre-Milestone linear bias trends.

RESULTS

During the pre-Milestone period, ratings biases against minoritized groups were large (-0.40 SDs [95% CI, -0.48 to -0.31 SDs; P < 0.001] for URiM residents, -0.24 SDs [CI, -0.30 to -0.18 SDs; P < 0.001] for U.S.-born Asian residents, and -0.36 SDs [CI, -0.45 to -0.27 SDs; P < 0.001] for non-U.S.-born Asian residents). These estimates decreased to less than -0.15 SDs after adoption of Milestone ratings for all groups except U.S.-born Black residents, among whom substantial (though lower) bias persisted (-0.26 SDs [CI, -0.36 to -0.17 SDs; P < 0.001]). Substantial deviations from pre-Milestone linear bias trends coincident with adoption of Milestone ratings were also observed.

LIMITATIONS

Unobserved variables correlated with ratings bias and Milestone ratings adoption, changes in identification of race/ethnicity, and generalizability to Milestones 2.0.

CONCLUSION

Knowledge ratings bias against URiM and Asian residents was ameliorated with the adoption of the Milestone ratings system. However, substantial ratings bias against U.S.-born Black residents persisted.

PRIMARY FUNDING SOURCE

None.

Words Used in Letters of Recommendation for Pediatric Residency Applicants: Demographic Differences and Impact on Interviews

OBJECTIVE

To describe differences in agentic (achievement) and communal (relationship) terms in letters of recommendation (LORs) for pediatric residency candidates by applicant and letter writer demographics and to examine if LOR language is associated with interview status.

METHODS

A random sample of applicant profiles and LORs submitted to one institution were analyzed from the 2020-21 Match. Letters of recommendation text was inputted into a customized natural language processing application which determined the frequency of agentic and communal words in each LOR. Neutral LORs were defined as having< 5% surplus of agentic or communal terms.

RESULTS

We analyzed 2094 LORs from 573 applicants: 78% were women, 24% were under-represented in medicine (URiM), and 39% were invited to interview. Most letter writers were women (55%) and of senior academic rank (49%). Overall, 53% of LORs were agency biased, 25% communal biased, and 23% neutral. There was no difference in agency and communally biased LORs by an applicant's gender (men 53% agentic vs women 53% agentic, P = .424), race or ethnicity (non-URiM 53% agentic vs URiM 51% agentic, P = .631). Male letter writers used significantly more agentic terms (8.5%) compared to women (6.7% agentic) or writers of both genders (3.1% communal) (P = .008). Applicants invited to interview were more likely to have a neutral LOR; however, no significant association existed between language and interview status.

CONCLUSIONS

No significant differences in language were found by applicant gender or race among pediatric residency candidates. Identifying potential biases within pediatric residency selection processes is important in creating an equitable approach to application review.

USMLE Step 2 CK Scores and Radiology Residency Applications: Does it Affect Diversity in Selection of Trainees?

RATIONALE AND OBJECTIVES

As residency programs in diagnostic radiology aspire to broaden trainee diversity, reliance on certain criteria may affect the selection of candidates from underrepresented groups. With the conversion of reporting of the United States Medical Licensing Examination (USMLE) Step 1 scores to pass/fail, programs may rely more on numerical USMLE Step 2 Clinical Knowledge (CK) scores. The purpose of our investigation is to assess the effects of Step 2 CK scores on the selection of underrepresented minority (URM) and female candidates.

MATERIALS AND METHODS

Applications from United States senior allopathic medical students to a radiology residency program from the 2021-2023 National Residency Matching Program cycles were analyzed. Subjects were classified as male or female and URM or non-URM by self-identification. Step 2 CK scores were compared and the use of cutoff scores was examined for disparate effects.

RESULTS

1017 subjects fulfilled the entry criteria. There were 721 males and 296 females, with 164 URM and 853 non-URM candidates. Comparing males to females, there was no significant difference in the mean score (p = 0.21) and no disparate effects of cutoff scores. There was a significant difference between the mean score of URM versus non-URM candidates of eight points (p < 0.00011). The use of cutoffs showed a disparate effect on URM candidates with a cutoff score of 250 (average score for 2022 matched applicants) excluding 71% of URM candidates while excluding only 46% of non-URM candidates.

CONCLUSION

Reliance on USMLE Step 2 CK scores to screen applications for radiology residency can disadvantage URM candidates. Females are not adversely affected.

Gender and culture bias in letters of recommendation for computer science and data science masters programs

Letters of Recommendation (LORs) are widely utilized for admission to both undergraduate and graduate programs, and are becoming even more important with the decreasing role that standardized tests play in the admissions process. However, LORs are highly subjective and thus can inject recommender bias into the process, leading to an inequitable evaluation of the candidates' competitiveness and competence. Our study utilizes natural language processing methods and manually determined ratings to investigate gender and cultural differences and biases in LORs written for STEM Master's program applicants. We generate features to measure important characteristics of the LORs and then compare these characteristics across groups based on recommender gender, applicant gender, and applicant country of origin. One set of features, which measure the underlying sentiment, tone, and emotions associated with each LOR, is automatically generated using IBM Watson's Natural Language Understanding (NLU) service. The second set of features is measured manually by our research team and quantifies the relevance, specificity, and positivity of each LOR. We identify and discuss features that exhibit statistically significant differences across gender and culture study groups. Our analysis is based on approximately 4000 applications for the MS in Data Science and MS in Computer Science programs at Fordham University. To our knowledge, no similar study has been performed on these graduate programs.

Inclusion, Diversity, Access, and Equity in Infectious Diseases Fellowship Training: Tools for Program Directors

The Infectious Diseases Society of America (IDSA) has set clear priorities in recent years to promote inclusion, diversity, access, and equity (IDA&E) in infectious disease (ID) clinical practice, medical education, and research. The IDSA IDA&E Task Force was launched in 2018 to ensure implementation of these principles. The IDSA Training Program Directors Committee met in 2021 and discussed IDA&E best practices as they pertain to the education of ID fellows. Committee members sought to develop specific goals and strategies related to recruitment, clinical training, didactics, and faculty development. This article represents a presentation of ideas brought forth at the meeting in those spheres and is meant to serve as a reference document for ID training program directors seeking guidance in this area.

How to Write Powerful Letters of Recommendation

Nurses at all levels may be asked to write a letter of recommendation to support colleagues' and students' career and academic advancement. This article outlines key steps and considerations in the letter-writing process.

The Gendered Work/Role of Program Directors in International Graduate Medical Education

Healthcare organizations offer numerous clinical and academic leadership pathways for physicians, among which the position of program director (PD) is considered to be a prominent educational leadership role. As PDs are instrumental in the recruitment and training of the next generations of physicians, PD gender distribution can affect the present and future of a medical specialty. This study offers a dialectical perspective in understanding how international PDs negotiate gendered understanding of their work/role by using the framework of Relational Dialectics Theory 2.0. Thirty-three interviews of PDs from Qatar, Singapore, and the United Arab Emirates were conducted and, using contrapuntal analysis, the competing discourses of meanings of gender in the PD work/role were examined. Competing discourses where structural, cultural, and professional meanings of gender were interrogated revealed inherent multiple meanings of how gender is understood in PD work/roles. In making sense of these meanings of gender, PDs express dilemmas of traditional gender binaries of masculine/feminine work/role meanings to explain the term in different ways in their everyday organizational and cultural struggles. The findings have implications for PD recruitment and retention in teaching hospitals.

Differences in international medical graduates' letters of recommendation by gender in pulmonary and critical care medicine: a cohort analysis

BACKGROUND

International Medical Graduates (IMGs) encounter barriers as they seek to match into fellowship programs in the United States (US). This study's objective is to determine if there are differences in letters of recommendation written for IMGs compared to U.S. Medical Graduates (USMGs) applying to pulmonary and critical care medicine (PCCM) fellowship programs.

METHODS

All applications submitted to a PCCM fellowship program in 2021 were included in this study. The applicant demographics and accomplishments were mined from applications. The gender of letter writers was identified by the author's pronouns on professional websites. Word count and language differences in the letters were analyzed for each applicant using the Linguistic Inquiry and Word Count (LWIC2015) program. Multivariable linear regressions were performed controlling for applicant characteristics to identify if IMG status was associated with total word count and degree of support, measured by a composite outcome encompassing several categories of adjectives, compared to USMG status.

RESULTS

Of the 573 applications, most of the applicants were USMGs (72%, N = 334/573). When adjusting for applicant characteristics, IMG applicants had shorter letters of recommendation (87.81 total words shorter 95% CI: - 118.61, - 57.00, p-value < 0.01) and less supportive letters (4.79 composite words shorter 95% CI: - 6.61, - 2.97, p-value < 0.01), as compared to USMG applicants. Notably, female IMG applicants had the biggest difference in their word counts compared to USMG applicants when the letter writer was a man.

CONCLUSIONS

IMG applicants to a PCCM fellowship received shorter and less supportive letters of recommendation compared to USMG applicants.

Preparing Effective Narrative Evaluations for the Medical School Performance Evaluation (MSPE)

Introduction

In 2016, the AAMC Medical School Performance Evaluation (MSPE) Task Force issued recommendations to standardize the MSPE but did not address the quality of the written narratives in that document. Narrative evaluations are hampered by code words, polite rhetoric, and bias to the detriment of students. To address this, the AAMC's Group on Student Affairs and Group on Educational Affairs convened an expert group to consider the state of narratives in the MSPE and develop resources to improve their quality.

Methods

A series of interactive workshops was developed and presented at an AAMC webinar and national meetings. A presentation outlining challenges and possible approaches to improvement was followed with large-group discussion and/or small-group breakout activity to analyze and improve upon sample clinical comments and create summary clerkship paragraphs. The initial webinar used polling questions and free-text prompts to gather feedback for future workshops. Anonymous survey responses were collected at the end of each subsequent workshop to determine perceived effectiveness and potential utility at participants' institutions.

Results

Over 680 administrators, faculty, and staff participated in the webinar or in one of four national-level workshops. Respondents agreed that the modules would be useful in faculty development and wanted to replicate their learning at their own institutions for overall better impact on the quality of MSPE narratives.

Discussion

This resource addresses an important gap in the medical education literature. A variety of stakeholders affirmed that these workshops have value in training writers to improve their narrative comments for the MSPE.

Words Matter: Examining Gender Differences in the Language Used to Evaluate Pediatrics Residents

BACKGROUND

Gender disparities in academic medicine continue to be pervasive. Written evaluations of residents may provide insight into perceptions of residents by faculty, which may influence letters of recommendation for positions beyond residency and reinforce perceived stereotype threat experienced by trainees.

OBJECTIVE

To examine language used in faculty evaluations of pediatrics residents to determine if there are differences in language used with respect to gender of resident.

DESIGN/METHODS

All faculty evaluations of residents in 3 consecutive intern classes from 2016 to 2018 were collected and redacted for name and gender identifiers. We performed a qualitative analysis of written comments in 2 mandatory free text sections. The study team initially coded text collectively, generating a code book, then individually to apply the coding scheme. Next, evaluations were unblinded to gender. Code applications were aggregated by resident, and frequencies of code application by resident were compared by standardized mean differences to detect imbalances between genders.

RESULTS

A total of 448 evaluations were analyzed: 88 evaluations of 17 male residents, and 360 evaluations of 70 female residents. Codes more frequently applied to women included "enthusiasm," and "caring," while codes more frequently applied to men included "intelligence," and "prepared." A conceptual model was created to reflect potential impacts of these differences using a lens of social role theory.

CONCLUSIONS

We identified differences in the way male and female residents are evaluated by faculty, which may have negative downstream effects on female residents, who may experience negative self-perception, differential development of clinical skills, and divergent career opportunities as a result.

Diversity, Equity, and Inclusion Milestones: Creation of a Tool to Evaluate Graduate Medical Education Programs

Background

As the Accreditation Council for Graduate Medical Education (ACGME) began to ask programs to report their efforts surrounding diversity, equity, and inclusion (DEI), program directors felt ill prepared to evaluate their programs and measure change.

Objective

To develop a tool that would allow graduate medical education (GME) programs to evaluate the current state of DEI within their residencies, identify areas of need, and track progress; to evaluate feasibility of using this assessment method within family medicine training programs; and to analyze and report pilot data from implementation of these milestones within family medicine residency programs.

Methods

The Association of Family Medicine Residency Directors (AFMRD) Diversity and Health Equity (DHE) Task Force developed a tool for program DEI evaluation modeled after the ACGME Milestones. These milestones focus on DEI assessment in 5 key domains: Institution, Curriculum, Evaluation, Resident Personnel, and Faculty Personnel. After finalizing a draft, a pilot implementation of the milestones was conducted by a convenience sample of 10 AFMRD DHE Task Force members for their own programs.

Results

Scores varied widely across surveyed programs for all milestones. Highest average scores were seen for the Curriculum milestone (2.65) and the lowest for the Faculty Personnel milestone (2.0). Milestone assessments were completed within 10 to 40 minutes using various methods.

Conclusions

The AFMRD DEI Milestones were developed for program assessment, goal setting, and tracking of progress related to DEI within residency programs. The pilot implementation showed these milestones were easily used by family medicine faculty members in diverse settings.

Does Objective Scoring of Applications for Radiology Residency Affect Diversity?

RATIONALE AND OBJECTIVES

It is important to recognize if reliance on certain factors in applications affects selection of trainees from under-represented groups. Our purpose is to determine if objective scoring of radiology residency applications based on quantifiable data regarding academic performance, United States Medical Licensing Examination (USMLE) scores and research productivity affects selection of female and under-represented minority (URM) candidates.

MATERIALS AND METHODS

We reviewed 502 applications from three successive match cycles from United States allopathic medical students. Each application was scored for academic performance, USMLE results and research productivity determining an overall score. The scores of males were compared to females and URM were compared to non-URM candidates. USMLE cutoff scores were evaluated for disparate effects.

RESULTS

There were 348 male, 154 female, 73 URM and 429 non-URM candidates. For male versus female applicants, there was no significant difference in mean academic performance, USMLE Step 2 Clinical Knowledge and research productivity scores. Males had higher mean USMLE Step 1 (p = 0.005) and overall candidate scores (p = 0.02). Between URM and non-URM candidates there was no significant difference in academic performance. Non-URM applicants had higher mean USMLE Step 1 (p = 0.008), USMLE Step 2 (p = 0.002), research productivity (p = 0.001) and overall scores (p = 0.02). Use of USMLE cutoff scores demonstrated disparate effects on female and URM candidates.

CONCLUSION

Objective scoring of applications and use of USMLE cutoff scores can disadvantage candidates from underrepresented groups. Screening filters can affect the diversity of candidate pools for radiology residency.