Assessment of Gender-Based Linguistic Differences in Physician Trainee Evaluations of Medical Faculty Using Automated Text Mining

Finding the Needle in the Haystack: Can Natural Language Processing of Students' Evaluations of Teachers Identify Teaching Concerns?

BACKGROUND

Institutions rely on student evaluations of teaching (SET) to ascertain teaching quality. Manual review of narrative comments can identify faculty with teaching concerns but can be resource and time-intensive.

AIM

To determine if natural language processing (NLP) of SET comments completed by learners on clinical rotations can identify teaching quality concerns.

SETTING AND PARTICIPANTS

Single institution retrospective cohort analysis of SET (n = 11,850) from clinical rotations between July 1, 2017, and June 30, 2018.

PROGRAM DESCRIPTION

The performance of three NLP dictionaries created by the research team was compared to an off-the-shelf Sentiment Dictionary.

PROGRAM EVALUATION

The Expert Dictionary had an accuracy of 0.90, a precision of 0.62, and a recall of 0.50. The Qualifier Dictionary had lower accuracy (0.65) and precision (0.16) but similar recall (0.67). The Text Mining Dictionary had an accuracy of 0.78 and a recall of 0.24. The Sentiment plus Qualifier Dictionary had good accuracy (0.86) and recall (0.77) with a precision of 0.37.

DISCUSSION

NLP methods can identify teaching quality concerns with good accuracy and reasonable recall, but relatively low precision. An existing, free, NLP sentiment analysis dictionary can perform nearly as well as dictionaries requiring expert coding or manual creation.

Ivory tower in MD/PhD programmes: sticky floor, broken ladder and glass ceiling

OBJECTIVE

Achieving gender equity in academic medicine is not only a matter of social justice but also necessary in promoting an innovative and productive academic community. The purpose of this study was to assess gender distribution in dual MD/PhD academic programme faculty members across North America.

METHODS

Academic metrics were analysed to quantify the relative career success of academic faculty members in MD/PhD programmes. Measured parameters included academic and leadership ranks along with nominal research factors such as peer-reviewed research publications, H-index, citation number and years of active research.

RESULTS

Χ² analysis revealed a statistically significant (p<0.0001, χ²=114.5) difference in the gender distribution of faculty and leadership across North American MD/PhD programmes. Men held 74.2% of full professor positions, 64% of associate professor positions, 59.4% of assistant professor positions and 62.8% of lecturer positions. Moreover, men occupied a larger share of faculty leadership roles with a statistically significant disparity across all ranks (p<0.001, χ²=20.4). A higher proportion of men held positions as department chairs (79.6%), vice chairs (69.1%) and programme leads (69.4%).

CONCLUSION

Gender disparity was prevalent in the MD/PhD programmes throughout North America with women achieving a lower degree of professional stature than men. Ultimately, steps must be taken to support women faculty to afford them better opportunities for academic and professional advancement.

Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation

BACKGROUND

While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender.

OBJECTIVE

To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known).

DESIGN

Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs).

SUBJECTS

Participating faculty in longitudinal rater training.

MAIN MEASURES

Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention).

KEY RESULTS

Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (β 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender.

CONCLUSIONS

Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

Dear Program Director: An Evaluation of Implicit Bias in Letters of Recommendation for Neurosurgery Residency

BACKGROUND AND OBJECTIVES

Despite comprising half of medical students, women represent only 29.6% of neurosurgery applicants and 17% of residents, suggesting a "leak" in the career pipeline for women neurosurgeons. Surveys persistently show that neurosurgery programs identify United States Medical Licensing Examination (USMLE®) Step 1 score and letters of recommendation (LORs) as the most important factors in selecting applicants to interview. A previous study in neurosurgery found no differences in LORs. However, multiple studies in other specialties have demonstrated implicit gender bias in LORs, which may influence resident selection. Our objective is to evaluate neurosurgery residency LORs for evidence of implicit gender bias.

METHODS

Retrospective analysis of LORs for interviewed neurosurgery applicants at a single institution during the 2014 to 2020 National Residency Matching Program (NRMP®) match cycles. Letters were evaluated using Linguistic Inquiry & Word Count (LIWC) software (Pennebaker Conglomerates), and additional applicant data were obtained from candidate applications. LIWC (Pennebaker Conglomerates) output data included custom dictionary categories and terms that were analyzed using Prism 10 and Rstudio.

RESULTS

Two hundred eighteen applications were reviewed for a total of 827 letters. LIWC (Pennebaker Conglomerates) analysis showed significant differences in word count (331 vs 297, difference = 34, 95% CI: 9-61, P = .008). LORs for applicants who were men were more likely to mention Alpha Omega Alpha Honor Medical Society (1.17 vs 0.778, difference = 0.4, 95% CI: 0.13-0.67, P = .023). USMLE® Step 1 scores were significantly lower for women (241 vs 247, difference = 6, 95% CI: 2-10, P = .004). There was no significant difference between letters for men and women for all categories evaluated in the linguistic evaluation.

CONCLUSION

LORs are vital to the neurosurgical residency application process. The data exhibit some differences between the men and women applicants but few differences in their LORs, consistent with the results of the previous neurosurgical study.

The Impact of Faculty Gender on Resident Evaluations of Faculty Performance in Emergency Medicine

INTRODUCTION

Gender bias impacts the promotion and tenure of female emergency medicine (EM) physicians and limits their ability to advance in academic rank. Many factors influence the promotion and tenure process including research, evaluations, opportunities for leadership, sponsorship, and mentorship. The goal of this study is to determine if resident evaluations of EM faculty differ by faculty gender.

METHODS

A quantitative analysis was used to examine 14,613 teaching evaluations of faculty by residents at a single academic center (The Ohio State University Wexner Medical Center, Columbus) in the years 2017-2019. Anonymized ratings of male and female faculty on a five-point Likert scale were compared using Fischer's exact test and adjusting for multiple comparisons.

RESULTS

Male faculty were more likely to hold the rank of Associate Professor or Professor. When taking both faculty gender and rank into account, male Clinical Instructors and Assistant Professors had significantly higher evaluation scores by residents in the domain of resident autonomy than their female counterparts. Regardless of gender or faculty rank, the majority of faculty received scores greater than four.

CONCLUSION

A significant gender difference was found in resident evaluation scores of faculty in the domain of resident autonomy at the level of Clinical Instructor and Assistant Professor. Resident autonomy refers to the degree of supervision by faculty which evolves over time and is primarily based on level of training. This is important as it demonstrates a gender difference in scores that could be used to determine faculty compensation and promotion. Evaluation tools used for promotion and tenure of academic faculty should be evaluated for implicit bias and appropriate statistical analysis.

Differences in Trainee Evaluations of Faculty by Rater and Ratee Gender

PURPOSE

To examine whether gender differences exist in medical trainees' (residents' and fellows') evaluations of faculty at a number of clinical departments.

METHOD

The authors conducted a single-institution (University of Minnesota Medical School) retrospective cohort analysis of 5,071 trainee evaluations of 447 faculty (for which trainee and faculty gender information was available) completed between July 1, 2019, and June 30, 2022. The authors developed and employed a 17-item measure of clinical teaching effectiveness, with 4 dimensions: overall teaching effectiveness, role modeling, facilitating knowledge acquisition, and teaching procedures. Using both between- and within-subject samples, they conducted analyses to examine gender differences among the trainees making ratings (rater effects), the faculty receiving ratings (ratee effects), and whether faculty ratings differed by trainee gender (interaction effects).

RESULTS

There was a statistically significant rater effect for the overall teaching effectiveness and facilitating knowledge acquisition dimensions (B = -0.28 and -0.14, 95% CI: [-0.35, -0.21] and [-0.20, -0.09], respectively, P < .001, medium corrected effect sizes between -0.34 and -0.54); female trainees rated male and female faculty lower than male trainees on both dimensions. There also was a statistically significant ratee effect for the overall teaching effectiveness and role modeling dimensions (B = -0.09 and -0.08, 95% CI: [-0.16, -0.02] and [-0.13, -0.04], P = .01 and < .001, respectively, small to medium corrected effect sizes between -0.16 and -0.44); female faculty were rated lower than male faculty on both dimensions. There was not a statistically significant interaction effect.

CONCLUSIONS

Female trainees rated faculty lower than male trainees and female faculty were rated lower than male faculty on 2 teaching dimensions each. The authors encourage researchers to continue to examine the reasons for the evaluation differences observed and how implicit bias interventions might help to address them.

Can AI Mitigate Bias in Writing Letters of Recommendation?

Letters of recommendation play a significant role in higher education and career progression, particularly for women and underrepresented groups in medicine and science. Already, there is evidence to suggest that written letters of recommendation contain language that expresses implicit biases, or unconscious biases, and that these biases occur for all recommenders regardless of the recommender's sex. Given that all individuals have implicit biases that may influence language use, there may be opportunities to apply contemporary technologies, such as large language models or other forms of generative artificial intelligence (AI), to augment and potentially reduce implicit biases in the written language of letters of recommendation. In this editorial, we provide a brief overview of existing literature on the manifestations of implicit bias in letters of recommendation, with a focus on academia and medical education. We then highlight potential opportunities and drawbacks of applying this emerging technology in augmenting the focused, professional task of writing letters of recommendation. We also offer best practices for integrating their use into the routine writing of letters of recommendation and conclude with our outlook for the future of generative AI applications in supporting this task.

Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content

PURPOSE

Faculty feedback on trainees is critical to guiding trainee progress in a competency-based medical education framework. The authors aimed to develop and evaluate a Natural Language Processing (NLP) algorithm that automatically categorizes narrative feedback into corresponding Accreditation Council for Graduate Medical Education Milestone 2.0 subcompetencies.

METHOD

Ten academic anesthesiologists analyzed 5,935 narrative evaluations on anesthesiology trainees at 4 graduate medical education (GME) programs between July 1, 2019, and June 30, 2021. Each sentence (n = 25,714) was labeled with the Milestone 2.0 subcompetency that best captured its content or was labeled as demographic or not useful. Inter-rater agreement was assessed by Fleiss' Kappa. The authors trained an NLP model to predict feedback subcompetencies using data from 3 sites and evaluated its performance at a fourth site. Performance metrics included area under the receiver operating characteristic curve (AUC), positive predictive value, sensitivity, F1, and calibration curves. The model was implemented at 1 site in a self-assessment exercise.

RESULTS

Fleiss' Kappa for subcompetency agreement was moderate (0.44). Model performance was good for professionalism, interpersonal and communication skills, and practice-based learning and improvement (AUC 0.79, 0.79, and 0.75, respectively). Subcompetencies within medical knowledge and patient care ranged from fair to excellent (AUC 0.66-0.84 and 0.63-0.88, respectively). Performance for systems-based practice was poor (AUC 0.59). Performances for demographic and not useful categories were excellent (AUC 0.87 for both). In approximately 1 minute, the model interpreted several hundred evaluations and produced individual trainee reports with organized feedback to guide a self-assessment exercise. The model was built into a web-based application.

CONCLUSIONS

The authors developed an NLP model that recognized the feedback language of anesthesiologists across multiple GME programs. The model was operationalized in a self-assessment exercise. It is a powerful tool which rapidly organizes large amounts of narrative feedback.

Mentors and Role Models: the Role Female Medical Educators Serve for Female Medical Students

Discussion of diversity, equity, and inclusion (DEI) has moved to the forefront in medical education, and in particular, efforts toward gender equity have emphasized the need for more women faculty and physicians. Gender parity was recently achieved for medical students matriculating into US allopathic schools during the 2017-2018 academic year1. However, this documented increase in women attending medical school as students is not matched by an increase in women teaching in the undergraduate medical education (UME) curriculum. In 2020, the faculty employed by medical schools across the USA (totaling 186,311) includes 43% women; this percentage drops significantly when considering the rank of full professor, of which only 26% are women [1]. For faculty representing graduate programs in science, technology, engineering, and math (STEM), many of which teach in the pre-clerkship phase of UME, less than 25% are women [2], according to the 2019 AAMC statement of gender equity.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01776-1.

Gender Distribution of Authors of Evaluation Letters for Promotion at One Medical School

PURPOSE

Women remain underrepresented in key leadership positions and advanced ranks in academic medicine. This study examines the numbers of men and women letter writers for promotion candidates during a 5-year period across departments, tracks, ranks, and candidate gender.

METHOD

A descriptive study characterized the gender of evaluation letter writers for candidates for promotion to associate or full professor at the University of Minnesota Medical School between 2015 and 2020. Letter writer and candidate gender were characterized by self-identified pronouns in the faculty biography or dossier. Letter writer gender was described by candidate department, promotion track, rank, terminal degree, and gender.

RESULTS

Among 299 candidates for promotion, 172 (58%) were men and 127 (42%) were women; dossiers included 3,995 evaluation letters. Across all years, men wrote more letters than women (external letters, range, 69% in 2019-2020 to 75% in 2015-2016; internal letters, range, 67% in 2018-2019 to 77% in 2015-2016). Candidates in the family medicine and pediatrics departments had the highest percentages of letters written by women (44% and 40%, respectively). No differences were found in the number of women letter writers by candidate promotion track; however, differences were found by candidate rank (associate professor, 30%; full professor, 23%) and terminal degree (MD/DO, 25%; PhD, 33%; MD-PhD, 20%). Regardless of candidate gender, most evaluation letters were written by men. Women candidates had 15% to 20% more letters authored by women than men candidates (34%-40% vs 18%-23%).

CONCLUSIONS

The gender pattern of letter writers may reflect implicit biases regarding gender and perceived leadership status, expertise, and success. Adopting policies that promote or require gender diversity among letter writers for promotion candidates may provide an opportunity to encourage faculty to seek diverse networks and recognize the achievements of women faculty.

Gendered Language in Letters of Recommendation for Applicants to Pulmonary Critical Care Fellowships

Background

Previous work has demonstrated letters of recommendation for women in academic medicine are shorter and emphasize communal traits over grindstone or agentic traits.

Objective

To determine if there are sex-based differences in letters of recommendation written for applicants applying to pulmonary critical care medicine fellowships and if the sex of the letter writer impacts these differences.

Methods

All fellowship applications submitted to a pulmonary critical care medicine fellowship program in 2020 were included in this study. The applicant demographics and self-reported accomplishments were extracted from their application. The sex of letter writers was identified through public online searches. Word count and language differences in the letters of recommendation were analyzed for each applicant using the Linguistic Inquiry and Word Count (LIWC2015) program. Multivariable linear regressions were performed controlling for applicant characteristics to identify if applicant sex was associated with total word counts and total agentic word counts.

Results

Of the 529 complete applications, 2,024 letters of recommendation were reviewed. A majority of the applicants (70%, n = 370/530) and letter writers (75%, n = 1,515/2,024) were male. When adjusting for applicant demographic and accomplishments, female applicants had longer letters of recommendation (30.91 words longer, 95% confidence interval [CI], 1.53–60.29; P = 0.04) and more supportive letters (3.27 words longer, 95% CI, 1.59–4.95; P < 0.01) as compared with male applicants. Female letter writers wrote longer and more supportive letters than male letter writers, and this difference was greatest for female applicants.

Conclusion

Female applicants received longer and more supportive letters of recommendation. Further work is needed to understand if this finding is the beginning of a change in the letters of recommendation for female applicants.

A cross-sectional study of dental students perception of dental faculty gender differences

Objective

The objective of this study was to evaluate students’ perceptions of differences in learning from faculty of different gender.

Method

This cross-sectional study involved pre-doctoral dental students (years 2 to 4) who had a simulation and/or clinical experience working with dental faculty for at least one year. Students completed a self-administered questionnaire with three sections: demographic, difference between faculty related to their knowledge, skill, critical thinking, acceptance of cultural differences, and students’ preferences in working with faculty in specialty clinics.

Results

A total of 136 students completed the survey (75.4% response rate). Participants were 52.6% women, 62.2% self-identified as Caucasian/White. Students reported that female faculty are more understanding (p = 0.001) and accepting of cultural differences (p<0.001) compared to male faculty (p<0.05). Students reported perceiving female faculty more as being a role model than male faculty (p = 0.034). When comparing male and female students, male student’s perception of male faculty as a role model was significantly higher than female students (p<0.05). There was no significant difference in student’s perceptions between male and female faculty in their knowledge, skills, compassion, critical thinking, providing feedback, communication skills, and grading (p>0.05). Caucasian/White students perceived female faculty as more encouraging for discussions and male faculty as more rigid/inflexible (p<0.05).

Conclusions

Students perceived female faculty as more understanding and culturally competent compared to male faculty. There were no significant differences in student’s perceptions of male and female faculty in their knowledge, skills, compassion, critical thinking, feedback, communication skills, and grading. Students perceived female faculty as role models more than male faculty.

The Context of "Confidence": Analyzing the Term Confidence in Resident Evaluations

BACKGROUND

Despite similar performance metrics, women medical trainees routinely self-assess their own skills lower than men. The phenomenon of a "confidence gap" between genders, where women report lower self-confidence independent of actual ability or competency, may have an important interaction with gender differences in assessment. Identifying whether there are gender-based differences in how confidence is mentioned in written evaluations is a necessary step to understand the interaction between evaluation and the gender-based confidence gap.

OBJECTIVE

To analyze faculty evaluations of internal medicine (IM) residents for gender-based patterns in the use of iterations of "confidence."

DESIGN

We performed a retrospective cohort study of all inpatient faculty evaluations of University of Pennsylvania IM residents from 2018 to 2021. We performed n-gram text-mining to identify evaluations containing the terms "confident," "confidence," or "confidently." We performed univariable and multivariable logistic regression to determine the association between resident gender and references to confidence (including comments reflecting too little confidence), adjusting for faculty gender, post-graduate year (PGY), numeric rating, and service.

SUBJECTS

University of Pennsylvania IM residents from 2018 to 2021.

KEY RESULTS

There were 5416 evaluations of IM residents (165 women [51%], 156 men [49%]) submitted by 356 faculty members (149 women [51%]), of which 7.1 % (n=356) contained references to confidence. There was a significant positive association between the mention of confidence and women resident gender (OR 1.54, CI 1.23-1.92; p<0.001), which persisted after adjustment for faculty gender, numeric rating, and PGY level. Eighty evaluations of the cohort explicitly mentioned the resident having "too little confidence," which was also associated with women resident gender (OR 1.66, CI 1.05-2.62; p=0.031).

CONCLUSION

Narrative evaluations of women residents were more likely to contain references to confidence, after adjustment for numerical score, PGY level, and faculty gender, which may perpetuate the gender-based confidence gap, introduce bias, and ultimately impact professional identity development.

Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education

BACKGROUND

Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance.

OBJECTIVE

This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings.

METHODS

NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported.

RESULTS

The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%).

CONCLUSIONS

The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.

Role of Female Research at the Asociacion Mexicana de Cirugia General Annual Meeting: A Retrospective Analysis From 2013 to 2019

Background

Academic surgery has been a traditionally male-dominated field. Female contribution remains challenging. In Mexico, there is no published evidence regarding gender disparity in academic surgery. We aimed to analyze the female role in clinical research submitted to the Asociación Mexicana de Cirugía General (AMCG).

Methods

Retrospective study evaluating abstracts submitted to AMCG annual meetings from 2013 to 2019. Categorical variables were compared using χ² test. Univariate logistic regression was performed to calculate odds ratios (OR) followed by a log-binomial logistic regression model to obtain the adjusted relative risk (aRR) for acceptance as an oral presentation.

Results

Overall, 7,439 abstracts were analyzed of which 24.2% were submitted by females. Female-submitted abstracts increased from 22.5% to 25.3% during 2013–2019 (p = 0.15). The proportion of 47 abstracts submitted by females was higher in the resident group (27.7% vs. 18.8%; p < 0.001). The percentage of females’ abstracts selected for oral presentation was less than the percentage of males’ 49 abstracts selected for presentation (9% vs. 11.5%; p = 0.002). Females’ abstracts submitted have a 50 23.5% decreased chance of being selected for oral presentation (OR = 0.765, CI 95%, 0.639–0.917, 51 p = 0.003). However, after adjusting for research type and trainee status, the gender of the oral 52 presenting author showed no association (aRR = 0.95, CI 95%, 0.8–1.1, p = 0.56).

Conclusion

In Mexico, the female role in academic surgery is still limited. These results should 55 encourage professors and program directors to identify and address factors contributing to gender 56 disparities.

Holistic Review, Mitigating Bias, and Other Strategies in Residency Recruitment for Diversity, Equity, and Inclusion: An Evidence-based Guide to Best Practices from the Council of Residency Directors in Emergency Medicine

Advancement of diversity, equity, and inclusion (DEI) in emergency medicine can only occur with intentional recruitment of residency applicants underrepresented in medicine (UIM). Shared experiences from undergraduate and graduate medical education highlight considerations and practices that can contribute to improved diversity in the resident pool, such as holistic review and mitigating bias in the recruitment process. This review, written by members of the Council of Residency Directors in Emergency Medicine (CORD) Best Practices Subcommittee, offers best practice recommendations for the recruitment of UIM applicants. Recommendations address pre-interview readiness, interview approach, and post-interview strategies that residency leadership may use to implement holistic review and mitigate bias for recruitment of a diverse class.

Beyond the ratings: gender effects in written comments from clinical teaching assessments

Assessment of clinical teachers by learners is problematic. Construct-irrelevant factors influence ratings, and women teachers often receive lower ratings than men. However, most studies focus only on numeric scores. Therefore, the authors analyzed written comments on 4032 teacher assessments, representing 282 women and 448 men teachers in one Department of Medicine, to explore for gender differences. NVivo was used to search for 61 evidence- and theoretically-based terms purported to reflect teaching excellence, which were analyzed using 2 × 2 chi-squared tests. The Linguistic Index and Word Count (LIWC) was used to categorize comment data, which were analyzed using linear regressions. The only significant difference in NVivo was that men were more likely than women to have the word "available" in a comment (OR 1.4, p < .05). A subset of LIWC variables showed significant gender differences, but all effects were modest. Men teachers had more positive emotion words written about them, while negative emotion words appeared equally. Significant differences occurred more often between the men and women residents who wrote the comments, rather than those attributed to the gender of the teachers. For example, women residents used more social and gender-related words (β 1.87, p < 0.001) and fewer words related to power or achievement (β -3.78, p < 0.001) than men residents. Profound gender differences were not found in teacher assessment comments in this large, diverse academic department of medicine, which differs from other studies. The authors explore possible reasons including differences in departmental culture and issues related to the methods used.

Leadership Titles in Radiology: Usage of Non-Inclusive Terminology among Academic Radiology Departments and Societies

RATIONALE AND OBJECTIVES

To characterize the use of gender-specific (chairman) and non-inclusive terminology (chief) to describe departmental and divisional/sectional leadership within academic radiology department and radiological society websites.

METHODS

From a total of 157 unique institutions identified, we gathered a list of 123 departments with functioning websites. We screened for use of "chairman" and "chief" on departmental home pages, welcome messages, faculty listings, and residency program pages, noting location on websites, and alternative terms, if present. We also assessed for use of "chairman" through a list of 14 radiological society websites. The overall usage of these terms was determined and the relationships between geographic region, faculty gender, and use of terminology were assessed.

RESULTS

"Chairman" is common among radiology department websites, with 42.1% (51/121) of websites using the term at least once. Usage of "chairman" was higher in departments whose chairs were men (OR 4.32, 95% CI: 1.36-13.69) and lower in those located in the Midwest (versus Northeast, OR 0.34, 95% CI: 0.12-0.96). Use of "chief" is extremely common among departments listing divisional leadership, at 82.7% (67/81). Alternative terms used included "chair," "head," "chairperson," and "director." Among departments, the proportion of women in leadership roles is low, representing 19.7% (22/122) of chairs and 31.1% (272/874) of other leadership. Radiological societies were unlikely to use "chairman," with only one website using the term.

CONCLUSION

Use of "chairman" and "chief" is frequent among academic radiology departments. Further efforts should be made to assess the usage of non-inclusive terminology and its potential impact within academic radiology departments.

Gender-Based Linguistic Analysis of Pediatric Clinical Faculty Evaluations

BACKGROUND

Gendered stereotypes are embedded in the culture of medicine. Women are stereotypically expected to act collaboratively and less assertively, while men are expected to act with authority and power. Whether gender-biased language is expressed in academic pediatric teaching evaluations is unknown.

OBJECTIVE

Determine whether stereotypic gender-based linguistic differences exist in resident evaluations of pediatric faculty.

METHODS

We performed a retrospective cross-sectional study of clinical faculty evaluations by pediatric residents in a single program from July 2016 to June 2019. Using Linguistic Inquiry and Word Count, responses to 2 open-ended questions were analyzed for stereotypic language. Categories were reported as a percent of total words written. Comparisons between gender groups were conducted using nonparametric Wilcoxon rank sum tests. Rates of word use within each category were analyzed using logistic regression where faculty and resident gender were included as predictor variables.

RESULTS

A total of 6436 free-text responses from 3218 unique evaluations were included. As hypothesized, evaluations of women faculty were less likely than those of men to include certain agentic language like power (odds ratio [OR] 0.9, P < .001) and insight (OR 0.9, P < .001), and research words (OR 0.6, P = .003). As expected, evaluations of women were more likely to include grindstone words, like "hardworking" (OR 1.2, P = .012). Contrary to our hypothesis, women received fewer teaching words like "mentor" (OR 0.9, P = .048) and communal words like "friendly" (OR 0.6, P = .001).

CONCLUSION

Certain stereotypic language was demonstrated in clinical teaching evaluations of pediatric faculty. These findings should be further examined to improve gender inequities in academic pediatrics.

The effect of gender dyads on the quality of narrative assessments of general surgery trainees

BACKGROUND

Prior studies have shown that gender can influence how learners are assessed and the feedback they receive. We investigated the quality of faculty narrative comments in general surgery trainee evaluation using trainee-assessor gender dyads.

METHODS

Narrative assessments of surgical trainees at the University of British Columbia were collected and rated using the McMaster Narrative Comment Rating Scale (MNCRS). Variables from the MNCRS were inputted into a generalized linear mixed model to explore the impact of gender dyads on the quality of narrative feedback.

RESULTS

2,469 assessments were collected. Women assessors tended to give higher-quality comments (p's < 0.05) than men assessors. Comments from men assessors to women trainees were significantly more positive than comments from men assessors to men trainees (p = 0.02). Men assessors also tended to give women trainees more reinforcing than corrective comments than to men trainees (p < 0.01).

CONCLUSIONS

There are significant differences in the quality of faculty feedback to trainees by gender dyads. A range of solutions to improve and reduce differences in feedback quality are discussed.

Professional Address during Obstetrics and Gynecology Grand Rounds Introductions: Setting the Stage, Setting the Standard

OBJECTIVE

This study aimed to assess gender dynamics during Obstetrics and Gynecology (Ob/Gyn) Grand Rounds.

DESIGN

This was an observational cohort study of Ob/Gyn Grand Rounds introductions at a large academic center. Ob/Gyn Grand Rounds introductions from December 2016 to February 2020 were included. Audio and video components of introductions for those with doctorate degrees were reviewed. Each named reference to the presenter and use of descriptors were collected. Statistical analyses included Fisher's exact test for categorical variables and Student's t-test for continuous variables.

SETTING

This study was completed at the University of Wisconsin in the Department of Ob/Gyn PARTICIPANTS: Ob/Gyn Grand Rounds introducers who had complete audio and video components of introductions for those with doctorate degrees.

RESULTS

Sixty-four Grand Rounds introductions were reviewed; 57 met inclusion criteria. The majority of introducers and presenters were women. Consistent use of "doctor" was similar by men and women introducers (50% vs. 29%, p = 0.427). Assistant professors were more likely to maintain professional address during introductions, compared to associate or full professors (86% vs. 0% vs. 10%, p < 0.001). Trainees were less likely than faculty to be addressed professionally at any time during introductions (42% vs. 81%, p = 0.017). Descriptors were used for men and women presenters, though men received more female-gendered descriptors than women (5 vs. 1, p = 0.011). Women introducers used productivity descriptors less often than men introducers (8 [15.1%] vs. 5 [55.6%] (p = 0.015)).

CONCLUSIONS

Use of professional address was associated with academic rank, but not gender. Men endorsed and received more descriptors emphasizing accomplishments, highlighting qualifications as an expert. Given the professional environment, all Grand Rounds presenters should be introduced using professional titles.

Gendered Differences in Teaching Performance Evaluations of Obstetrics and Gynecology Residents

OBJECTIVE

Teaching performance evaluations are commonly used for career development and advancement. Due to possible gendered expectations, implicit or explicit bias may emerge in evaluations completed by learners. This study investigated how third-year medical students evaluated teaching performance of obstetrics and gynecology resident physicians based on resident gender.

SETTING

This study was conducted at the Medical College of Wisconsin in Milwaukee, Wisconsin.

DESIGN

This retrospective mixed methods study examined teaching performance evaluations of obstetrics and gynecology resident physicians from 2010 to 2018, completed by third-year medical students. A two-sample, two-sided t-test was used to compare numerical scores. Deductive content analysis of written comments focused on specific categories: positive or negative agentic or communal demeanors and characteristics, teaching skills, character and professionalism, leadership abilities, clinical skills and knowledge, and frequency of words and phrases used to describe residents.

SETTING

This study was conducted at the Medical College of Wisconsin in Milwaukee, Wisconsin.

RESULTS

Of 83 residents (71 females, 12 males), there was no statistical significance in the teaching performance evaluation scores between male and female residents (n = 10,753 total completed evaluations). Female residents had lower scores than male residents; males tended not to score below 4 (5-point response scale; 5 = outstanding). Of 3,813 written comments, male residents had more positive comments, with statistical significance in communal characteristics (71.4% male, 53.9% female, p = 0.01). Female residents received more negative comments, with statistical significance in communal characteristics (7.5% female, 2.8% male, p = 0.01). Frequency of words presented that male residents had more "standout" traits ("outstanding," "excellent," "exemplary"), "ability" terms ("intelligent," "bright," "talented," "smart"), and were often considered "fun," "funny," and "humorous." Female residents were described by "compassion" terms ("kind," "compassionate").

CONCLUSION

Student-completed teaching performance evaluations are a valuable assessment of teaching skills and influence department recognition, award distribution, fellowship and employment opportunities. This study found that medical students did evaluate female residents differently than male residents. Understanding gendered expectations may assist in findings ways to address discrepancies between male and female physician evaluations.

Impact of Gender on Clinical Evaluation of Trainees in the Intensive Care Unit

Background: Gender disparities in medical education are increasingly demonstrated, including in trainee assessment.

Objective: This study aimed to evaluate whether gender differences exist in trainees’ evaluation during intensive care unit (ICU) rotations, which has not been previously studied.

Methods: We reviewed the in-training evaluation reports (ITERs) for trainees rotating through five academic ICUs at the University of Toronto over a 10-year period (2007–2017). We compared the mean global score for the rotation and the mean score for seven training subdomains between men and women trainees. All scores were reported on a scale of 1 (unsatisfactory) to 5 (outstanding).

Results: Over the 10-year period, there were 3,203 ITERS overall, representing 1,207 women and 1,996 men trainees. The mean overall score was lower for women than for men trainees: 4.26 (standard deviation [SD], 0.58) for women and 4.30 (SD, 0.60) for men (P = 0.04). This difference was driven by anesthesia trainees, in whom the mean overall score was 4.21 for women and 4.37 for men (P < 0.001), with men trainees scoring consistently higher across all seven training subdomains. Within surgical, internal medicine, and critical care residents, there were no differences between men and women in the overall score or the scores across any of the seven subdomains. Across all ITERS, women were less likely than men to receive an overall rating of 5 (outstanding) for the ICU rotation (33% women vs. 37% men; odds ratio, 0.83; 95% confidence interval, 0.71–0.96).

Conclusion: Overall, quantitative evaluation scores between women and men trainees in the ICU are relatively similar. Within anesthesia trainees, scores for men were consistently higher across all domains of evaluation, a finding that requires further investigation.

Assessment of Gender-based Qualitative Differences within Trainee Evaluations of Faculty

Rationale: Recent studies of trainee evaluations of medical faculty have demonstrated subtle gender-based word choice differences. However, it is not known if this manifests in major contextual differences in written comments.Objectives: To characterize qualitative differences in narrative evaluations of female and male pulmonary and critical care medicine faculty.Methods: We performed a single-center, retrospective cohort analysis of trainee narrative evaluations of pulmonary and critical care medicine faculty at the Hospital of the University of Pennsylvania, written from 2015 to 2016. Directive and summative content analysis was performed by four raters. Major contextual themes were identified using constant comparative techniques. Thematic differences based on faculty gender were identified, with statistical significance determined using χ² analysis (P < 0.05). Effect sizes were calculated using Cramér's V.Results: A total of 1,216 total narrative evaluations were analyzed, representing 62 faculty members (17 women and 45 men), with analysis continued for the full cohort beyond achieving saturation of themes. Five overarching themes emerged: teaching skills, clinical skills, supervision, interpersonal and communication skills, and leadership skills. Within subthemes, we found no significant gender differences in reference to general teaching skills, learning environment, enthusiasm for teaching, or interpersonal concern. We identified subtheme differences between male and female faculty evaluations in regard to mentions of learner autonomy, clinical learning environment, humor, and motivating the trainee for patient care. Although the mention of constructive criticisms did not differ between genders, constructive criticisms of female faculty were more likely to mention demeanor (P = 0.06).Conclusions: Our data suggest minimal thematic differences in trainee narrative evaluations of male and female pulmonary and critical care medicine faculty. However, we noted several subtheme contextual differences between male and female faculty, which warrants further investigation.

Diversity in Neurosurgery

Over the past century, the field of neurosurgery has evolved and expanded in various directions. Neurosurgeons have continuously pushed the boundaries of the specialty with scientific discovery and innovation. A diverse array of neurosurgical techniques, treatments, and new areas of study have emerged within the field. Meanwhile, the neurosurgical workforce has stayed demographically homogeneous throughout time. Certain groups remain underrepresented owing to systemic barriers based on social identities and categorizations, including gender, race, ability status, and others. In this article, we highlight some of the underrepresented groups in neurosurgery and chronicle the important contributions and achievements that individuals from these groups have made in the field despite structural barriers and discrimination. We present evidence from the basic sciences, economics, business, and other disciplines that illustrate that diversity is not only just but also the most rational pursuit for positive growth and advancement. Diversity is needed to enrich the specialty and augment its capacity to serve the heterogeneous population of patients that reflect our society. To promote equity and diversity in the field, ongoing deliberate, organized, and systematic efforts to change the status quo and make the field more inclusive are needed.

Gender Effects in Assessment of Clinical Teaching: Does Concordance Matter?

BACKGROUND

Gender bias is thought to exist in the assessment of clinical teachers, yet its extent in different specialties is not well-documented nor has it been studied at the individual-dyadic level.

OBJECTIVE

The authors sought to determine whether gender bias exists in residents' assessments of faculty teaching in 3 clinical departments, and if present, whether this is influenced by gender concordance or discordance between the faculty and resident.

METHODS

Residents' ratings of faculty in internal medicine (800 faculty, 5753 ratings), surgery (377, 2249), and family medicine (672, 3438) at the University of Toronto from 2016-2017 were analyzed using the overall global rating on a 5-point scale. A mixed-effects linear regression analysis accounted for nesting of ratings within each faculty member.

RESULTS

Overall scores of teaching effectiveness showed a strong skew to favorable ratings for all faculty and a ceiling effect. However, gender effects differed across departments. In internal medicine (38.5% female faculty), no significant gender effects were detected. In surgery (16.2% female) and family medicine (53.0% female), male faculty received significantly higher scores than female faculty. In surgery this was driven by male residents giving male faculty higher ratings (4.46 vs 4.26, P < .001). In family medicine this was driven by male faculty receiving higher ratings regardless of resident gender (4.65 to 4.57, P < .001).

CONCLUSIONS

Although effects were very small and inconsistent, with gender concordance mattering only for one department, it suggests that gender is a meaningful source of variance in teaching assessments.

Alpha Omega Alpha in the 21st Century: One School's Approach

Recent data suggest that students from population groups that have been underrepresented in medicine are disproportionately excluded from admission into the national medical honor society, Alpha Omega Alpha (AΩA). This finding, in combination with increasing concerns about bias in medical student assessment, has led some medical schools to reexamine their AΩA selection process and/or their relationship with the organization. The Pritzker School of Medicine at the University of Chicago formed a task force to study the schools process of choosing students for recognition and to make recommendations regarding this issue.

Association between patient-physician gender concordance and patient experience scores. Is there gender bias?

BACKGROUND

Patient satisfaction, a commonly measured indicator of quality of care and patient experience, is often used in physician performance reviews and promotion decisions. Patient satisfaction surveys may introduce gender-related bias.

OBJECTIVE

Examine the effect of patient and physician gender concordance on patient satisfaction with emergency care.

METHODS

We performed a cross-sectional analysis of electronic health record and Press Ganey patient satisfaction survey data of adult patients discharged from the emergency department (2015-2018). Logistic regression models were used to examine relationships between physician gender, patient gender, and physician-patient gender dyads. Binary outcomes included: perfect care provider score and perfect overall assessment score.

RESULTS

Female patients returned surveys more often (n=7 612; 61.55%) and accounted for more visits (n=232 024; 55.26%). Female patients had lower odds of perfect scores for provider score and overall assessment score (OR: 0.852, 95% CI: 0.790, 0.918; OR: 0.782, 95% CI: 0.723, 0.846). Female physicians had 1.102 (95% CI: 1.001, 1.213) times the odds of receiving a perfect provider score. Physician gender did not influence male patients' odds of reporting a perfect care provider score (95% CI: 0.916, 1.158) whereas female patients treated by female physicians had 1.146 times the odds (95% CI: 1.019, 1.289) of a perfect provider score.

CONCLUSION

Female patients prefer female emergency physicians but were less satisfied with their physician and emergency department visit overall. Over-representation of female patients on patient satisfaction surveys introduces bias. Patient satisfaction surveys should be deemphasized from physician compensation and promotion decisions.

Unconscious Gender Bias in Academia: Scarcity of Empirical Evidence

Implicit or unconscious bias is commonly proposed to be responsible for women’s underrepresentation in academia. The aim of this scoping review was to identify and discuss the evidence supporting this proposition. Publications about unconscious/implicit gender bias in academia indexed in Scopus or psycInfo up to February 2020 were identified. More than half were published in the period 2018–2020. Studies reporting empirical data were scrutinized for data, as well as analyses showing an association of a measure of implicit or unconscious bias and lesser employment or career opportunities in academia for women than for men. No studies reported empirical evidence as thus defined. Reviews of unconscious bias identified via informal searches referred exclusively to studies that did not self-identify as addressing unconscious bias. Reinterpretations and misrepresentations of studies were common in these reviews. More empirical evidence about unconscious gender bias in academia is needed. With the present state of knowledge, caution should be exercised when interpreting data about gender gaps in academia. Ascribing observed gender gaps to unconscious bias is unsupported by the scientific literature.

The Standardized Letter of Evaluation Narrative: Differences in Language Use by Gender

Introduction

Prior research demonstrates gender differences in language used in letters of recommendation. The emergency medicine (EM) Standardized Letter of Evaluation (SLOE) format limits word count and provides detailed instructions for writers. The objective of this study is to examine differences in language used to describe men and women applicants within the SLOE narrative.

Methods

All applicants to a four-year academic EM residency program within a single application year with a first rotation SLOE available were included in the sample. We used the Linguistic Inquiry and Word Count (LIWC) program to analyze word frequency within 16 categories. Descriptive statistics, chi-squared, and t-tests were used to describe the sample; gender differences in word frequency were tested for using Mann-Whitney U tests.

Results

Of 1117 applicants to the residency program, 822 (82%) first-rotation SLOEs were available; 64% were men, and 36% were women. We did not find a difference in baseline characteristics including age (mean 27 years), top 25 schools (22.5%), Alpha Omega Alpha Honor Medical Society rates (13%), and having earned advanced degrees (10%). The median word count per SLOE narrative for men was 171 and for women was 180 (p = 0.15). After adjusting for letter length, word frequency differences between genders were only present in two categories: social words (women: 23 words/letter; men: 21 words/letter, p = 0.02) and ability words (women: 2 words/letter; men: 1 word/letter, p = 0.04). We were unable to detect a statistical difference between men and women applicants in the remaining categories, including words representing communal traits, agentic traits, standout adjectives, grindstone traits, teaching words, and research words.

Conclusion

The small wording differences between genders noted in two categories were statistically significant, but of unclear real-world significance. Future work is planned to evaluate how the SLOE format may contribute to this relative lack of bias compared to other fields and formats.