Gender Differences in Milestone Ratings and Medical Knowledge Examination Scores Among Internal Medicine Residents

Internal Medicine Resident Barriers to Advance Care Planning in the Primary Care Continuity Clinic

Background: While primary care providers regularly engage in Advance Care Planning (ACP) conversations, it is not well known what challenges resident physicians face to achieving this core competency. Objectives: We aimed to assess resident perceptions of barriers and potential interventions to outpatient ACP. Methods: We distributed an electronic survey to Internal Medicine and Medicine-Psychiatry residents at our institution in 2022. Questions addressed outpatient ACP barriers and potential interventions in several domains: structural issues, personal knowledge, and communication skills. We reported results using descriptive statistics and Wilcoxon rank-sum tests, comparing responses by residency year (interns vs upperyears). Likert-scale responses were dichotomized to a "not at all or slightly" vs "moderate or extreme" barrier or helpful intervention. Results: Of 149 residents, 71 completed the survey (48%). Highest scoring barriers were structural, including 1) lack of clinic time (99%), 2) need to prioritize other medical problems (94%), and 3) lack of patient continuity (62%). Highest scoring interventions included the ability to schedule dedicated ACP visits with themselves (96%) or another clinician (82%). Interns were statistically significantly less confident in their ability to conduct ACP, and more likely to report lack of knowledge (i.e., not understanding ACP, patient prognosis, or how to complete paperwork, P < .05). Conclusions: Residents report significant structural barriers to outpatient ACP, including limitations in time, continuity, and competing medical priorities, that may warrant greater program attention to interventions such as clinic schedules and work-flow. Additional trainings may be most beneficial if targeted to the beginning of intern year.

Gender Bias and Ophthalmology Accreditation Council for Graduate Medical Education Milestones Evaluations

Importance

Women remain underrepresented in ophthalmology and gender-based disparities exist in salary, grant receipt, publication rates, and surgical volume throughout training and in practice. Although studies in emergency medicine and general surgery showed mixed findings regarding gender differences in Accreditation Council for Graduate Medical Education (ACGME) Milestones ratings, limited data exist examining such differences within ophthalmology.

Objective

To examine gender differences in ophthalmology ACGME Milestones.

Design, Setting, and Participants

This was a retrospective cross-sectional study of postgraduate year 4 (PGY-4) residents from 120 ophthalmology programs graduating in 2019.

Main Outcomes and Measures

PGY-4 midyear and year-end medical knowledge (MK) and patient care (PC) ratings and Written Qualifying Examination (WQE) scaled scores for residents graduating in 2019 were included. Differential prediction techniques using Generalized Estimating Equations models were performed to identify differences by gender.

Results

Of 452 residents (median [IQR] age, 30.0 [29.0-32.0] years), 275 (61%) identified as men and 177 (39%) as women. There were no differences in PC domain average between women and men for both midyear (-0.07; 95% CI, -0.11 to 0; P =.06) and year-end (-0.04; 95% CI, -0.07 to 0.03; P =.51) assessment periods. For the MK domain average in the midyear assessment period, women (mean [SD], 3.76 [0.50]) were rated lower than men (mean [SD], 3.88 [0.47]; P = .006) with a difference in mean of -0.12 (95% CI, -0.18 to -0.03). For the year-end assessment, however, the average MK ratings were not different for women (mean [SD], 4.10 [0.47]) compared with men (mean [SD], 4.18 [0.47]; P = .20) with a difference in mean of -0.08 (95% CI, -0.13 to 0.03).

Conclusions and Relevance

Results suggest that ACGME ophthalmology Milestones in 2 general competencies did not demonstrate major gender bias on a national level at the time of graduation. There were, however, differences in MK ratings at the midyear mark, and as low ratings on evaluations and examinations may adversely affect career opportunities for trainees, it is important to continue further work examining other competencies or performance measures for potential biases.

Intersectional Disparities in Emergency Medicine Residents' Performance Assessments by Race, Ethnicity, and Sex

Importance

Previous studies have demonstrated sex-specific disparities in performance assessments among emergency medicine (EM) residents. However, less work has focused on intersectional disparities by ethnoracial identity and sex in resident performance assessments.

Objective

To estimate intersectional sex-specific ethnoracial disparities in standardized EM resident assessments.

Design, Setting, and Participants

This retrospective cohort study used data from the Association of American Medical Colleges and the Accreditation Council for Graduate Medical Education Milestones (Milestones) assessments to evaluate ratings for EM residents at 128 EM training programs in the US. Statistical analyses were conducted in June 2020 to January 2023.

Exposure

Training and assessment environments in EM residency programs across comparison groups defined by ethnoracial identity (Asian, White, or groups underrepresented in medicine [URM], ie, African American/Black, American Indian/Alaska Native, Hispanic/Latine, and Native Hawaiian/Other Pacific Islander) and sex (female/male).

Main Outcomes and Measures

Mean Milestone scores (scale, 0-9) across 6 core competency domains: interpersonal and communications skills, medical knowledge, patient care, practice-based learning and improvement, professionalism, and system-based practice. Overall assessment scores were calculated as the mean of the 6 competency scores.

Results

The study sample comprised 128 ACGME-accredited programs and 16 634 assessments for 2708 EM residents of which 1913 (70.6%) were in 3-year and 795 (29.4%) in 4-year programs. Most of the residents were White (n = 2012; 74.3%), followed by Asian (n = 477; 17.6%), Hispanic or Latine (n = 213; 7.9%), African American or Black (n = 160; 5.9%), American Indian or Alaska Native (n = 24; 0.9%), and Native Hawaiian or Other Pacific Islander (n = 4; 0.1%). Approximately 14.3% (n = 386) and 34.6% (n = 936) were of URM groups and female, respectively. Compared with White male residents, URM female residents in 3-year programs were rated increasingly lower in the medical knowledge (URM female score, -0.47; 95% CI, -0.77 to -0.17), patient care (-0.18; 95% CI, -0.35 to -0.01), and practice-based learning and improvement (-0.37; 95% CI, -0.65 to -0.09) domains by postgraduate year 3 year-end assessment; URM female residents in 4-year programs were also rated lower in all 6 competencies over the assessment period.

Conclusions and Relevance

This retrospective cohort study found that URM female residents were consistently rated lower than White male residents on Milestone assessments, findings that may reflect intersectional discrimination in physician competency evaluation. Eliminating sex-specific ethnoracial disparities in resident assessments may contribute to equitable health care by removing barriers to retention and promotion of underrepresented and minoritized trainees and facilitating diversity and representation among the emergency physician workforce.

"My Assessments Are Biased!" Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education

Assessing learners is foundational to their training and developmental growth throughout the medical education continuum. However, growing evidence shows the prevalence and impact of harmful bias in assessments in medical education, accelerating the urgency to identify solutions. Assessment bias presents a critical problem for all stages of learning and the broader educational system. Bias poses significant challenges to learners, disrupts the learning environment, and threatens the pathway and transition of learners into health professionals. While the topic of assessment bias has been examined within the context of measurement literature, limited guidance and solutions exist for learners in medical education, particularly in the clinical environment. This article presents an overview of assessment bias, focusing on clinical learners. A definition of bias and its manifestations in assessments are presented. Consequences of assessment bias are discussed within the contexts of validity and fairness and their impact on learners, patients/caregivers, and the broader field of medicine. Messick's unified validity framework is used to contextualize assessment bias; in addition, perspectives from sociocultural contexts are incorporated into the discussion to elaborate the nuanced implications in the clinical training environment. Discussions of these topics are conceptualized within the literature and the interventions used to date. The article concludes with practical recommendations to overcome bias and to develop an ideal assessment system. Recommendations address articulating values to guide assessment, designing assessment to foster learning and outcomes, attending to assessment procedures, promoting continuous quality improvement of assessment, and fostering equitable learning and assessment environments.

Association of Gender and Race/Ethnicity with Internal Medicine In-Training Examination Performance in Graduate Medical Education

BACKGROUND

Disparities in objective assessments in graduate medical education such as the In-Training Examination (ITE) that disadvantage women and those self-identifying with race/ethnicities underrepresented in medicine (URiM) are of concern.

OBJECTIVE

Examine ITE trends longitudinally across post-graduate year (PGY) with gender and race/ethnicity.

DESIGN

Longitudinal analysis of resident ITE metrics at 7 internal medicine residency programs, 2014-2019. ITE trends across PGY of women and URiM residents compared to non-URiM men assessed via ANOVA. Those with ITE scores associated with less than 90% probability of passing the American Board of Internal Medicine certification exam (ABIM-CE) were identified and odds of being identified as at-risk between groups were assessed with chi square.

PARTICIPANTS

A total of 689 IM residents, including 330 women and URiM residents (48%).

MAIN MEASURES

ITE score KEY RESULTS: There was a significant difference in ITE score across PGY for women and URiM residents compared to non-URiM men (F_{(2, 1321)} 4.46, p=0.011). Adjusting for program, calendar year, and baseline ITE, women and URiM residents had smaller ITE score gains (adjusted mean change in score between PGY1 and PGY3 (se), non-URiM men 13.1 (0.25) vs women and URiM residents 11.4 (0.28), p<0.001). Women and URiM residents had greater odds of being at potential risk for not passing the ABIM-CE (OR 1.75, 95% CI 1.10 to 2.78) with greatest odds in PGY3 (OR 3.13, 95% CI 1.54 to 6.37).

CONCLUSION

Differences in ITE over training were associated with resident gender and race/ethnicity. Women and URiM residents had smaller ITE score gains across PGY translating into greater odds of potentially being seen as at-risk for not passing the ABIM-CE. Differences in ITE over training may reflect differences in experiences of women and URiM residents during training and may lead to further disparities.

Gender bias in resident evaluations: Natural language processing and competency evaluation

BACKGROUND

Research shows that female trainees experience evaluation penalties for gender non-conforming behaviour during medical training. Studies of medical education evaluations and performance scores do reflect a gender bias, though studies are of varying methodology and results have not been consistent.

OBJECTIVE

We sought to examine the differences in word use, competency themes and length within written evaluations of internal medicine residents at scale, considering the impact of both faculty and resident gender. We hypothesised that female internal medicine residents receive more negative feedback, and different thematic feedback than male residents.

METHODS

This study utilised a corpus of 3864 individual responses to positive and negative questions over the course of six years (2012-2018) within Yale University School of Medicine's internal medicine residency. Researchers developed a sentiment model to assess the valence of evaluation responses. We then used natural language processing (NLP) to evaluate whether female versus male residents received more positive or negative feedback and if that feedback focussed on different Accreditation Council for Graduate Medical Education (ACGME) core competencies based on their gender. Evaluator-evaluatee gender dyad was analysed to see how it impacted quantity and quality of feedback.

RESULTS

We found that female and male residents did not have substantively different numbers of positive or negative comments. While certain competencies were discussed more than others, gender did not seem to influence which competencies were discussed. Neither gender trainee received more written feedback, though female evaluators tended to write longer evaluations.

CONCLUSIONS

We conclude that when examined at scale, quantitative gender differences are not as prevalent as has been seen in qualitative work. We suggest that further investigation of linguistic phenomena (such as context) is warranted to reconcile this finding with prior work.

Effect of resident and assessor gender on entrustment-based observational assessment in an internal medicine residency program

Background: Implicit gender bias leads to differences in assessment. Studies examining gender differences in resident milestone assessment data demonstrate variable results. The purpose of this study was to determine if observational entrustment scores differ by resident and assessor gender in a program of assessment based on discrete, observable skills.  Methods: We analyzed overall entrustment scores and entrustment scores by Accreditation Council for Graduate Medical Education (ACGME) core competency for 238 residents (49% female) from 396 assessors (38% female) in one internal medicine residency program from July 2012 to June 2019. We conducted analyses at 1-12 months, 1-36 months, 1-6 months, 7-12 months, and 31-36 months. We used linear mixed-effect models to assess the role of resident and assessor gender, with resident-specific and assessor-specific random effect to account for repeated measures.  Results: Statistically significant interactions existed between resident and assessor gender for overall entrustment at 1-12 months (p < 0.001), 1-36 months (p< 0.001), 1-6 months (p<0.001), 7-12 months (p=0.04), and 31-36 months (p<0.001). However, group differences were not statistically significant. In several instances an interaction was significant between resident and assessor gender by ACGME core competency, but there were no statistically significant group differences for all competencies at any time point. When applicable, subsequent analysis of main effect of resident or assessor gender independently of one another revealed no statistically significant differences.   Conclusions: No significant differences in entrustment scores were found based on resident or assessor gender in our large, robust entrustment-based program of assessment. Determining the reasons for our findings may help identify ways to mitigate gender bias in assessment.