Comparison of Male and Female Resident Milestone Assessments During Emergency Medicine Residency Training: A National Study

Emergency Medicine Milestones Final Ratings Are Often Subpar

Background

The emergency medicine (EM) milestones are objective behaviors that are categorized into thematic domains called "subcompetencies" (eg, emergency stabilization). The scale for rating milestones is predicated on the assumption that a rating (level) of 1.0 corresponds to an incoming EM-1 resident and a rating of 4.0 is the "target rating" (albeit not an expectation) for a graduating resident. Our aim in this study was to determine the frequency with which graduating residents received the target milestone ratings.

Methods

This retrospective, cross-sectional study was a secondary analysis of a dataset used in a prior study but was not reported previously. We analyzed milestone subcompetency ratings from April 25-June 24, 2022 for categorical EM residents in their final year of training. Ratings were dichotomized as meeting the expected level at the time of program completion (ratings of ≥3.5) and not meeting the expected level at the time of program completion (ratings of ≤3.0). We calculated the number of residents who did not achieve target ratings for each of the subcompetencies.

Results

In Spring 2022, of the 2,637 residents in the spring of their last year of training, 1,613 (61.2%) achieved a rating of ≥3.5 on every subcompetency and 1,024 (38.8%) failed to achieve that rating on at least one subcompetency. There were 250 residents (9.5%) who failed to achieve half of their expected subcompetency ratings and 105 (4.0%) who failed to achieve the expected rating (ie, rating was ≤3.0) on every subcompetency.

Conclusion

When using an EM milestone rating threshold of 3.5, only 61.2% of physicians achieved the target ratings for program graduation; 4.0% of physicians failed to achieve target ratings for any milestone subcompetency; and 9.5% of physicians failed to achieve the target ratings for graduating residents in half of the subcompetencies.

Race and Gender Differences in Pediatric Milestone Levels: A Multi-Institutional Study

The Accreditation Council for Graduate Medical Education milestones assess resident competency in 6 domains. We hypothesized that disparities in milestones exist across race and gender in pediatric residencies. This is a retrospective, cross-sectional, multi-institutional study (3 pediatric residencies, 1446 scores; 316 residents). African American residents received the lowest scores in patient care (PC) (P = .030), medical knowledge (MK) (P = .005), practice-based learning and improvement (PBLI) (P = .003), professionalism (PROF) (P < .001), and interpersonal communication skills (ICS) (P = .005). Differences were most pronounced in PROF (African American mean 3.35 [SD .75], Asian 3.51 (.66), Hispanic 3.58 (.66), white 3.59 (.67)). Female residents received higher scores than male residents in PC (P = .002) and system-based practice (SBP) (P = .049). Female interns received higher MK scores, 2.53 (.44) versus 2.48 (.48), P = .044, but lower scores as third years, 4.00 (.43) versus 4.14 (.45), P = .030. In this study, pediatric milestones differed based on race and gender.

Assessing Pediatric Rheumatology Fellow Competence in the Milestone Era: Past, Present, and Future

Starting in 2015, pediatric rheumatology fellowship training programs were required by the Accreditation Council for Graduate Medical Education to assess fellows' academic performance within 21 subcompetencies falling under six competency domains. Each subcompetency had four or five milestone levels describing developmental progression of knowledge and skill acquisition. Milestones were standardized across all pediatric subspecialties. As part of the Milestones 2.0 revision project, the Accreditation Council for Graduate Medical Education convened a workgroup in 2022 to write pediatric rheumatology-specific milestones. Using adult rheumatology's Milestones 2.0 as a starting point, the workgroup revised the patient care and medical knowledge subcompetencies and milestones to reflect requirements and nuances of pediatric rheumatology care. Milestones within four remaining competency domains (professionalism, interpersonal and communication skills, practice-based learning and improvement, and systems-based practice) were standardized across all pediatric subspecialties, and therefore not revised. The workgroup created a supplemental guide with explanations of the intent of each subcompetency, 25 in total, and examples for each milestone level. The new milestones are an important step forward for competency-based medical education in pediatric rheumatology. However, challenges remain. Milestone level assignment is meant to be informed by results of multiple assessment methods. The lack of pediatric rheumatology-specific assessment tools typically result in clinical competency committees determining trainee milestone levels without such collated results as the foundation of their assessments. Although further advances in pediatric rheumatology fellowship competency-based medical education are needed, Milestones 2.0 importantly establishes the first pediatric-specific rheumatology Milestones to assess fellow performance during training and help measure readiness for independent practice.

Stereotype Threat and Gender Bias in Internal Medicine Residency: It is Still Hard to be in Charge

BACKGROUND

Despite similar numbers of women and men in internal medicine (IM) residency, women face unique challenges. Stereotype threat is hypothesized to contribute to underrepresentation of women in academic leadership, and exploring how it manifests in residency may provide insight into forces that perpetuate gender disparities.

OBJECTIVE

To quantify the prevalence of stereotype threat in IM residency and explore experiences contributing to that stereotype threat.

DESIGN

We used a mixed methods study design. First, we surveyed IM residents using the Stereotype Vulnerability Scale (SVS) to screen for stereotype threat. Second, we conducted focus groups with women who scored high on the SVS to understand experiences that led to stereotype threat.

PARTICIPANTS

The survey was sent to all IM residents at University of California, San Francisco (UCSF), in September-November 2019. Focus groups were conducted at UCSF in Spring 2020.

APPROACH

The survey included an adapted version of the SVS. For focus groups, we developed a focus group guide informed by literature on stereotype threat. We used a thematic approach to data analysis. The mixed methods design enabled us to draw metainferences by integrating the two data sources.

KEY RESULTS

Survey response rate was 61% (110/181). Women were significantly more likely than men to have a score indicating stereotype threat vulnerability (77% vs 0%, p < 0.001). Four themes from focus groups characterized women's experiences of gender bias and stereotype threat: gender norm tension, microaggressions and sexual harassment, authority questioned, and support and allyship.

CONCLUSIONS

Gender-based stereotype threat is highly prevalent among women IM residents. This phenomenon poses a threat to confidence and ability to execute patient care responsibilities, detracting from well-being and professional development. These findings indicate that, despite robust representation of women in IM training, further attention is needed to address gendered experiences and contributors to women's vulnerability to stereotype threat.

An Automated System for Physician Trainee Procedure Logging via Electronic Health Records

Importance

Procedural proficiency is a core competency for graduate medical education; however, procedural reporting often relies on manual workflows that are duplicative and generate data whose validity and accuracy are difficult to assess. Failure to accurately gather these data can impede learner progression, delay procedures, and negatively impact patient safety.

Objective

To examine accuracy and procedure logging completeness of a system that extracts procedural data from an electronic health record system and uploads these data securely to an application used by many residency programs for accreditation.

Design, Setting, and Participants

This quality improvement study of all emergency medicine resident physicians at University of California, San Diego Health was performed from May 23, 2023, to June 25, 2023.

Exposures

Automated system for procedure data extraction and upload to a residency management software application.

Main Outcomes and Measures

The number of procedures captured by the automated system when running silently compared with manually logged procedures in the same timeframe, as well as accuracy of the data upload.

Results

Forty-seven residents participated in the initial silent assessment of the extraction component of the system. During a 1-year period (May 23, 2022, to May 7, 2023), 4291 procedures were manually logged by residents, compared with 7617 procedures captured by the automated system during the same period, representing a 78% increase. During assessment of the upload component of the system (May 8, 2023, to June 25, 2023), a total of 1353 procedures and patient encounters were evaluated, with the system operating with a sensitivity of 97.4%, specificity of 100%, and overall accuracy of 99.5%.

Conclusions and Relevance

In this quality improvement study of emergency medicine resident physicians, an automated system demonstrated that reliance on self-reported procedure logging resulted in significant procedural underreporting compared with the use of data obtained at the point of performance. Additionally, this system afforded a degree of reliability and validity heretofore absent from the usual after-the-fact procedure logging workflows while using a novel application programming interface-based approach. To our knowledge, this system constitutes the first generalizable implementation of an automated solution to a problem that has existed in graduate medical education for decades.

The Quality of Assessment for Learning score for evaluating written feedback in anesthesiology postgraduate medical education: a generalizability and decision study

Background

Competency based residency programs depend on high quality feedback from the assessment of entrustable professional activities (EPA). The Quality of Assessment for Learning (QuAL) score is a tool developed to rate the quality of narrative comments in workplace-based assessments; it has validity evidence for scoring the quality of narrative feedback provided to emergency medicine residents, but it is unknown whether the QuAL score is reliable in the assessment of narrative feedback in other postgraduate programs.

Methods

Fifty sets of EPA narratives from a single academic year at our competency based medical education post-graduate anesthesia program were selected by stratified sampling within defined parameters [e.g. resident gender and stage of training, assessor gender, Competency By Design training level, and word count (≥17 or <17 words)]. Two competency committee members and two medical students rated the quality of narrative feedback using a utility score and QuAL score. We used Kendall's tau-b co-efficient to compare the perceived utility of the written feedback to the quality assessed with the QuAL score. The authors used generalizability and decision studies to estimate the reliability and generalizability coefficients.

Results

Both the faculty's utility scores and QuAL scores (r = 0.646, p < 0.001) and the trainees' utility scores and QuAL scores (r = 0.667, p < 0.001) were moderately correlated. Results from the generalizability studies showed that utility scores were reliable with two raters for both faculty (Epsilon=0.87, Phi=0.86) and trainees (Epsilon=0.88, Phi=0.88).

Conclusions

The QuAL score is correlated with faculty- and trainee-rated utility of anesthesia EPA feedback. Both faculty and trainees can reliability apply the QuAL score to anesthesia EPA narrative feedback. This tool has the potential to be used for faculty development and program evaluation in Competency Based Medical Education. Other programs could consider replicating our study in their specialty.

Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine

PURPOSE

The developmental trajectory of learning during residency may be attributed to multiple factors, including variation in individual trainee performance, program-level factors, graduating medical school effects, and the learning environment. Understanding the relationship between medical school and learner performance during residency is important in prioritizing undergraduate curricular strategies and educational approaches for effective transition to residency and postgraduate training. This study explores factors contributing to longitudinal and developmental variability in resident Milestones ratings, focusing on variability due to graduating medical school, training program, and learners using national cohort data from emergency medicine (EM) and family medicine (FM).

METHOD

Data from programs with residents entering training in July 2016 were used (EM: n=1,645 residents, 178 residency programs; FM: n=3,997 residents, 487 residency programs). Descriptive statistics were used to examine data trends. Cross-classified mixed-effects regression were used to decompose variance components in Milestones ratings.

RESULTS

During postgraduate year (PGY)-1, graduating medical school accounted for 5% and 6% of the variability in Milestones ratings, decreasing to 2% and 5% by PGY-3 for EM and FM, respectively. Residency program accounted for substantial variability during PGY-1 (EM=70%, FM=53%) but decreased during PGY-3 (EM=62%, FM=44%), with greater variability across training period in patient care (PC), medical knowledge (MK), and systems-based practice (SBP). Learner variance increased significantly between PGY-1 (EM=23%, FM=34%) and PGY-3 (EM=34%, FM=44%), with greater variability in practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal communication skills (ICS).

CONCLUSIONS

The greatest variance in Milestone ratings can be attributed to the residency program and to a lesser degree, learners, and medical school. The dynamic impact of program-level factors on learners shifts during the first year and across the duration of residency training, highlighting the influence of curricular, instructional, and programmatic factors on resident performance throughout residency.

Gender Bias and Ophthalmology Accreditation Council for Graduate Medical Education Milestones Evaluations

Importance

Women remain underrepresented in ophthalmology and gender-based disparities exist in salary, grant receipt, publication rates, and surgical volume throughout training and in practice. Although studies in emergency medicine and general surgery showed mixed findings regarding gender differences in Accreditation Council for Graduate Medical Education (ACGME) Milestones ratings, limited data exist examining such differences within ophthalmology.

Objective

To examine gender differences in ophthalmology ACGME Milestones.

Design, Setting, and Participants

This was a retrospective cross-sectional study of postgraduate year 4 (PGY-4) residents from 120 ophthalmology programs graduating in 2019.

Main Outcomes and Measures

PGY-4 midyear and year-end medical knowledge (MK) and patient care (PC) ratings and Written Qualifying Examination (WQE) scaled scores for residents graduating in 2019 were included. Differential prediction techniques using Generalized Estimating Equations models were performed to identify differences by gender.

Results

Of 452 residents (median [IQR] age, 30.0 [29.0-32.0] years), 275 (61%) identified as men and 177 (39%) as women. There were no differences in PC domain average between women and men for both midyear (-0.07; 95% CI, -0.11 to 0; P =.06) and year-end (-0.04; 95% CI, -0.07 to 0.03; P =.51) assessment periods. For the MK domain average in the midyear assessment period, women (mean [SD], 3.76 [0.50]) were rated lower than men (mean [SD], 3.88 [0.47]; P = .006) with a difference in mean of -0.12 (95% CI, -0.18 to -0.03). For the year-end assessment, however, the average MK ratings were not different for women (mean [SD], 4.10 [0.47]) compared with men (mean [SD], 4.18 [0.47]; P = .20) with a difference in mean of -0.08 (95% CI, -0.13 to 0.03).

Conclusions and Relevance

Results suggest that ACGME ophthalmology Milestones in 2 general competencies did not demonstrate major gender bias on a national level at the time of graduation. There were, however, differences in MK ratings at the midyear mark, and as low ratings on evaluations and examinations may adversely affect career opportunities for trainees, it is important to continue further work examining other competencies or performance measures for potential biases.

Intersectional Disparities in Emergency Medicine Residents' Performance Assessments by Race, Ethnicity, and Sex

Importance

Previous studies have demonstrated sex-specific disparities in performance assessments among emergency medicine (EM) residents. However, less work has focused on intersectional disparities by ethnoracial identity and sex in resident performance assessments.

Objective

To estimate intersectional sex-specific ethnoracial disparities in standardized EM resident assessments.

Design, Setting, and Participants

This retrospective cohort study used data from the Association of American Medical Colleges and the Accreditation Council for Graduate Medical Education Milestones (Milestones) assessments to evaluate ratings for EM residents at 128 EM training programs in the US. Statistical analyses were conducted in June 2020 to January 2023.

Exposure

Training and assessment environments in EM residency programs across comparison groups defined by ethnoracial identity (Asian, White, or groups underrepresented in medicine [URM], ie, African American/Black, American Indian/Alaska Native, Hispanic/Latine, and Native Hawaiian/Other Pacific Islander) and sex (female/male).

Main Outcomes and Measures

Mean Milestone scores (scale, 0-9) across 6 core competency domains: interpersonal and communications skills, medical knowledge, patient care, practice-based learning and improvement, professionalism, and system-based practice. Overall assessment scores were calculated as the mean of the 6 competency scores.

Results

The study sample comprised 128 ACGME-accredited programs and 16 634 assessments for 2708 EM residents of which 1913 (70.6%) were in 3-year and 795 (29.4%) in 4-year programs. Most of the residents were White (n = 2012; 74.3%), followed by Asian (n = 477; 17.6%), Hispanic or Latine (n = 213; 7.9%), African American or Black (n = 160; 5.9%), American Indian or Alaska Native (n = 24; 0.9%), and Native Hawaiian or Other Pacific Islander (n = 4; 0.1%). Approximately 14.3% (n = 386) and 34.6% (n = 936) were of URM groups and female, respectively. Compared with White male residents, URM female residents in 3-year programs were rated increasingly lower in the medical knowledge (URM female score, -0.47; 95% CI, -0.77 to -0.17), patient care (-0.18; 95% CI, -0.35 to -0.01), and practice-based learning and improvement (-0.37; 95% CI, -0.65 to -0.09) domains by postgraduate year 3 year-end assessment; URM female residents in 4-year programs were also rated lower in all 6 competencies over the assessment period.

Conclusions and Relevance

This retrospective cohort study found that URM female residents were consistently rated lower than White male residents on Milestone assessments, findings that may reflect intersectional discrimination in physician competency evaluation. Eliminating sex-specific ethnoracial disparities in resident assessments may contribute to equitable health care by removing barriers to retention and promotion of underrepresented and minoritized trainees and facilitating diversity and representation among the emergency physician workforce.

"My Assessments Are Biased!" Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education

Assessing learners is foundational to their training and developmental growth throughout the medical education continuum. However, growing evidence shows the prevalence and impact of harmful bias in assessments in medical education, accelerating the urgency to identify solutions. Assessment bias presents a critical problem for all stages of learning and the broader educational system. Bias poses significant challenges to learners, disrupts the learning environment, and threatens the pathway and transition of learners into health professionals. While the topic of assessment bias has been examined within the context of measurement literature, limited guidance and solutions exist for learners in medical education, particularly in the clinical environment. This article presents an overview of assessment bias, focusing on clinical learners. A definition of bias and its manifestations in assessments are presented. Consequences of assessment bias are discussed within the contexts of validity and fairness and their impact on learners, patients/caregivers, and the broader field of medicine. Messick's unified validity framework is used to contextualize assessment bias; in addition, perspectives from sociocultural contexts are incorporated into the discussion to elaborate the nuanced implications in the clinical training environment. Discussions of these topics are conceptualized within the literature and the interventions used to date. The article concludes with practical recommendations to overcome bias and to develop an ideal assessment system. Recommendations address articulating values to guide assessment, designing assessment to foster learning and outcomes, attending to assessment procedures, promoting continuous quality improvement of assessment, and fostering equitable learning and assessment environments.

Identifying Gender-Related Differences in Graduate Medical Education with the Use of a Web-Based Professionalism Monitoring Tool

OBJECTIVES

Medical education is required to ensure a healthy training and learning environment for resident physicians. Trainees are expected to demonstrate professionalism with patients, faculty, and staff. West Virginia University Graduate Medical Education (GME) initiated a Web-based professionalism and mistreatment form ("button") on our Web site for reporting professionalism breaches, mistreatment, and exemplary behavior events. The purpose of this study was to identify characteristics in resident trainees who had a "button push" activation about their behavior to better understand ways to improve professionalism in GME.

METHODS

This West Virginia University institutional review board-approved quality improvement study is a descriptive analysis of GME button push activations from July 2013 through June 2021. We compared characteristics of all of those trainees who had specific button activation(s) about their behavior. Data are reported as frequency and percentage. Nominal data and interval data were analyzed using the χ2 and the t test, respectively. P < 0.05 was significant. Logistic regression was used to analyze those differences that were significant.

RESULTS

In the 8-year study period, there were 598 button activations, and 54% (n = 324) of the activations were anonymous. Nearly all of the button reports (n = 586, 98%) were constructively resolved within 14 days. Of the 598 button activations, 95% (n = 569) were identified as involving one sex, with 66.3% (n = 377) identified as men and 33.7% (n = 192) as women. Of the 598 activations, 83.7% (n = 500) involved residents and 16.3% (n = 98) involved attendings. One-time offenders comprised 90% (n = 538), and 10% (n = 60) involved individuals who had previous button pushes about their behavior.

CONCLUSIONS

Implementation of a professionalism-monitoring tool, such as our Web-based button push, identified gender differences in the reporting of professionalism breaches, because twice as many men as women were identified as the instigator of a professionalism breech. The tool also facilitated timely interventions and exemplary behavior recognition.

The outcomes of lockdown in the higher education sector during the COVID-19 pandemic

To control COVID-19 pandemic, complete lockdown was initiated in 2020. We investigated the impact of lockdown on tertiary-level academic performance, by comparing educational outcomes amongst first-year students during second semester of their medical course prior to and during lockdown. Evidence: The demographics, including educational outcomes of the two groups were not significantly different during semester one (prior to the lockdown). The academic performance amongst women was better than men prior to lockdown. However, the scores were improved significantly for both sexes during lockdown in 2020, following the complete online teaching, compared to that in 2019, showing no significant difference between men and women in 2020, for English and Chinese History. There were significant different scores between men and women in lab-based Histology Practice in 2019 (in-person tuition) and 2020 (online digital tuition), although only a significant improvement in women was observed between 2019 and 2020. Implication: the forced change to online delivery of the second semester of the first-year medical program in 2020 due to the COVID-19 pandemic did not result in any decline in assessment outcomes in any of the subjects undertaken. We believe extensive online digital media should continue to be available to students in future.

Gender Differences in Emergency Medicine Attending Physician Comments to Residents: A Qualitative Analysis

IMPORTANCE

Prior studies have revealed gender differences in the milestone and clinical competency committee assessment of emergency medicine (EM) residents.

OBJECTIVE

To explore gender disparities and the reasons for such disparities in the narrative comments from EM attending physicians to EM residents.

DESIGN, SETTING, AND PARTICIPANTS

This multicenter qualitative analysis examined 10 488 narrative comments among EM faculty and EM residents between 2015 to 2018 in 5 EM training programs in the US. Data were analyzed from 2019 to 2021.

MAIN OUTCOMES AND MEASURES

Differences in narrative comments by gender and study site. Qualitative analysis included deidentification and iterative coding of the data set using an axial coding approach, with double coding of 20% of the comments at random to assess intercoder reliability (κ, 0.84). The authors reviewed the unmasked coded data set to identify emerging themes. Summary statistics were calculated for the number of narrative comments and their coded themes by gender and study site. χ2 tests were used to determine differences in the proportion of narrative comments by gender of faculty and resident.

RESULTS

In this study of 283 EM residents, of whom 113 (40%) identified as women, and 277 EM attending physicians, of whom 95 (34%) identified as women, there were notable gender differences in the content of the narrative comments from faculty to residents. Men faculty, compared with women faculty, were more likely to provide either nonspecific comments (115 of 182 [63.2%] vs 40 of 95 [42.1%]), or no comments (3387 of 10 496 [32.3%] vs 1169 of 4548 [25.7%]; P < .001) to men and women residents. Compared with men residents, more women residents were told that they were performing below level by men and women faculty (36 of 113 [31.9%] vs 43 of 170 [25.3%]), with the most common theme including lack of confidence with procedural skills.

CONCLUSIONS AND RELEVANCE

In this qualitative study of narrative comments provided by EM attending physicians to residents, multiple modifiable contributors to gender disparities in assessment were identified, including the presence, content, and specificity of comments. Among women residents, procedural competency was associated with being conflated with procedural confidence. These findings can inform interventions to improve parity in assessment across graduate medical education.

Gender differences in emergency medicine resident assessment: A scoping review

Background

Growing literature within postgraduate medical education demonstrates that female resident physicians experience gender bias throughout their training and future careers. This scoping review aims to describe the current body of literature on gender differences in emergency medicine (EM) resident assessment.

Methods

We conducted a scoping review which adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. We included research involving resident physicians or fellows in EM (population and context), which focused on the impact of gender on assessments (concept). We searched seven databases from the databases' inception to April 4, 2022. Two reviewers independently screened citations, completed full-text review, and abstracted data. A third reviewer resolved any discrepancies.

Results

A total of 667 unique citations were identified; 10 studies were included, and all were conducted within the United States. Four studies reported differences in EM resident assessments attributable to gender within workplace-based assessments (qualitative comments and quantitative scores) by both attending physicians and nonphysicians. Six studies investigating clinical competency committee scores, procedural scores, and simulation-based assessments did not report any significant differences attributable to gender.

Conclusions

This scoping review found that gender bias exists within EM resident assessment most notably at the level of narrative comments typically received via workplace-based assessments. As female EM residents receive higher rates of negative or critical comments and discordant feedback documented on assessment, these findings raise concern about added barriers female EM residents may face while progressing through residency and the impact on their clinical and professional development.

An Empirical Investigation Into Milestones Factor Structure Using National Data Derived From Clinical Competency Committees

PURPOSE

To investigate whether milestone data obtained from clinical competency committee (CCC) ratings in a single specialty reflected the 6 general competency domains framework.

METHOD

The authors examined milestone ratings from all 275 U.S. Accreditation Council for Graduate Medical Education-accredited categorical obstetrics and gynecology (OBGYN) programs from July 1, 2018, to June 30, 2019. The sample size ranged from 1,371 to 1,438 residents from 275 programs across 4 postgraduate years (PGYs), each with 2 assessment periods. The OBGYN milestones reporting form consisted of 28 subcompetencies under the 6 general competency domains. Milestone ratings were determined by each program's CCC. Intraclass correlations (ICCs) and design effects were calculated for each subcompetency by PGY and assessment period. A multilevel confirmatory factor analysis (CFA) perspective was used, and the pooled within-program covariance matrix was obtained to compare the fit of the 6-domain factor model against 3 other plausible models.

RESULTS

Milestone ratings from 5,618 OBGYN residents were examined. Moderate to high ICCs and design effects greater than 2.0 were prevalent among all subcompetencies for both assessment periods, warranting the use of the multilevel approach in applying CFA to the milestone data. The theory-aided split-patient care (PC) factor model, which used the 6 general competency domains but also included 3 factors within the PC domain (obstetric technical skills, gynecology technical skills, and ambulatory care), was consistently shown as the best-fitting model across all PGYs by assessment period conditions, except for one.

CONCLUSIONS

The findings indicate that in addition to using the 6 general competency domains framework in their rating process, CCCs may have further distinguished the PC competency domain into 3 meaningful factors. This study provides internal structure validity evidence for the milestones within a single specialty and may shed light on CCCs' understanding of the distinctive content embedded within the milestones.

Gendered Expectations: the Impact of Gender, Evaluation Language, and Clinical Setting on Resident Trainee Assessment of Faculty Performance

BACKGROUND

Gender inequity is pervasive in academic medicine. Factors contributing to these gender disparities must be examined. A significant body of literature indicates men and women are assessed differently in teaching evaluations. However, limited data exist on how faculty gender affects resident evaluation of faculty performance based on the skill being assessed or the clinical practice settings in which the trainee-faculty interaction occurs.

OBJECTIVE

Evaluate for gender-based differences in the assessment of general internal medicine (GIM) faculty physicians by trainees in inpatient and outpatient settings.

DESIGN

Retrospective cohort study SUBJECTS: Inpatient and outpatient GIM faculty physicians in an Internal Medicine residency training program from July 1, 2015, to December 31, 2018.

MAIN MEASURES

Faculty scores on trainee teaching evaluations including overall teaching ability and Accreditation Council for Graduate Medical Education (ACGME) competencies (medical knowledge [MK], patient care [PC], professionalism [PROF], interpersonal and communication skills [ICS], practice-based learning and improvement [PBLI], and systems-based practice [SBP]) based on the institutional faculty assessment form.

KEY RESULTS

In total, 3581 evaluations by 445 trainees (55.1% men, 44.9% women) assessing 161 GIM faculty physicians (50.3% men, 49.7% women) were included. Male faculty were rated higher in overall teaching ability (male=4.69 vs. female=4.63, p=0.003) and in four of the six ACGME competencies (MK, PROF, PBLI, and SBP) based on our institutional evaluation form. In the inpatient setting, male faculty were rated more favorably for overall teaching (male = 4.70, female = 4.53, p=<0.001) and across all ACGME competencies. The only observed gender difference in the outpatient setting favored female faculty in PC (male = 4.65, female = 4.71, p=0.01).

CONCLUSIONS

Male and female GIM faculty performance was assessed differently by trainees. Gender-based differences were impacted by the setting of evaluation, with the greatest difference by gender noted in the inpatient setting.

Dressing the Part: Gender Differences in Residents' Experiences of Feedback in Internal Medicine

PURPOSE

Multiple studies demonstrate that assessment of residents differs by gender, yet little is known about how these differences are experienced by women and men. The authors sought to understand whether the experience of being assessed and receiving feedback differs between men and women internal medicine (IM) residents and how women respond to these experiences.

METHOD

A constructivist grounded theory approach to data collection and interpretation was used. The authors invited all IM residents in postgraduate years 1-3 at the University of Toronto to participate in semistructured focus groups (August-October 2019). Twenty-two residents participated (8 men, 14 women). Focus groups were divided by gender and training level.

RESULTS

The authors found a profound difference in experiences of receiving feedback between men and women residents. The themes of challenges to power and authority, tactics to reestablish power and authority, conflicting feedback from attendings, and ways of moving forward all diverged between men and women residents. Women repeatedly brought up feedback outside of official assessment moments and relied on symbols, such as a white coat, stethoscope, and demure clothing, to "dress the part" of a physician. Women also encountered conflicting feedback from supervisors regarding confidence and assertiveness (e.g., sometimes told to be more assertive, other times to be less), often resulting in self-censorship; similar feedback was rarely noted by men.

CONCLUSIONS

Gendered differences in the experiences of being assessed and receiving feedback are not always reflected in standard measures. Gender and medicine can be considered performative, and these findings demonstrate women IM residents integrate multiple forms of feedback to create the persona of the woman physician. The authors believe this research contributes a unique vantage point to the experience of women residents interpreting explicit and implicit feedback in IM and highlights the socialization that occurs to become a woman physician.

Assessment of Implicit Gender Bias During Evaluation of Procedural Competency Among Emergency Medicine Residents

IMPORTANCE

Gender disparities exist throughout medicine. Recent studies have highlighted an attainment gap between male and female residents in performance evaluations on Accreditation Council for Graduate Medical Education (ACGME) milestones. Because of difficulties in blinding evaluators to gender, it remains unclear whether these observed disparities are because of implicit bias or other causes.

OBJECTIVE

To estimate the magnitude of implicit gender bias in assessments of procedural competency in emergency medicine residents and whether the gender of the evaluator is associated with identified implicit gender bias.

DESIGN, SETTING, AND PARTICIPANTS

A cross-sectional study was performed from 2018 to 2020 in which emergency medicine residency faculty assessed procedural competency by evaluating videos of residents performing 3 procedures in a simulated environment. They were blinded to the intent of the study. Proceduralists were filmed performing each procedure from 2 different viewpoints simultaneously by 2 different cameras. One was a gender-blinded (ie, hands-only) view, and the other a wide-angled gender-evident (ie, whole-body) view. The faculty evaluators viewed videos in a random order and assessed procedural competency on a global rating scale with extensive validity evidence for the evaluation of video-recorded procedural performance.

MAIN OUTCOMES AND MEASURES

The primary outcome was to determine if there was a difference in the evaluation of procedural competency based on gender. The secondary outcome was to determine if there was a difference in the evaluations based on the gender of the evaluator.

RESULTS

Fifty-one faculty evaluators enrolled from 19 states, with 22 male participants (43.1%), 29 female participants (56.9%), and a mean (SD) age of 37 (6.4) years. Each evaluator assessed all 60 procedures: 30 gender-blinded (hands-only view) videos and 30 identical gender-evident (wide angle) videos. There were no statistically significant differences in the study evaluators' scores of the proceduralists based on their gender, and the gender of the evaluator was not associated with the difference in mean scores.

CONCLUSIONS AND RELEVANCE

In this study, we did not identify a difference in the evaluation of procedural competency based upon the gender of the resident proceduralist or the gender of the faculty evaluator.

Faculty Versus Resident Self-Assessment Using Pathology Milestones: How Aligned Are We?

Competent physicians must be able to self-assess skill level; however, previous studies suggest that medical trainees may not accurately self-assess. We utilized Pathology Milestones (PM) data to determine whether there were discrepancies in self- versus Clinical Competency Committee (CCC) ratings by sex, program year (PGY), time of evaluation, and question category (Patient Care, Medical Knowledge, Systems-Based Practice [SBP], Practice-Based Learning and Improvement [PBL], Professionalism [PRO], and Interpersonal and Communication Skills) and Residency In-Service Examination (RISE) score. We completed retrospective analyses of PM evaluation scores from 2016 to 2019 (n = 23 residents) 2 times per year. Discrepancies in evaluation scores were calculated by subtracting CCC scores from resident self-evaluation scores. There was no significant difference in discrepancy scores between male versus female residents (P = .94). Discrepancy scores among all PGYs were significantly different (P < .0001), with PGY1 tending to overrate the most, followed by PGY2. PGY3 and PGY4 underrated themselves on average compared to CCC ratings, with PGY4 having significantly lower self-ratings than CCC compared to any other PGY. In January, residents underscored themselves and in July residents overscored themselves compared to CCC (P < .0001 for both). Question types resulted in variable discrepancy scores, with SBP significantly lower than and PRO significantly higher than all other categories (P < .05 for both). Increases in RISE score correlated to increases in self- and CCC-scoring. These discrepancies can help trainees improve self-assessment. Discrepancies indicate potential areas for amelioration, such as curriculum adjustments or Milestone’s verbiage.

Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents

IMPORTANCE

Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice.

OBJECTIVE

To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents.

DESIGN, SETTING, AND PARTICIPANTS

This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019.

INTERVENTIONS

Longitudinal formative assessment using the Milestones assessment system reported to the ACGME.

MAIN OUTCOMES AND MEASURES

To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories.

RESULTS

This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth.

CONCLUSIONS AND RELEVANCE

These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

Effect of resident and assessor gender on entrustment-based observational assessment in an internal medicine residency program

Background: Implicit gender bias leads to differences in assessment. Studies examining gender differences in resident milestone assessment data demonstrate variable results. The purpose of this study was to determine if observational entrustment scores differ by resident and assessor gender in a program of assessment based on discrete, observable skills.  Methods: We analyzed overall entrustment scores and entrustment scores by Accreditation Council for Graduate Medical Education (ACGME) core competency for 238 residents (49% female) from 396 assessors (38% female) in one internal medicine residency program from July 2012 to June 2019. We conducted analyses at 1-12 months, 1-36 months, 1-6 months, 7-12 months, and 31-36 months. We used linear mixed-effect models to assess the role of resident and assessor gender, with resident-specific and assessor-specific random effect to account for repeated measures.  Results: Statistically significant interactions existed between resident and assessor gender for overall entrustment at 1-12 months (p < 0.001), 1-36 months (p< 0.001), 1-6 months (p<0.001), 7-12 months (p=0.04), and 31-36 months (p<0.001). However, group differences were not statistically significant. In several instances an interaction was significant between resident and assessor gender by ACGME core competency, but there were no statistically significant group differences for all competencies at any time point. When applicable, subsequent analysis of main effect of resident or assessor gender independently of one another revealed no statistically significant differences.   Conclusions: No significant differences in entrustment scores were found based on resident or assessor gender in our large, robust entrustment-based program of assessment. Determining the reasons for our findings may help identify ways to mitigate gender bias in assessment.

Parenthood and Parental Leave Decisions in Pediatric Residency

OBJECTIVES

The demands of residency training may impact trainees' decision to have children. We examined characteristics of pediatric residents' decisions regarding childbearing, determinants of resident parental leave, and associations with well-being.

METHODS

A survey of 845 pediatric residents at 13 programs was conducted between October 2019 and May 2020. Survey items included demographics, desire for future children, and logistics of parental leave. Outcomes included parental leave length, burnout and depression screening results, satisfaction with duration of breastfeeding, and satisfaction with parental leave and parenthood decisions.

RESULTS

Seventy-six percent (639 of 845) of residents responded to the survey. Fifty-two percent (330) of respondents reported delaying having children during residency, and 29% (97) of those were dissatisfied with their decision to do so. Busy work schedule (89.7%), finances (50.9%), and a desire not to extend residency (41.2%) were the most common reasons for delay. Of respondents, 16% were parents and 4% were pregnant or had pregnant partners. Sixty-one parental leaves were reported, and 67% of parents reported dissatisfaction with leave length. The most frequently self-reported determinant of leave duration was the desire not to extend residency training (74%). Program mean leave length was negatively associated with burnout, measured as a dichotomous outcome (odds ratio = 0.81 [95% confidence interval 0.68-0.98]; P = .02).

CONCLUSIONS

Many pediatric trainees delay parenthood during residency and are not satisfied with their decision to do so. Pediatric resident parental leave remains short and variable in duration, despite the positive association between longer leaves and overall well-being.

Trainee Sex and Accreditation Council for Graduate Medical Education Milestone Assessments During General Surgery Residency

Importance

In evaluating the effectiveness of general surgery (GS) training, an unbiased assessment of the progression of residents with attention to individual learner factors is imperative.

Objective

To evaluate the role of trainee sex in milestone achievement over the course of GS residency using national data from the Accreditation Council for Graduate Medical Education (ACGME).

Design, Setting, and Participants

This cross-sectional study evaluated female and male GS residents enrolled in ACGME-accredited programs in the US from 2014 to 2018 with reported variation in milestones performance across years in training and representation. Data were analyzed from November 2019 to June 2021.

Main Outcomes and Measures

Mean reported milestone score at initial and final assessment, and predicted time-to-attainment of equivalent performance by sex.

Results

Among 4476 GS residents from 250 programs who had milestone assessments at any point in their clinical training, 1735 were female (38.8%). Initially, female and male residents received similar mean (SD) milestone scores (1.95 [0.50] vs 1.94 [0.50]; P = .69). At the final assessment, female trainees received overall lower mean milestone scores than male trainees (4.25 vs 4.31; P < .001). Significantly lower mean milestone scores were reported for female residents at the final assessment for several subcompetencies in both univariate and multivariate analyses, with only medical knowledge 1 (pathophysiology, diagnosis, and initial management) common to both. Multilevel mixed-effects linear modeling demonstrated that female trainees had significantly lower rates of monthly milestone attainment in the subcompetency of medical knowledge 1, which was associated with a significant difference in training time of approximately 1.8 months.

Conclusions and Relevance

Both female and male GS trainees achieved the competency scores necessary to transition to independence after residency as measured by the milestones assessment system. Initially, there were no sex differences in milestone score. By graduation, there were differences in the measured assessment of female and male trainees across several subcompetencies. Careful monitoring for sex bias in the evaluation of trainees and scrutiny of the training process is needed to ensure that surgical residency programs support the educational needs of both female and male trainees.

Milestones comparisons from residency to pediatric emergency medicine fellowship: Resetting expectations

BACKGROUND

Pediatric emergency medicine (PEM) fellowships accept trainees who have completed a residency in either emergency medicine (EM) or pediatrics and have adopted 17 subcompetencies with accompanying set of milestones from these two residency programs. This study aims to examine the changes in milestone scores among common subcompetencies from the end of EM or pediatrics residency to early PEM fellowship and evaluates time to reattainment of scores for subcompetencies in which a decline was noted.

METHODS

This is a national, retrospective cohort study of trainees enrolled in PEM fellowship programs from July 2014 to June 2018. PEM fellowship program directors voluntarily submitted deidentified milestone reports within the study time frame, including end-of-residency reports. Descriptive analyses of milestone scores between end of residency and PEM fellowship were performed.

RESULTS

Forty-eight U.S. PEM fellowship programs (65%) provided fellowship milestone data on 638 fellows, 218 (34%) of whom also had end-of-residency milestone scores submitted. Of 218 fellows eligible for analysis, 210 (96%) had completed a pediatrics residency and eight (4%) had completed an EM residency. Pediatric-trained fellows had statistically significant decreases in mean milestone scores in all 10 shared subcompetencies. Reattainment of milestone scores across all common subcompetencies for both EM and pediatric-trained PEM fellows occurred by the end of fellowship.

CONCLUSIONS

This study demonstrated declines in milestone scores from the end of primary residency training in pediatrics to early PEM fellowship in shared subcompetencies, which may suggest that performance expectations are reset at the beginning of PEM fellowship. Changes in subcompetency milestone anchors to provide subspecialty-specific context may be needed to more accurately define skills acquisition in the residency-to-fellowship transition.

Gender Differences Among Milestone Assessments in a National Sample of Pediatric Emergency Medicine Fellowship Programs

BACKGROUND

Understanding gender gaps in trainee evaluations is critical because these may ultimately determine the duration of training. Currently, no studies describe the influence of gender on the evaluation of pediatric emergency medicine (PEM) fellows.

OBJECTIVE

The objective of our study was to compare milestone scores of female versus male PEM fellows.

METHODS

This is a multicenter retrospective cohort study of a national sample of PEM fellows from July 2014 to June 2018. Accreditation Council for Medical Education (ACGME) subcompetencies are scored on a 5-point scale and span six domains: patient care (PC), medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and interpersonal and communication skills (ICS). Summative assessments of the 23 PEM subcompetencies are assigned by each program's clinical competency committee and submitted semiannually for each fellow. Program directors voluntarily provided deidentified ACGME milestone reports. Demographics including sex, program region, and type of residency were collected. Descriptive analysis of milestones was performed for each year of fellowship. Multivariate analyses evaluated the difference in scores by sex for each of the subcompetencies.

RESULTS

Forty-eight geographically diverse programs participated, yielding data for 639 fellows (66% of all PEM fellows nationally); sex was recorded for 604 fellows, of whom 67% were female. When comparing the mean milestone scores in each of the six domains, there were no differences by sex in any year of training. When comparing scores within each of the 23 subcompetencies and correcting the significance level for comparison of multiple milestones, the scores for PC3 and ICS2 were significantly, albeit not meaningfully, higher for females.

CONCLUSION

In a national sample of PEM fellows, we found no major differences in milestone scores between females and males.

Workplace-based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

OBJECTIVE

In the era of competency-based medical education (CBME), the collection of more and more trainee data is being mandated by accrediting bodies such as the Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada. However, few efforts have been made to synthesize the literature around the current issues surrounding workplace-based assessment (WBA) data. This scoping review seeks to synthesize the landscape of literature on the topic of data collection and utilization for trainees' WBAs in emergency medicine (EM).

METHODS

The authors conducted a scoping review in the style of Arksey and O'Malley, seeking to synthesize and map literature on collecting, aggregating, and reporting WBA data. The authors extracted, mapped, and synthesized literature that describes, supports, and substantiates effective data collection and utilization in the context of the CBME movement within EM.

RESULTS

Our literature search retrieved 189 potentially relevant references (after removing duplicates) that were screened to 29 abstracts and papers relevant to collecting, aggregating, and reporting WBAs. Our analysis shows that there is an increasing temporal trend toward contributions in these topics, with the majority of the papers (16/29) being published in the past 3 years alone.

CONCLUSION

There is increasing interest in the areas around data collection and utilization in the age of CBME. The field, however, is only beginning to emerge, leaving more work that can and should be done in this area.

Gender Differences in Milestone Ratings and Medical Knowledge Examination Scores Among Internal Medicine Residents

PURPOSE

To examine whether there are group differences in milestone ratings submitted by program directors working with clinical competency committees (CCCs) based on gender for internal medicine (IM) residents and whether women and men rated similarly on milestones perform comparably on subsequent in-training and certification examinations.

METHOD

This national retrospective study examined end-of-year medical knowledge (MK) and patient care (PC) milestone ratings and IM In-Training Examination (IM-ITE) and IM Certification Examination (IM-CE) scores for 2 cohorts (2014-2017, 2015-2018) of U.S. IM residents at ACGME-accredited programs. It included 20,098/21,440 (94%) residents, with 9,424 women (47%) and 10,674 men (53%). Descriptive statistics and differential prediction techniques using hierarchical linear models were performed.

RESULTS

For MK milestone ratings in PGY-1, men and women showed no statistical difference at a significance level of .01 (P = .02). In PGY-2 and PGY-3, men received statistically higher average MK ratings than women (P = .002 and P < .001, respectively). In contrast, men and women received equivalent average PC ratings in each PGY (P = .47, P = .72, and P = .80, for PGY-1, PGY-2, and PGY-3, respectively). Men slightly outperformed women with similar MK or PC ratings in PGY-1 and PGY-2 on the IM-ITE by about 1.7 and 1.5 percentage points, respectively, after adjusting for covariates. For PGY-3 ratings, women and men with similar milestone ratings performed equivalently on the IM-CE.

CONCLUSIONS

Milestone ratings were largely similar for women and men. Generally, women and men with similar MK or PC milestone ratings performed similarly on future examinations. Although there were small differences favoring men on earlier examinations, these differences disappeared by the final training year. It is questionable whether these small differences are educationally or clinically meaningful. The findings suggest fair, unbiased milestone ratings generated by program directors and CCCs assessing residents.

Association Between Internal Medicine Residency Applicant Characteristics and Performance on ACGME Milestones During Intern Year

BACKGROUND

Residency programs apply varying criteria to the resident selection process. However, it is unclear which applicant characteristics reflect preparedness for residency.

OBJECTIVE

We determined the applicant characteristics associated with first-year performance in internal medicine residency as assessed by performance on Accreditation Council for Graduate Medical Education (ACGME) Milestones.

METHODS

We examined the association between applicant characteristics and performance on ACGME Milestones during intern year for individuals entering Northwestern University's internal medicine residency between 2013 and 2018. We used bivariate analysis and a multivariable linear regression model to determine the association between individual factors and Milestone performance.

RESULTS

Of 203 eligible residents, 198 (98%) were included in the final sample. One hundred fourteen residents (58%) were female, and 116 residents (59%) were White. Mean Step 1 and Step 2 CK scores were 245.5 (SD 12.0) and 258 (SD 10.8) respectively. Step 1 scores, Alpha Omega Alpha membership, medicine clerkship grades, and interview scores were not associated with Milestone performance in the bivariate analysis and were not included in the multivariable model. In the multivariable model, overall clerkship grades, ranking of the medical school, and year entering residency were significantly associated with Milestone performance (P ≤ .04).

CONCLUSIONS

Most traditional metrics used in residency selection were not associated with early performance on ACGME Milestones during internal medicine residency.

Residents' Perceptions of Effective Features of Educational Podcasts

INTRODUCTION

Educational podcasts are used by emergency medicine (EM) trainees to supplement clinical learning and to foster a sense of connection to broader physician communities. Yet residents report difficulties remembering what they learned from listening, and the features of podcasts that residents find most effective for learning remain poorly understood. Therefore, we sought to explore residents' perceptions of the design features of educational podcasts that they felt most effectively promoted learning.

METHODS

We used a qualitative approach to explore EM trainees' experiences with educational podcasts, focusing on design features that they found beneficial to their learning. We conducted 16 semi-structured interviews with residents from three institutions from March 2016-August 2017. Interview transcripts were analyzed line-by-line using constant comparison and organized into focused codes, conceptual categories, and then key themes.

RESULTS

The five canons of classical rhetoric provided a framework for thematically grouping the disparate features of podcasts that residents reported enhanced their learning. Specifically, they reported valuing the following: 1) Invention: clinically relevant material presented from multiple perspectives with explicit learning points; 2) Arrangement: efficient communication; 3) Style: narrative incorporating humor and storytelling; 4) Memory: repetition of key content; and 5) Delivery: short episodes with good production quality.

CONCLUSION

This exploratory study describes features that residents perceived as effective for learning from educational podcasts and provides foundational guidance for ongoing research into the most effective ways to structure medical education podcasts.

A Critical Disconnect: Residency Selection Factors Lack Correlation With Intern Performance

BACKGROUND

Emergency medicine (EM) residency programs want to employ a selection process that will rank best possible applicants for admission into the specialty.

OBJECTIVE

We tested if application data are associated with resident performance using EM milestone assessments. We hypothesized that a weak correlation would exist between some selection factors and milestone outcomes.

METHODS

Utilizing data from 5 collaborating residency programs, a secondary analysis was performed on residents trained from 2013 to 2018. Factors in the model were gender, underrepresented in medicine status, United States Medical Licensing Examination Step 1 and 2 Clinical Knowledge (CK), Alpha Omega Alpha (AOA), grades (EM, medicine, surgery, pediatrics), advanced degree, Standardized Letter of Evaluation global assessment, rank list position, and controls for year assessed and program. The primary outcomes were milestone level achieved in the core competencies. Multivariate linear regression models were fitted for each of the 23 competencies with comparisons made between each model's results.

RESULTS

For the most part, academic performance in medical school (Step 1, 2 CK, grades, AOA) was not associated with residency clinical performance on milestones. Isolated correlations were found between specific milestones (eg, higher surgical grade increased wound care score), but most had no correlation with residency performance.

CONCLUSIONS

Our study did not find consistent, meaningful correlations between the most common selection factors and milestones at any point in training. This may indicate our current selection process cannot consistently identify the medical students who are most likely to be high performers as residents.

Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study

BACKGROUND

In 2018, Canadian postgraduate emergency medicine (EM) programs began implementing a competency-based medical education (CBME) assessment program. Studies evaluating these programs have focused on broad outcomes using data from national bodies and lack data to support program-specific improvement.

OBJECTIVE

We evaluated the implementation of a CBME assessment program within and across programs to identify successes and opportunities for improvement at the local and national levels.

METHODS

Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of entrustable professional activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages were compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations.

RESULTS

Data from 15 of 17 (88%) programs containing 9842 EPA observations from 68 of 77 (88%) EM residents in the 2018 cohort were analyzed. Average numbers of EPAs observed per resident in each program varied from 92.5 to 229.6, correlating with the number of blocks spent on EM and pediatric EM (r = 0.83, P < .001). Relative to the specialty committee's guidelines, residents were promoted later than expected (eg, one-third of residents had a 2-month delay to promotion from the first to second stage) and with fewer EPA observations than suggested.

CONCLUSIONS

There was demonstrable variation in EPA-based assessment numbers and promotion timelines between programs and with national guidelines.

Association of Gender With Learner Assessment in Graduate Medical Education

IMPORTANCE

Gender bias may affect assessment in competency-based medical education.

OBJECTIVE

To evaluate the association of gender with assessment of internal medicine residents.

DESIGN, SETTING, AND PARTICIPANTS

This multisite, retrospective, cross-sectional study included 6 internal medicine residency programs in the United States. Data were collected from July 1, 2016, to June 30, 2017, and analyzed from June 7 to November 6, 2019.

EXPOSURES

Faculty assessments of resident performance during general medicine inpatient rotations.

MAIN OUTCOMES AND MEASURES

Standardized scores were calculated based on rating distributions for the Accreditation Council for Graduate Medical Education's core competencies and internal medicine Milestones at each site. Standardized scores are expressed as SDs from the mean. The interaction of gender and postgraduate year (PGY) with standardized scores was assessed, adjusting for site, time of year, resident In-Training Examination percentile rank, and faculty rank and specialty.

RESULTS

Data included 3600 evaluations for 703 residents (387 male [55.0%]) by 605 faculty (318 male [52.6%]). Interaction between resident gender and PGY was significant in 6 core competencies. In PGY2, female residents scored significantly higher than male residents in 4 of 6 competencies, including patient care (mean standardized score [SE], 0.10 [0.04] vs 0.22 [0.05]; P = .04), systems-based practice (mean standardized score [SE], -0.06 [0.05] vs 0.13 [0.05]; P = .003), professionalism (mean standardized score [SE], -0.04 [0.06] vs 0.21 [0.06]; P = .001), and interpersonal and communication skills (mean standardized score [SE], 0.06 [0.05] vs 0.32 [0.06]; P < .001). In PGY3, male residents scored significantly higher than female patients in 5 of 6 competencies, including patient care (mean standardized score [SE], 0.47 [0.05] vs 0.32 [0.05]; P = .03), medical knowledge (mean standardized score [SE], 0.47 [0.05] vs 0.24 [0.06]; P = .003), systems-based practice (mean standardized score [SE], 0.30 [0.05] vs 0.12 [0.06]; P = .02), practice-based learning (mean standardized score [SE], 0.39 [0.05] vs 0.16 [0.06]; P = .004), and professionalism (mean standardized score [SE], 0.35 [0.05] vs 0.18 [0.06]; P = .03). There was a significant increase in male residents' competency scores between PGY2 and PGY3 (range of difference in mean adjusted standardized scores between PGY2 and PGY3, 0.208-0.391; P ≤ .002) that was not seen in female residents' scores (range of difference in mean adjusted standardized scores between PGY2 and PGY3, -0.117 to 0.101; P ≥ .14). There was a significant increase in male residents' scores between PGY2 and PGY3 cohorts in 6 competencies with female faculty and in 4 competencies with male faculty. There was no significant change in female residents' competency scores between PGY2 to PGY3 cohorts with male or female faculty. Interaction between faculty-resident gender dyad and PGY was significant in the patient care competency (β estimate [SE] for female vs male dyad in PGY1 vs PGY3, 0.184 [0.158]; β estimate [SE] for female vs male dyad in PGY2 vs PGY3, 0.457 [0.181]; P = .04).

CONCLUSIONS AND RELEVANCE

In this study, resident gender was associated with differences in faculty assessments of resident performance, and differences were linked to PGY. In contrast to male residents' scores, female residents' scores displayed a peak-and-plateau pattern whereby assessment scores peaked in PGY2. Notably, the peak-and-plateau pattern was seen in assessments by male and female faculty. Further study of factors that influence gender-based differences in assessment is needed.

Orthopaedic Surgery Residency Milestones: Initial Formulation and Future Directions

Milestones specific to orthopaedic surgical training document individual resident progress through skill development in multiple dimensions. Residents increasingly interact with and are assessed by surgeons in both academic and private practice environments. Milestones describe the skills that support competence. One of the primary goals of milestones is to provide continuous data for educational quality improvement of residency programs. They provide a dialogue between surgeons who supervise residents or fellows and the program's Clinical Competency Committee throughout a resident's education. The orthopaedic milestones were developed jointly by the Accreditation Council for Graduate Medical Education and the American Board of Orthopaedic Surgery. The working team was designed with broad representation within the specialty. The milestones were introduced to orthopaedic residencies in 2013. Orthopaedics is a 5-year training program; the first comprehensive longitudinal data set is now available for study. This summary provides historical perspective on the development of the milestones, state of current milestone implementation, attempts to establish validity, challenges with the milestones, and the development of next-generation assessment tools.