Trainee Sex and Accreditation Council for Graduate Medical Education Milestone Assessments During General Surgery Residency

Belonging in Surgery: A Validated Instrument and Single Institutional Pilot

OBJECTIVE

The aim of this study was to develop and validate an instrument to measure Belonging in Surgery among surgical residents.

BACKGROUND

Belonging is the essential human need to maintain meaningful relationships and connections to one's community. Increased belongingness is associated with better well-being, job performance, and motivation to learn. However, no tools exist to measure belonging among surgical trainees.

METHODS

A panel of experts adapted a belonging instrument for use among United States surgery residents. After administration of the 28-item instrument to residents at a single institution, a Cronbach alpha was calculated to measure internal consistency, and exploratory principal component analyses were performed. Multiple iterations of analyses with successively smaller item samples suggested the instrument could be shortened. The expert panel was reconvened to shorten the instrument. Descriptive statistics measured demographic factors associated with Belonging in Surgery.

RESULTS

The overall response rate was 52% (114 responses). The Cronbach alpha among the 28 items was 0.94 (95% CI: 0.93-0.96). The exploratory principal component analyses and subsequent Promax rotation yielded 1 dominant component with an eigenvalue of 12.84 (70% of the variance). The expert panel narrowed the final instrument to 11 items with an overall Cronbach alpha of 0.90 (95% CI: 0.86, 0.92). Belonging in Surgery was significantly associated with race (Black and Asian residents scoring lower than White residents), graduating with one's original intern cohort (residents who graduated with their original class scoring higher than those that did not), and inversely correlated with resident stress level.

CONCLUSIONS

An instrument to measure Belonging in Surgery was validated among surgical residents. With this instrument, Belonging in Surgery becomes a construct that may be used to investigate surgeon performance and well-being.

Experiences With Unionization Among General Surgery Resident Physicians, Faculty, and Staff

Importance

Labor unions are a mechanism for employee advocacy, but their role in surgery resident wellness is poorly characterized.

Objective

To understand experiences with unionization among general surgery residents and residency program faculty and staff.

Design, Setting, and Participants

This exploratory qualitative study included data from the Surgical Education Culture Optimization Through Targeted Interventions Based on National Comparative Data (SECOND) trial. In the exploratory phase of the SECOND trial (from March 6, 2019, to March 12, 2020), semistructured interviews about wellness were conducted with residents, faculty (attending physicians), and staff (program administrators) at 15 general surgery residency programs. Unionization was identified as an emergent theme in the interviews. Data analysis was performed from March 2019 to May 2023.

Main Outcomes and Measures

The main outcome was resident and faculty experience with resident labor unions. In the qualitative analysis, lexical searches of interview transcripts identified content regarding resident labor unions. A codebook was developed inductively. Transcripts were coded by dyads, using a constant comparative approach, with differences reconciled by consensus.

Results

A total of 22 interview transcripts were identified with relevant content. Of these, 19 were individual interviews conducted with residents (n = 10), faculty (n = 4), administrative staff (n = 1), a program director (n = 1), a department chair (n = 1), and designated institutional officials (n = 2), and 3 were from resident focus groups. Residents from all postgraduate year levels, including professional development (ie, research) years, were represented. Interviewees discussed resident unions at 2 programs (1 recently unionized and 1 with a decades-long history). Interviewees described the lack of voice and the lack of agency as drivers of unionization ("Residents…are trying to take control of their well-being"). Increased salary stipends and/or housing stipends were the most concretely identified union benefits. Unanticipated consequences of unionization were described by both residents and faculty, including (1) irrelevance of union-negotiated benefits to surgical residents, (2) paradoxical losses of surgery department-provided benefits, and (3) framing of resident-faculty relationships as adversarial. Union executives were noted to be nonphysician administrators whose participation in discussions about clinical education progression may increase the time and effort to remediate a resident and/or reduce educators' will to meaningfully intervene. Active surgical resident participation within the union allows for an understanding of surgical trainees' unique needs and reduced conflict.

Conclusions and Relevance

In this qualitative study, unionization was a mechanism for resident voice and agency; the desire to unionize likely highlighted the lack of other such mechanisms in the training environment. However, these findings suggest that unionization may have had unintended consequences on benefits, flexibility, and teaching. Effective advocacy, whether within or outside the context of a union, was facilitated by participation from surgical residents. Future research should expand on this exploratory study by including a greater number of institutions and investigating the evolution of themes over time.

Differential attainment in assessment of postgraduate surgical trainees: a scoping review

INTRODUCTION

Solving disparities in assessments is crucial to a successful surgical training programme. The first step in levelling these inequalities is recognising in what contexts they occur, and what protected characteristics are potentially implicated.

METHODS

This scoping review was based on Arksey & O'Malley's guiding principles. OVID and Embase were used to identify articles, which were then screened by three reviewers.

RESULTS

From an initial 358 articles, 53 reported on the presence of differential attainment in postgraduate surgical assessments. The majority were quantitative studies (77.4%), using retrospective designs. 11.3% were qualitative. Differential attainment affects a varied range of protected characteristics. The characteristics most likely to be investigated were gender (85%), ethnicity (37%) and socioeconomic background (7.5%). Evidence of inequalities are present in many types of assessment, including: academic achievements, assessments of progression in training, workplace-based assessments, logs of surgical experience and tests of technical skills.

CONCLUSION

Attainment gaps have been demonstrated in many types of assessment, including supposedly "objective" written assessments and at revalidation. Further research is necessary to delineate the most effective methods to eliminate bias in higher surgical training. Surgical curriculum providers should be informed by the available literature on inequalities in surgical training, as well as other neighbouring specialties such as medicine or general practice, when designing assessments and considering how to mitigate for potential causes of differential attainment.

National Profile of the ACGME Milestones 1.0 and 2.0 within General Surgery: A Seven-Year National Study from 2014 to 2021

PURPOSE

The Accreditation Council for Graduate Medical Education (ACGME) introduced General Surgery Milestones 1.0 in 2014 and Milestones 2.0 in 2020 as steps toward competency-based training. Analysis will inform residency programs on curriculum development, assessment, feedback, and faculty development. This study describes the distributions and trends for Milestones 1.0 and 2.0 ratings and proportion of residents not achieving the level 4.0 graduation target.

METHODS

A deidentified dataset of milestone ratings for all ACGME-accredited General Surgery residency programs in the United States was used. Medians and interquartile ranges (IQR) were reported for milestone ratings at each PGY level. Percentages of PGY-5s receiving final year ratings of less than 4.0 were calculated. Wilcoxon rank sum tests were used to compare 1.0 and 2.0 median ratings. Kruskal-Wallis and Bonferroni post-hoc tests were used to compare median ratings across time periods and PGY levels. Chi-squared tests were used to compare the proportion of level 4.0 nonachievement under both systems.

RESULTS

Milestones 1.0 data consisted of 13,866 residents and Milestones 2.0 data consisted of 7,633 residents. For 1.0 and 2.0, all competency domain median ratings were higher for subsequent years of training. Milestones 2.0 had significantly higher median ratings at all PGY levels for all competency domains except Medical Knowledge. Percentages of PGY-5 residents not achieving the graduation target in Milestones 1.0 ranged from 27% to 42% and in 2.0 from 5% to 13%. For Milestones 1.0, all subcompetencies showed an increased number of residents achieving the graduation target from 2014 to 2019.

CONCLUSIONS

This study of General Surgery Milestones 1.0 and 2.0 data uncovered significant increases in average ratings and significantly fewer residents not achieving the graduation target under the 2.0 system. We hypothesize that these findings may be related more to rating bias given the change in rating scales, rather than a true increase in resident ability.

Sex influences how attendings evaluate surgical residents in the operating room

BACKGROUND

The aim of this study was to evaluate the influence of sex on facultys' perception of resident autonomy and performance.

METHODS

Autonomy/performance/complexity evaluations performed by faculty of categorical general surgery residents (2015-2021) were analyzed. Comparisons of scores by faculty and resident sex were performed.

RESULTS

A total of 10967 paper/electronic evaluations were collected. Female attendings rated female residents significantly lower in autonomy when compared to males (2.75 vs 2.91, p ​= ​0.0037). There was no significant difference in autonomy ratings for male versus female residents when evaluated by a male attending (2.93 vs 2.96, p ​= ​0.054) but male attendings did rate female residents significantly lower in autonomy at the highest complexities (2.37 vs 2.50, p ​= ​0.012).

CONCLUSION

The data suggests a unique interaction between attending and resident sex. A periodic evaluation of evaluations within one's program may provide invaluable implicit bias insight and should be considered.

The domino effect: the impact of gender on operative self-efficacy

BACKGROUND

Studies suggest that there are key differences in operative experience based on a trainee's gender. A large-scale self-efficacy (SE) survey, distributed to general surgery residents after the American Board of Surgery In-Training Examination in 2020, found that female gender was associated with decreased SE in graduating PGY5 residents for all 4 laparoscopic procedures included on the survey (cholecystectomy, appendectomy, right hemicolectomy, and diagnostic laparoscopy). We sought to determine whether these differences were reflected at the case level when considering operative performance and supervision using an operative assessment tool (SIMPL OR).

METHODS

Supervision and performance data reported through the SIMPL OR platform for the same 4 laparoscopic procedures included in the SE survey were aggregated for residents who were PGY5s in 2020. Independent t-tests and multiple linear regression were used to determine the relationship between trainee gender and supervision/performance ratings.

RESULTS

For laparoscopic cases in aggregate (n = 2708), male residents rated their performance higher than females (3.57 vs. 3.26, p < 0.001, 1 = critical deficiency, 5 = exceptional performance) and reported less supervision (3.15 vs. 2.85, p < 0.001, 1 = show and tell, 4 = supervision only); similar findings were seen when looking at attending reports of resident supervision and performance. A multiple linear regression model showed that attending gender did not significantly predict resident-reported supervision or performance levels, while case complexity and trainee gender significantly affected both supervision and performance (p < 0.001).

DISCUSSION

Female residents perceive themselves to be less self-efficacious at core laparoscopic procedures compared to their male colleagues. Comparison to more case-specific data confirm that female residents receive more supervision and lower performance ratings. This may create a domino effect in which female residents receive less operative independence, preventing the opportunity to establish SE. Further research should identify opportunities to break this cycle and consider gender identity beyond the male/female construct.

Gender Bias and Ophthalmology Accreditation Council for Graduate Medical Education Milestones Evaluations

Importance

Women remain underrepresented in ophthalmology and gender-based disparities exist in salary, grant receipt, publication rates, and surgical volume throughout training and in practice. Although studies in emergency medicine and general surgery showed mixed findings regarding gender differences in Accreditation Council for Graduate Medical Education (ACGME) Milestones ratings, limited data exist examining such differences within ophthalmology.

Objective

To examine gender differences in ophthalmology ACGME Milestones.

Design, Setting, and Participants

This was a retrospective cross-sectional study of postgraduate year 4 (PGY-4) residents from 120 ophthalmology programs graduating in 2019.

Main Outcomes and Measures

PGY-4 midyear and year-end medical knowledge (MK) and patient care (PC) ratings and Written Qualifying Examination (WQE) scaled scores for residents graduating in 2019 were included. Differential prediction techniques using Generalized Estimating Equations models were performed to identify differences by gender.

Results

Of 452 residents (median [IQR] age, 30.0 [29.0-32.0] years), 275 (61%) identified as men and 177 (39%) as women. There were no differences in PC domain average between women and men for both midyear (-0.07; 95% CI, -0.11 to 0; P =.06) and year-end (-0.04; 95% CI, -0.07 to 0.03; P =.51) assessment periods. For the MK domain average in the midyear assessment period, women (mean [SD], 3.76 [0.50]) were rated lower than men (mean [SD], 3.88 [0.47]; P = .006) with a difference in mean of -0.12 (95% CI, -0.18 to -0.03). For the year-end assessment, however, the average MK ratings were not different for women (mean [SD], 4.10 [0.47]) compared with men (mean [SD], 4.18 [0.47]; P = .20) with a difference in mean of -0.08 (95% CI, -0.13 to 0.03).

Conclusions and Relevance

Results suggest that ACGME ophthalmology Milestones in 2 general competencies did not demonstrate major gender bias on a national level at the time of graduation. There were, however, differences in MK ratings at the midyear mark, and as low ratings on evaluations and examinations may adversely affect career opportunities for trainees, it is important to continue further work examining other competencies or performance measures for potential biases.

Bias in Surgical Residency Evaluations: A Scoping Review

OBJECTIVE

Given widespread disparities in the surgical workforce and the advent of competency-based training models that rely on objective evaluations of resident performance, this review aims to describe the landscape of bias in the evaluation methods of residents in surgical training programs in the United States.

DESIGN

A scoping review was conducted within PubMed, Embase, Web of Science, and ERIC in May 2022, without a date restriction. Studies were screened and reviewed in duplicate by 3 reviewers. Data were described descriptively.

SETTING/PARTICIPANTS

English-language studies conducted in the United States that assessed bias in the evaluation of surgical residents were included.

RESULTS

The search yielded 1641 studies, of which 53 met inclusion criteria. Of the included studies, 26 (49.1%) were retrospective cohort studies, 25 (47.2%) were cross-sectional studies, and 2 (3.8%) were prospective cohort studies. The majority included general surgery residents (n = 30, 56.6%) and nonstandardized examination modalities (n = 38, 71.7%), such as video-based skills evaluations (n = 5, 13.2%). The most common performance metric evaluated was operative skill (n = 22, 41.5%). Overall, the majority of studies demonstrated bias (n = 38, 73.6%) and most investigated gender bias (n = 46, 86.8%). Most studies reported disadvantages for female trainees regarding standardized examinations (80.0%), self-evaluations (73.7%), and program-level evaluations (71.4%). Four studies (7.6%) assessed racial bias, of which all reported disadvantages for trainees underrepresented in surgery.

CONCLUSIONS

Evaluation methods for surgery residents may be prone to bias, particularly with regard to female trainees. Research is warranted regarding other implicit and explicit biases, such as racial bias, as well as for nongeneral surgery subspecialties.

Identifying Gender-Related Differences in Graduate Medical Education with the Use of a Web-Based Professionalism Monitoring Tool

OBJECTIVES

Medical education is required to ensure a healthy training and learning environment for resident physicians. Trainees are expected to demonstrate professionalism with patients, faculty, and staff. West Virginia University Graduate Medical Education (GME) initiated a Web-based professionalism and mistreatment form ("button") on our Web site for reporting professionalism breaches, mistreatment, and exemplary behavior events. The purpose of this study was to identify characteristics in resident trainees who had a "button push" activation about their behavior to better understand ways to improve professionalism in GME.

METHODS

This West Virginia University institutional review board-approved quality improvement study is a descriptive analysis of GME button push activations from July 2013 through June 2021. We compared characteristics of all of those trainees who had specific button activation(s) about their behavior. Data are reported as frequency and percentage. Nominal data and interval data were analyzed using the χ2 and the t test, respectively. P < 0.05 was significant. Logistic regression was used to analyze those differences that were significant.

RESULTS

In the 8-year study period, there were 598 button activations, and 54% (n = 324) of the activations were anonymous. Nearly all of the button reports (n = 586, 98%) were constructively resolved within 14 days. Of the 598 button activations, 95% (n = 569) were identified as involving one sex, with 66.3% (n = 377) identified as men and 33.7% (n = 192) as women. Of the 598 activations, 83.7% (n = 500) involved residents and 16.3% (n = 98) involved attendings. One-time offenders comprised 90% (n = 538), and 10% (n = 60) involved individuals who had previous button pushes about their behavior.

CONCLUSIONS

Implementation of a professionalism-monitoring tool, such as our Web-based button push, identified gender differences in the reporting of professionalism breaches, because twice as many men as women were identified as the instigator of a professionalism breech. The tool also facilitated timely interventions and exemplary behavior recognition.

Resident Operative Autonomy and Attending Verbal Feedback Differ by Resident and Attending Gender

Objectives

This study tests the null hypotheses that overall sentiment and gendered words in verbal feedback and resident operative autonomy relative to performance are similar for female and male residents.

Background

Female and male surgical residents may experience training differently, affecting the quality of learning and graduated autonomy.

Methods

A longitudinal, observational study using a Society for Improving Medical Professional Learning collaborative dataset describing resident and attending evaluations of resident operative performance and autonomy and recordings of verbal feedback from attendings from surgical procedures performed at 54 US general surgery residency training programs from 2016 to 2021. Overall sentiment, adjectives, and gendered words in verbal feedback were quantified by natural language processing. Resident operative autonomy and performance, as evaluated by attendings, were reported on 5-point ordinal scales. Performance-adjusted autonomy was calculated as autonomy minus performance.

Results

The final dataset included objective assessments and dictated feedback for 2683 surgical procedures. Sentiment scores were higher for female residents (95 [interquartile range (IQR), 4-100] vs 86 [IQR 2-100]; P < 0.001). Gendered words were present in a greater proportion of dictations for female residents (29% vs 25%; P = 0.04) due to male attendings disproportionately using male-associated words in feedback for female residents (28% vs 23%; P = 0.01). Overall, attendings reported that male residents received greater performance-adjusted autonomy compared with female residents (P < 0.001).

Conclusions

Sentiment and gendered words in verbal feedback and performance-adjusted operative autonomy differed for female and male general surgery residents. These findings suggest a need to ensure that trainees are given appropriate and equitable operative autonomy and feedback.

A Consensus-Driven Revision of the Accreditation Council for Graduate Medical Education Case Log System: Pediatric Anesthesiology Fellowship Education

BACKGROUND

Clinical experiences, quantified by case logs, are an integral part of pediatric anesthesiology fellowship programs. Accreditation of pediatric anesthesiology fellowships by the Accreditation Council of Graduate Medical Education (ACGME) and establishment of case log reporting occurred in 1997 and 2009, respectively. The specialty has evolved since then, but the case log system remains largely unchanged. The Pediatric Anesthesiology Program Directors Association (PAPDA) embarked on the development of an evidence-based case log proposal through the efforts of a case log task force (CLTF). This proposal was part of a larger consensus-building process of the Society for Pediatric Anesthesia (SPA) Task Force for Pediatric Anesthesiology Graduate Medical Education. The primary aim of case log revision was to propose an evidence-based, consensus-driven update to the pediatric anesthesiology case log system.

METHODS

This study was executed in 2 phases. The CLTF, composed of 10 program directors representing diverse pediatric anesthesiology fellowship programs across the country, utilized evidence-based literature to develop proposed new categories. After an approval vote by PAPDA membership, this proposal was included in the nationally representative, stakeholder-based Delphi process executed by the SPA Task Force on Graduate Medical Education. Thirty-seven participants engaged in this Delphi process, during which iterative rounds of surveys were used to select elements of the old and newly proposed case logs to create a final revision of categories and minimums for updated case logs. The Delphi methodology was used, with a two-thirds agreement as the threshold for inclusion.

RESULTS

Participation in the Delphi process was robust, and consensus was almost completely achieved by round 2 of 3 survey rounds. Participants suggested that total case minimums should increase from 240 to 300 (300-370). Participants agreed (75.86%) that the current case logs targeted the right types of cases, but requirements were too low (82.75%). They also agreed (85.19%) that the case log system and minimums deserved an update, and that this should be used as part of a competency-based assessment in pediatric anesthesia fellowships (96%). Participants supported new categories and provided recommended minimum numbers.

CONCLUSIONS

The pediatric anesthesiology case log system continues to have a place in the assessment of fellowship programs, but it requires an update. This Delphi process established broad support for new categories and benchmarked minimums to ensure the robustness of fellowship programs and to better prepare the pediatric anesthesiology workforce of the future for independent clinical practice.

Chinese digestive surgery interns' learning quality and English reading proficiency during COVID-19 pandemic: Comparison between face-to-face versus WeChat teaching and learning

Gastrointestinal clinical teaching is of vital important for gastrointestinal surgery interns. However, during COVID-19 pandemic, due to frequent lockdowns and essential social distancing policy implemented in China, face-to-face teaching was interrupted significantly. To find a cost-effective way to deliver medical education to ensure that teaching and learning would be continued and uninterrupted, many social media tools and mobile applications have been used in medical teaching and learning. WeChat has been frequently employed in teaching and learning in many disciplines in Chinese universities due to its powerful functions and free cost. This study compared Chinese digestive surgery interns' learning quality, English reading proficiency, and learning satisfaction in two teaching conditions: the traditional face-to-face teaching versus WeChat teaching via an experiment. The study recruited 60 final year clinical medical students, who were randomly and equally assigned into two groups: traditional face-to-face teaching versus WeChat teaching. Interns' learning quality and learning satisfaction were measured by Likert-scale questionnaires; and their English reading proficiency was measured by the reading section in a standardized English test. The results showed that interns in WeChat group had significantly higher learning quality on understanding mechanisms and current knowledge by both self-assessment and peer-assessment. WeChat group also outperformed face-to-face group on inferencing, details, and main ideas in English reading. With regard to learning satisfaction, WeChat group were higher on learning interests, learning objectives, learning format, and English reading proficiency than face-to-face group. However, interns did not differ in terms of their learning satisfaction on medical skills, which might indicate that WeChat had limitations on training interns' medical practical ability.

The History of Surgical Education in the United States: Past, Present, and Future

In just over 100 years, surgical education in the United States has evolved from a disorganized practice to a refined system esteemed worldwide as one of the premier models for the training of physicians and surgeons. But in the changing environment of health care, new challenges have arisen that could warrant a reform. To design our future, we must understand our past. The present work is not intended to be a comprehensive account of the history of American surgery. Instead, it tells the abridged history of surgical education in our country: the evolution from apprenticeships to residencies; the birth of hospital-based teaching; the impact of key historical events on training; the marks left by some preeminent characters; the conception of regulatory entities that steer our education; and, finally, how our process of training surgeons might need to be refined for the continued progress of our profession. Told in chronological order in a manner that will be memorable to readers, this story weaves together the key events that explain how our current surgical training models came to be. We conclude with a timely invitation to draw from these past lessons to redesign the future of graduate medical education, making a case for the transition to time-variable, competency-based medical education for surgical residency programs in America.