Application Factors May Not Be Predictors of Success Among General Surgery Residents as Measured by ACGME Milestones

Detecting residents at risk of attrition - A Singapore pathology residency's experience

The SingHealth Pathology Residency Program (SHPRP) is a 5-year postgraduate training program in Singapore. We face the problem of resident attrition, which has a significant impact on the individual, program and healthcare providers. Our residents are regularly evaluated, using in-house evaluations as well as assessments required in our partnership with the Accreditation Council for Graduate Medical Education International (ACGME-I). We hence sought to determine if these assessments were able to distinguish residents who would attrite from residents who would graduate successfully. Retrospective analysis of existing residency assessments was performed on all residents who have separated from SHPRP and compared with residents currently in senior residency or graduated from the program. Statistical analysis was performed on quantitative assessment methods of Resident In-Service Examination (RISE), 360-degree feedback, faculty assessment, Milestones and our own annual departmental mock examination. Word frequency analysis of narrative feedback from faculty assessment was used to generate themes. Since 2011, 10 out of 34 residents have separated from the program. RISE, Milestone data and the departmental mock examination showed statistical significance in discriminating residents at risk of attrition for specialty-related reasons from successful residents. Analysis of narrative feedback showed that successful residents performed better in areas of organization, preparation with clinical history, application of knowledge, interpersonal communication and achieving sustained progress. Existing assessment methods used in our pathology residency program are effective in detecting residents at risk of attrition. This also suggests applications in the way that we select, assess and teach residents.

Simulation as Soothsayer: Simulated Surgical Skills MMIs During Residency Interviews are Associated With First Year Residency Performance

OBJECTIVE

The main consideration during residency recruitment is identifying applicants who will succeed during residency. However, few studies have identified applicant characteristics that are associated with competency development during residency, such as the Accreditation Council for Graduate Medical Education milestones. As mini multiple interviews (MMIs) can be used to assess various competencies, we aimed to determine if simulated surgical skills MMI scores during a general surgery residency interview were associated with Accreditation Council for Graduate Medical Education milestone ratings at the conclusion of intern year.

DESIGN

Retrospective cohort study. Interns' Step 1 and 2 clinical knowledge (CK) scores, interview day simulated surgical skills MMI overall score, traditional faculty interview scores, average overall milestone ratings in the spring of residency, and intern American Board of Surgery In-Training Examination (ABSITE) percentile scores were gathered. Two multiple linear regression were performed analyzing the association between Step 1, Step 2 CK, MMI, and traditional faculty interview scores with (1) average overall milestone rating and (2) ABSITE percentile scores, controlling for categorical/preliminary intern classification.

SETTING

One academic medical center PARTICIPANTS: General surgery interns matriculating in 2020-2021 RESULTS: Nineteen interns were included. Multiple linear regression revealed that higher overall simulated surgical skills MMI score was associated with higher average milestone ratings (β = .45, p = 0.03) and higher ABSITE score (β = .43, p = 0.02) while neither Step 1, Step 2 CK, nor faculty interview scores were significantly associated with average milestone ratings.

CONCLUSIONS

Surgical residency programs invest a tremendous amount of effort into training residents, thus metrics for predicting applicants that will succeed are needed. Higher scores on a simulated surgical skills MMIs are associated with higher milestone ratings 1 year into residency and higher intern ABSITE percentiles. These results indicate a noteworthy method, simulated surgical skills MMIs, as an additional metric that may select residents that will have early success in residency.

Characteristics Associated With Outstanding General Surgery Residency Graduate Performance, as Rated by Surgical Educators

Importance

Characteristics of outstanding graduating surgical residents are currently undefined. Identifying these qualities may be important in guiding resident selection and resident education.

Objective

To determine characteristics that are most strongly associated with being rated as an outstanding graduating surgical resident.

Design, Setting, and Participants

The multi-institutional study had 3 phases. First, an expert panel developed a list of characteristics embodied by top graduating surgical residents. Second, groups of faculty from 14 US general-surgery residency programs ranked 2017 through 2020 graduates into quartiles of overall performance. Third, faculty evaluated their graduates on each characteristic using a 5-point Likert scale. Data were analyzed using Spearman rank-order correlation to identify which individual characteristics were associated with overall graduate performance. A least absolute shrinkage and selection operator (LASSO) ordinal regression was performed to select a parsimonious model to predict the outcome of overall performance rating from individual characteristic scores.

Main Outcome and Measures

Surgical educators' rankings of general surgery residency graduates' overall performance.

Results

Fifty faculty from 14 US residency programs with a median of 13 (range, 5-30) years of surgical education experience evaluated 297 general surgery residency graduates. Surgical educators identified 21 characteristics that they believed outstanding graduating surgical residents possessed. Two hundred ninety-seven surgical residency graduates were evaluated. Higher scores in every characteristic correlated with better overall performance. Characteristics most strongly associated with higher overall performance scores were surgical judgment (r = 0.728; P < .001), leadership (r = 0.726; P < .001), postoperative clinical skills (r = 0.715; P < .001), and preoperative clinical skills (r = 0.707; P < .001). The remainder of the characteristics were moderately associated with overall performance. The LASSO regression model identified 3 characteristics from which overall resident performance could be accurately predicted without measuring other qualities: surgical judgment (odds ratio [OR] per 1 level of 5-level Likert scale OR, 1.27; 95% CI, 1.03-1.51), leadership (OR, 1.27; 95% CI, 1.06-1.48), and medical knowledge (OR, 1.16; 95% CI, 1.01-1.33).

Conclusions and Relevance

All individual characteristics identified by surgical educators as being qualities of outstanding graduating surgical residents were positively associated with overall graduate performance. Surgical judgment and leadership skills had the strongest individual associations. Assessment of only 3 qualities (surgical judgment, leadership, and medical knowledge) were required to predict overall resident performance ratings. These findings highlight the importance of developing specific surgical judgment and leadership skills curricula and assessments during surgical residency.

The Potential Application of Mindset Theory to Surgical Education

Mindset theory proposes that individuals hold a range of beliefs regarding the malleability of attributes such as intellect and skill. Within surgery, mindset theory has been discussed as a way of understanding achievement in both the cognitive and technical aspects of learning surgery. A review of the literature reveals a limited body of research that has addressed the mindsets of surgeons or by extension, tied those mindsets to outcomes. Within health professions education, mindset theory has been studied more broadly, but the benefits of mindset theory are largely assumed and drawn from education research regarding children and adolescents. Though mindset theory has gained traction, there has been debate regarding the traits associated with growth and fixed mindsets. The strongest evidence from primary and secondary education shows that low socioeconomic status and academically at-risk students can benefit the most from mindset interventions, and these findings may extend to surgical learning as well. Mindset theory offers an interesting lens to better understand surgical education, but more research is needed to characterize the mindsets of surgeons and understand how these mindsets influence performance and outcomes.

Standardized oral examinations allow for assessment of medical student clinical knowledge and decrease racial grading differences in a surgery clerkship

BACKGROUND

Oral examinations are not consistently included in third-year medical student clerkships. When included, they are often unstructured, leaving room for variations in difficulty or scoring. Previous research has demonstrated differences in clinical grade achievement, with underrepresented in medicine students receiving significantly lower grades than White students.

METHODS

We designed a structured oral examination for third-year medical students on the surgery clerkship. Students completed 2 oral examination scenarios and were evaluated on their ability to complete a history and diagnostic workup, interpret laboratory and imaging results, and devise a treatment plan. Scores from our examination were compared to previous, unstructured oral examination scores and to student demographics. Students and faculty were surveyed regarding their experience.

RESULTS

Third-year medical students demonstrated strong knowledge of multiple surgical diseases. The greatest number of errors occurred in treatment planning (P < .001). Third-year medical students receiving honors clerkship grades achieved higher percentages of correct items on their oral examination. (94.8% vs 90.4%) (P = .02). Evaluation of prior unstructured oral examinations found underrepresented in medicine students received lower scores than White students (P = .04). After implementation of our structured examination, no difference was seen between the scores of underrepresented in medicine and White students (P = .99).

CONCLUSION

We implemented a standardized oral examination for third-year medical students on the surgery clerkship with student and faculty satisfaction and demonstrated the ability to determine domains of knowledge weakness. The application of our structured oral examination helped to address nonspecific grading practices and eliminate oral examination grade differences between underrepresented in medicine and White students.

General Surgery Residency Match: Time for More than a Virtual Change

The 2020-2021 General Surgery Residency Match presents unique challenges in the setting of the COVID-19 pandemic and highlights pre-existing concerns. In order to move toward an equitable and manageable surgical residency application process for both programs and applicants, systemic change is warranted.

Ethnic/Racial Bias in Medical School Performance Evaluation of General Surgery Residency Applicants

OBJECTIVE

Differential use of communal terms (caring/unselfish traits) versus agentic terms (goal-oriented/leadership/achievement traits) may reveal bias and has been extensively reported in letters of recommendation for residency. We evaluated bias in medical student performance evaluations (MSPE) of general surgery residency applicants.

DESIGN

This is a retrospective study evaluating ethnic/race bias, as measured by differential use of agentic and communal terms, in MSPEs of residency applicants. 50% of MSPEs were randomly selected. An ethnic bias calculator derived from an open-source online gender bias calculator was populated with a list of validated agentic and communal terms. Relative frequency of communal and agentic terms was used to estimate bias. Multivariable regression was used to assess the association between the terms and ethnicity/race.

PARTICIPANTS

US medical students applying for a categorical surgery residency position at a single academic institution for a single Match cycle.

RESULTS

A total of 339 MSPEs were reviewed from 119 US medical schools. Genders were equally represented (women, 51.6%); most participants were white and Asian applicants (79.1%). Overall, MSPEs were more agency biased (65.2%) than communal biased (16.2%) or neutral (18.6%). MSPEs for Black and Hispanic/Latinx applicants were more likely to contain communal rather than agentic terms (adjusted OR: 3.02, 95% CI: 1.52-6.02) when compared to white and Asian applicants. This finding was independent of MSPE writer's gender or rank.

CONCLUSIONS

Surgery residency applicants self-identifying as Black and Hispanic/Latinx were more likely to be described using communal traits compared to white and Asian applicants, suggesting ethnic/racial bias. Such differences in language utilized in MSPEs may impact residency opportunities for applicants who are under-represented in medicine. Educational efforts aimed at MSPE writers may help to reduce bias.

Evidence-Based Selection of Surgical Residents

Residency programs should use a systematic method of recruitment that begins with defining unique desired candidate attributes. Commonly sought-after characteristics may be delineated via the residency application. Scores from standardized examinations taken in medical school predict academic success, and may correlate to overall performance. Strong letters of recommendation and a personal history of prior success outside the medical field both forecast success in residency. Interviews are crucial to determining fit within a program, and remain a valid measure of an applicant's ability to prosper in a particular program, even with many interviews being completed in the virtual realm.