Defining Honors in the Surgery Clerkship

Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail

Clerkship grading is a core feature of evaluation for medical students' skills as physicians and is considered by most residency program directors to be an indicator of future performance and success. With the transition of the U.S. Medical Licensing Examination Step 1 score to pass/fail, there will likely be even greater reliance on clerkship grades, which raises several important issues that need to be urgently addressed. This article details the current landscape of clerkship grading and the systemic discrepancies in assessment and allocation of honors. The authors examine not only objectivity and fairness in clerkship grading but also the reliability of clerkship grading in predicting residency performance and the potential benefits and drawbacks to adoption of a pass/fail clinical clerkship grading system. In the promotion of a more fair and equitable residency selection process, there must be standardization of grading systems with consideration of explicit grading criteria, grading committees, and/or structured education of evaluators and assessors regarding implicit bias. In addition, greater adherence and enforcement of transparency in grade distributions in the Medical Student Performance Evaluation is needed. These changes have the potential to level the playing field, foster equitable comparisons, and ultimately add more fairness to the residency selection process.

New Heuristics to Stratify Applicants: Predictors of General Surgery Residency Applicant Step 1 Scores

OBJECTIVE

In 2022, United States Medical Licensing Examination (USMLE) Step 1 scores will become pass/fail. This may be problematic, as residency programs heavily rely on USMLE Step 1 scores as a metric when determining interview invitations. This study aimed to assess candidate application metrics associated with USMLE Step 1 scores to offer programs new cues for stratifying applicants.

DESIGN

Retrospective cohort study analyzing interviewed applicants to one general surgery residency program in 2019 and 2020. Applicant data analyzed included USMLE Step 1 scores, number of publications, clerkship scores, letter of recommendation scores (out of 2, scored by 0.25 interval), interview overall score (out of 5, scored by integer level), and standardized question score (out of 10). Each year, applicant's answers to one standardized behavioral question during their interview were scored by interviewers.

SETTING

Tertiary medical center, academic general surgery residency program.

PARTICIPANTS

Interviewed applicants at one general surgery residency program whose applications contained complete demographic information (203 out of 247).

RESULTS

Multiple Linear Regression revealed that higher surgical clerkship (β = 0.19, p = 0.006) and higher standardized interview question (β = 0.32, p < 0.001) scores were positively associated with applicant USMLE Step 1 score (F[7, 195] = 6.61, p < 0.001, R2 = 0.19). Letter of recommendation score, number of peer reviewed publications, gender, race, and applicant type (preliminary/categorical) were not associated with USMLE Step 1 scores.

CONCLUSIONS

With USMLE Step 1 scores transitioning to pass/fail, surgical residency programs need new selection heuristics. Surgery clerkship scores and standardized behavioral questions answered by applicants prior to the interview could provide a holistic view of applicants and help programs better stratify candidates without USMLE Step 1 scores.

What Does Honors Mean? National Analysis of Medical School Clinical Clerkship Grading

OBJECTIVE

Medical Student Performance Evaluation (MSPE) letters provide critical comparative information about clerkship performance, and are a crucial part of the surgical residency application. The elimination of USMLE Step 1 numeric reporting increases the importance of transparency, standardization, and accessibility of comparative information reported on the MSPE. The objective of our study was to measure the variability in clerkship grade reporting on the MSPE from US medical schools, particularly focusing on the highest (honors) grades.

DESIGN, SETTING, AND PARTICIPANTS

We identified representative MSPE letters from US medical schools and recorded the percentage of honors for 5 core clerkships. We grouped medical schools according to medical school rankings, geographic region, and number of grading categories RESULTS: Of 122 medical schools, 106 schools (87%) reported their grading scheme and percent honors. The most commonly used grading scheme was a 4-tier system (51/122; 42%). The percentage of honors was highly variable (from 1-91%) and did not vary by region. However, schools in the top 20 research ranking were less likely to report grade comparisons (30% vs. 10%), and more likely to award more students honors in 4 of the 5 clerkships. Schools in the top 20 primary care ranking were more likely to award more honors in the medicine clerkship.

CONCLUSIONS

There is significant variability in the number of grading tiers used and the percentage of students awarded honors across US medical schools. Factors that correlated to higher grades included schools with higher rankings, and higher ranked schools were less likely to report comparative information at all.

Does Delaying the United States Medical Licensing Examination Step 1 to After Clerkships Affect Student Performance on Clerkship Subject Examinations?

Phenomenon: Schools are considering the optimal timing of Step 1 of the United States Medical Licensing Examination (USMLE). Two primary reasons for moving Step 1 after the core clerkships are to promote deeper, more integrated basic science learning in clinical contexts and to better prepare students for the increasingly clinical focus of Step 1. Positioning Step 1 after the core clerkships leverages a major national assessment to drive learning, encouraging students to deepen their basic science knowledge while in the clinical setting. Previous studies demonstrated small increases in Step 1 scores, reductions in failure rates, and similar Step 2 Clinical Knowledge scores when Step 1 was after the clerkships. Some schools that have moved Step 1 reported declines in clinical subject examination (CSE) performance. This may be due to shortened pre-clerkship curricula, the absence of the Step 1 study period for knowledge consolidation, or exposure to fewer National Board of Medical Examiners type questions prior to taking CSEs. This multi-institutional study aimed to determine whether student performance on CSEs was affected by moving Step 1 after the core clerkships. Approach: CSE scores for students from eight schools that moved Step 1 after core clerkships between 2012 and 2016 were analyzed in a pre-post format. Hierarchical linear modeling was used to quantify the effect of the curriculum on CSE performance. Additional analysis determined if clerkship order impacted clinical subject exam performance and whether the curriculum change resulted in more students scoring in the lowest percentiles (as defined as below the national fifth percentile) before and after the curricular change. Findings: After moving Step 1 to after the clerkships, collectively these eight schools demonstrated statistically significant lower performance on four CSEs (Medicine, Neurology, Pediatrics, and Surgery) but not Obstetrics/Gynecology or Psychiatry. Comparing performance within the three years pre and post Step 1 change, differences across all clerkships ranged from 0.3 to -2.0 points, with an average difference of -1.1. CSE performance in clerkships taken early in the sequence was more affected by the curricular change, and differences gradually disappeared with subsequent examinations. Medicine and Neurology showed the largest average differences between curricular-group when taken early in the clinical year. Finally, there was a slightly higher chance of scoring below the national fifth percentile in four of the clinical subject exams (Medicine, Neurology, Pediatrics, and Psychiatry) for the cohort with Step 1 after the clerkships. Insights: Moving Step 1 after core clerkships had a small impact on CSE scores overall, with decreased scores for exams early in the clerkship sequence and an increased number of students below the fifth percentile. Score differences have minor effects on clerkship grades, but overall the size of the effect is unlikely to be educationally meaningful. Schools can use a variety of mitigation strategies to address CSE performance and Step 1 preparation in the clerkship phase.

Evidence-Based Selection of Surgical Residents

Residency programs should use a systematic method of recruitment that begins with defining unique desired candidate attributes. Commonly sought-after characteristics may be delineated via the residency application. Scores from standardized examinations taken in medical school predict academic success, and may correlate to overall performance. Strong letters of recommendation and a personal history of prior success outside the medical field both forecast success in residency. Interviews are crucial to determining fit within a program, and remain a valid measure of an applicant's ability to prosper in a particular program, even with many interviews being completed in the virtual realm.

Are We Measuring What Matters? How Student and Clerkship Characteristics Influence Clinical Grading

PURPOSE

Given the growing emphasis placed on clerkship performance for residency selection, clinical evaluation and its grading implications are critically important; therefore, the authors conducted this study to determine which evaluation components best predict a clinical honors recommendation across 3 core clerkships.

METHOD

Student evaluation data were collected during academic years 2015-2017 from the third-year internal medicine (IM), pediatrics, and surgery clerkships at the University of Alabama at Birmingham School of Medicine. The authors used factor analysis to examine 12 evaluation components (12 items), and they applied multilevel logistic regression to correlate evaluation components with a clinical honors recommendation.

RESULTS

Of 3,947 completed evaluations, 1,508 (38%) recommended clinical honors. The top item that predicted a clinical honors recommendation was clinical reasoning skills for IM (odds ratio [OR] 2.8; 95% confidence interval [CI], 1.9 to 4.2; P < .001), presentation skills for surgery (OR 2.6; 95% CI, 1.6 to 4.2; P < .001), and knowledge application for pediatrics (OR 4.8; 95% CI, 2.8 to 8.2; P < .001). Students who spent more time with their evaluators were more likely to receive clinical honors (P < .001), and residents were more likely than faculty to recommend clinical honors (P < .001). Of the top 5 evaluation items associated with clinical honors, 4 composed a single factor for all clerkships: clinical reasoning, knowledge application, record keeping, and presentation skills.

CONCLUSIONS

The 4 characteristics that best predicted a clinical honors recommendation in all disciplines (clinical reasoning, knowledge application, record keeping, and presentation skills) correspond with traditional definitions of clinical competence. Structural components, such as contact time with evaluators, also correlated with a clinical honors recommendation. These findings provide empiric insight into the determination of clinical honors and the need for heightened attention to structural components of clerkships and increased scrutiny of evaluation rubrics.

Clerkship Grading and the U.S. Economy: What Medical Education Can Learn From America's Economic History

Clerkship grades (like money) are a social construct that function as the currency through which value exchanges in medical education are negotiated between the system's various stakeholders. They provide a widely recognizable and efficient medium through which learner development can be assessed, tracked, compared, and demonstrated and are commonly used to make decisions regarding progression, distinction, and selection for residency. However, substantial literature has demonstrated how grades imprecisely and unreliably reflect the value of learners. In this article, the authors suggest that challenges with clerkship grades are fundamentally tied to their role as currency in the medical education system. Associations are drawn between clerkship grades and the history of the U.S. economy; 2 major concepts are highlighted: regulation and stock prices. The authors describe the history of these economic concepts and how they relate to challenges in clerkship grading. Using lessons learned from the history of the U.S. economy, the authors then propose a 2-step solution to improve upon grading for future generations of medical students: (1) transition from grades to a federally regulated competency-based assessment model and (2) development of a departmental competency letter that incorporates competency-based assessments rather than letter grades and meets the needs of program directors.

Building a Shared Mental Model of Competence Across the Continuum: Trainee Perceptions of Subinternships for Residency Preparation

INTRODUCTION

Toward a vision of competency-based medical education (CBME) spanning the undergraduate to graduate medical education (GME) continuum, University of Minnesota Medical School (UMMS) developed the Subinternship in Critical Care (SICC) offered across specialties and sites. Explicit course objectives and assessments focus on internship preparedness, emphasizing direct observation of handovers (Core Entrustable Professional Activity, "EPA," 8) and cross-cover duties (EPA 10).

METHODS

To evaluate students' perceptions of the SICC's and other clerkships' effectiveness toward internship preparedness, all 2016 and 2017 UMMS graduates in GME training (n = 440) were surveyed regarding skill development and assessment among Core EPAs 1, 4, 6, 8, 9, 10. Analysis included descriptive statistics plus chi-squared and Kappa agreement tests.

RESULTS

Respondents (n = 147, response rate 33%) rated the SICC as a rotation during which they gained most competence among EPAs both more (#4, 57% rated important; #8, 75%; #10, 70%) and less explicit (#6, 53%; #9, 69%) per rotation objectives. Assessments of EPA 8 (80% rated important) and 10 (76%) were frequently perceived as important toward residency preparedness. Agreement between importance of EPA development and assessment was moderate (Kappa = 0.40-0.59, all surveyed EPAs).

CONCLUSIONS

Graduates' perceptions support the SICC's educational utility and assessments. Based on this and other insight from the SICC, the authors propose implications toward collectively envisioning the continuum of physician competency.

Evaluating Urology Residency Applications: What Matters Most and What Comes Next?

PURPOSE OF REVIEW

In light of the announcement that the United States Medical Licensing Examination Step 1 exam will transition to pass/fail reporting, we reviewed recent literature on evaluating residency applicants with a focus on identifying objective measurements of applicant potential.

RECENT FINDINGS

References from attending urologists, Step 1 scores, overall academic performance, and research publications are among the most important criteria used to assess applicants. There has been a substantial increase in the average number of applications submitted per applicant, with both applicants and residency directors indicating support for a cap on the number of applications that may be submitted. Additionally, there are increasing efforts to promote diversity with the goal of improving care and representation in urology. Despite progress in standardizing interview protocols, inappropriate questioning remains an issue. Opportunities to improve residency application include promoting diversity, enforcing prohibitions of illegal practices, limiting application numbers, and finding more transparent and equitable screening measures to replace Step 1.

The Current Status of Medical School Clerkship Grades in Residency Applicants

BACKGROUND

Two criteria that have been investigated for evaluating orthopedic surgery residency candidates are achieving an "honors" grade during a surgery clerkship and the total number of honors grades received in all clerkships. Unfortunately, the rate of honors grades given and the criteria for earning an honors grade differ between medical schools, making comparison of applicants from different medical schools difficult.

OBJECTIVE

We measured the rate of honors grades in clerkships at different medical schools in the United States to examine the utility of clerkship grades in evaluating orthopedic surgery residency applicants.

METHODS

Adequate data via the Electronic Residency Application Service were available for 86 of 142 Association of American Medical Colleges medical schools from the 2017 Match cycle. Descriptive statistics and Wilcoxon rank sum tests were performed to identify differences in grade distributions within each clerkship and in school ranking for research by U.S. News & World Report.

RESULTS

For the surgery clerkship, the median rate of honors grades given was 32.5% (range 5%-67%). There was a high rate of interinstitutional variability in all clerkships. We were unable to demonstrate a statistically significant relationship between research ranking and percentage honors grades given for individual clerkships.

CONCLUSIONS

A standardized method for grading medical students during clinical clerkships does not exist, resulting in a high degree of interinstitutional variability. Surgery clerkship grades are an unreliable measure for comparing orthopedic surgery residency applicants from different medical schools. Standardized measures of applicant evaluation might be helpful in the future.

Which Student Characteristics Are Most Important in Determining Clinical Honors in Clerkships? A Teaching Ward Attending Perspective

PURPOSE

To explore faculty perspectives on which characteristics of high-performing clerkship students are most important when determining an honors or top grade designation for clinical performance.

METHOD

In 2016-2017, the authors surveyed faculty (teaching ward attendings) for internal medicine clerkships and 1 pediatrics clerkship in inpatient settings at 5 U.S. academic medical centers. Survey items were framed around competencies, 24 student characteristics, and attitudes toward evaluation. Factor analysis examined constructs defining high-performing students.

RESULTS

Of 516 faculty invited, 319 (62%) responded. The top 5 characteristics as rated by respondents were taking ownership, clinical reasoning, curiosity, dependability, and high ethical standards (in descending order). Twenty-one characteristics fit into 3 factors (Cronbach alpha, 0.81-0.87). Clinical reasoning did not fit into a factor. Factor 1 was the most important (mean rating, 8.7/10 [95% confidence interval (CI), 8.6-8.8]). It included professionalism components (ownership, curiosity, dependability, high ethical standards), presentation and interviewing skills, seeking feedback, and documentation. Factor 2 (mean, 7.9 [95% CI, 7.7-8.0]) included aspects of teamwork and communication, such as positive attitude and comments from others. Factor 3 (mean, 7.6 [95% CI, 7.4-7.7]) addressed systems-based thinking, including patient safety and care transitions.

CONCLUSIONS

Professionalism components, clinical reasoning, and curiosity were among the most important characteristics distinguishing high-performing clerkship students. These may represent behaviors that are highly valued, observable, and relevant to training stage. Improved definition of the characteristics associated with clinical honors would assist students, faculty, and residency program directors when interpreting clinical performance within core clerkships.

What Defines an Honors Student? Survey of Pediatric and Internal Medicine Faculty Perspectives

OBJECTIVE

Although considerable emphasis is placed on the attainment of honors in core medical school clerkships, little is known about what student characteristics are used by attending physicians to earn this designation. The purpose of this study was to evaluate what values and characteristics that attending physicians consider important in the evaluation of Pediatrics and Internal Medicine clerkship students for clinical honors designation.

METHODS

This cross-sectional survey study was framed around Accreditation Council for Graduate Medical Education (ACGME) competencies. It was administered at three tertiary care hospitals associated with one large medical school in an urban setting. Teaching ward attendings in Pediatrics and Internal Medicine who evaluated third-year medical students between 2013 and 2016 were surveyed.

RESULTS

Overall, Pediatric and Internal Medicine faculty demonstrated close agreement in which competencies were most important in designating clinical honors. Both groups believed that professionalism was the most important factor and that systems-based practice and patient care were among the least important factors. The only competency with a significant difference between the two groups was systems-based practice, with Internal Medicine placing more emphasis on the coordination of patient care and understanding social determinants of health.

CONCLUSIONS

Professionalism, communication skills, and medical knowledge are the most important characteristics when determining clinical honors on Pediatrics and Internal Medicine clerkships.

Identifying High-Performing Students in Inpatient Clerkships: A Qualitative Study

OBJECTIVES

Examine fundamental behaviors and characteristics that attending physicians in inpatient settings utilize to identify high-performing clerkship students.

METHODS

We employed written comment data from a cross-sectional survey of Internal Medicine and Pediatrics attending physicians at a single academic medical center in the southern USA. Free-text responses regarding factors that faculty consider when assigning honors grades were analyzed by four trained researchers (interrater agreement 0.87) using conventional content analysis to identify themes.

RESULTS

Seventy-nine of 141 (56%) attending physicians who were surveyed provided 90 comments.Four major theme areas for recognizing higher performing clerkship students were identified: Taking Ownership of Patient Care (35%), Medical Knowledge and Clinical Reasoning (20%), Team Orientation (15%), and Awareness of Opportunities for Growth and Progress (13%).

CONCLUSION

Internal Medicine and Pediatric attending physicians identified characteristics that contributed to four themes in the determination of a high-performing medical student. These findings are particularly salient, as they highlight that commitment to patients, application of clinical knowledge and skills, teamwork, and awareness of growth and progress are valued by attending physicians for identifying top performing students in inpatient settings.

Using the Clerkship Shelf Exam Score as a Qualification for an Overall Clerkship Grade of Honors: A Valid Practice or Unfair to Students?

Most clerkships require medical students to pass the National Board of Medical Examiners (NBME) subject (shelf) exam to pass the clerkship. Many use the NBME's recommended honors cut score on the shelf exam to determine medical student eligibility for an overall clerkship grade of honors. This use of a conjunctive scoring model for determining honors is inconsistent with the logic behind the intended use of this model for making pass-fail determinations. Further, many clerkships use grading systems that employ both this conjunctive model for honors eligibility and a compensatory scoring model for determining the overall clerkship grade. For students who fall short of the shelf exam honors cut score, such a grading system effectively increases the weighting of shelf exam performance and decreases the clerkship's transparency about the weighting of performance on other assessments toward the clerkship composite score and overall grade. It may also lead to contradictory grading results in which student B has a higher composite score than student A, yet student B receives a lower overall grade. The author illustrates how to calculate a weight for shelf exam performance that would be fairer to students and would help create a more transparent weighting scheme for the grading system. The author recommends that clerkships restructure their grading systems so that shelf exam honors-eligibility cut scores are not used as conjunctive criteria for determining overall clerkship grades of honors. A reexamination of the NBME's practice of suggesting honors-eligibility cut scores for shelf exams is also recommended.

Generalizability of Competency Assessment Scores Across and Within Clerkships: How Students, Assessors, and Clerkships Matter

PURPOSE

Many factors influence the reliable assessment of medical students' competencies in the clerkships. The purpose of this study was to determine how many clerkship competency assessment scores were necessary to achieve an acceptable threshold of reliability.

METHOD

Clerkship student assessment data were collected during the 2015-2016 academic year as part of the medical school assessment program at the University of Michigan Medical School. Faculty and residents assigned competency assessment scores for third-year core clerkship students. Generalizability (G) and decision (D) studies were conducted using balanced, stratified, and random samples to examine the extent to which overall assessment scores could reliably differentiate between students' competency levels both within and across clerkships.

RESULTS

In the across-clerkship model, the residual error accounted for the largest proportion of variance (75%), whereas the variance attributed to the student and student-clerkship effects was much smaller (7% and 10.1%, respectively). D studies indicated that generalizability estimates for eight assessors within a clerkship varied across clerkships (G coefficients range = 0.000-0.795). Within clerkships, the number of assessors needed for optimal reliability varied from 4 to 17.

CONCLUSIONS

Minimal reliability was found in competency assessment scores for half of clerkships. The variability in reliability estimates across clerkships may be attributable to differences in scoring processes and assessor training. Other medical schools face similar variation in assessments of clerkship students; therefore, the authors hope this study will serve as a model for other institutions that wish to examine the reliability of their clerkship assessment scores.

Surgical clerkship or medical clerkship first: Does it make a difference?

INTRODUCTION

This study compares NBME surgical clerkship scores of students who completed their medicine clerkship before their surgical clerkship with the performance of those who had not previously completed their medical clerkship.

METHODS

The study included 815 New York University School of Medicine students from the years 2014-2018 (571 students took medicine first, while 244 took surgery first). Performance on the surgical clerkship was assessed using the NBME SHELF examination. Statistical comparisons were performed via 2-tailed, independent-samples, unequal-variance t-tests.

RESULTS

Mean NBME surgical SHELF scores of the students who had previously taken medicine were significantly higher than students who had not (mean 78.6 vs. 73.5, p < 0.001). Students who had solely medicine (as their first clerkship) before surgery also performed significantly better (mean 78.8 vs. 73.5, p < 0.001). Students who completed surgery later in the year did not perform better on the surgical SHELF, so long as both surgical clerkship cohorts had completed medicine.

CONCLUSION

Students who completed their core medical clerkship prior to their surgical clerkship scored significantly better on the NBME surgical SHELF examination.