Clerkship Grading Committees: the Impact of Group Decision-Making for Clerkship Grading

Efforts to Reduce Bias in Clerkship Evaluations: A CERA Study

Introduction

As the number of medical students who identify as underrepresented in medicine (URiM) increases, the disparities related to gender and URiM status persist. This study examines the current initiatives within family medicine clerkships to reduce bias in evaluations.

Methods

Our 10-item survey was included as a module in the 2022 Council of Academic Family Medicine Educational Research Alliance national survey of family medicine clerkship directors. Our survey questions asked about whether programs had strategies to reduce bias in student evaluations, antiracism initiatives, perceptions on effectiveness of the initiatives, and type and cadence of faculty development on evaluations for preceptors.

Results

The overall response rate for the survey was 59.12% (94/159); all respondents completed our module. Seventy percent said they had implemented strategies to reduce bias in evaluations, 60% felt these were effective, and 80% felt that reducing bias in evaluations was a priority. The majority, 89/91(95%), indicated that their medical schools had a current social justice, diversity, or antiracism initiative. We identified a positive association between specific antibias medical school initiatives and clerkship directors undertaking practices to reduce bias in evaluations (P=.005).

Conclusions

Most programs had implemented strategies to reduce bias and felt that doing so was a priority. Community-based preceptors were less likely to have faculty development around reducing bias compared to those in academics. Further improvements may need to prioritize including community preceptors in educational efforts to reduce bias.

Innovations report: A grading committee for a required fourth-year emergency medicine clerkship

Background

Assessment of medical students in the clinical learning environment is fraught with challenges. Seemingly small variations in clinical clerkship evaluation can significantly impact a student's future. As such, the integrity of the grade selection process must be heavily scrutinized. Group decision making in the form of a clerkship grading committee may be part of a solution to address this complex problem.

Methods

We conducted a retrospective observational study to describe grading committee decisions for a required fourth-year EM clerkship from August 2021 to April 2022. Literature on best practices for group decision making and assessment were reviewed. This informed the development and implementation of the committee process. Each committee meeting was video recorded and coded for discussion time per student, times the committee grade differed from historical-grade cutoffs with reasoning, and the frequency a committee member voiced a first-hand account of student performance.

Results

Data from nine meetings was reviewed (86 students). The mean discussion time per student was 2 min 13 s (range 11 s to 9 min 22 s). The final committee decision differed from historical-grade cutoffs for nine students (10%), six were adjusted above and three below. In 64% (55/86) of student reviews a committee member voiced a first-hand experience of working with the student. Positive grade adjustments were made due to outlier evaluations and negative adjustments were made for professionalism concerns.

Conclusions

Grading committees are a means to conduct a comprehensive review of student performance and offer shared ownership of the grade decision among committee members. More study is needed to directly determine their potential benefit and role in clerkship grading.

Frontline assessors' opinions about grading committees in a medicine clerkship

BACKGROUND

Collective decision-making by grading committees has been proposed as a strategy to improve the fairness and consistency of grading and summative assessment compared to individual evaluations. In the 2020-2021 academic year, Washington University School of Medicine in St. Louis (WUSM) instituted grading committees in the assessment of third-year medical students on core clerkships, including the Internal Medicine clerkship. We explored how frontline assessors perceive the role of grading committees in the Internal Medicine core clerkship at WUSM and sought to identify challenges that could be addressed in assessor development initiatives.

METHODS

We conducted four semi-structured focus group interviews with resident (n = 6) and faculty (n = 17) volunteers from inpatient and outpatient Internal Medicine clerkship rotations. Transcripts were analyzed using thematic analysis.

RESULTS

Participants felt that the transition to a grading committee had benefits and drawbacks for both assessors and students. Grading committees were thought to improve grading fairness and reduce pressure on assessors. However, some participants perceived a loss of responsibility in students' grading. Furthermore, assessors recognized persistent challenges in communicating students' performance via assessment forms and misunderstandings about the new grading process. Interviewees identified a need for more training in formal assessment; however, there was no universally preferred training modality.

CONCLUSIONS

Frontline assessors view the switch from individual graders to a grading committee as beneficial due to a perceived reduction of bias and improvement in grading fairness; however, they report ongoing challenges in the utilization of assessment tools and incomplete understanding of the grading and assessment process.

Current practices and perspectives on clerkship grading in obstetrics and gynecology

BACKGROUND

Clerkship grades in obstetrics and gynecology play an increasingly important role in the competitive application process to residency programs. An analysis of clerkship grading practices has not been queried in the past 2 decades in our specialty.

OBJECTIVE

This study aimed to investigate obstetrics and gynecology clerkship directors' practices and perspectives in grading.

STUDY DESIGN

A 12-item electronic survey was developed and distributed to clerkship directors with active memberships in the Association of Professors of Gynecology and Obstetrics.

RESULTS

A total of 174 of 236 clerkship directors responded to the survey (a response rate of 73.7%). Respondents reported various grading systems with the fewest (20/173 [11.6%]) using a 2-tiered or pass or fail system and the most (72/173 [41.6%]) using a 4-tiered system. Nearly one-third of clerkship directors (57/163 [35.0%]) used a National Board of Medical Examiners subject examination score threshold to achieve the highest grade. Approximately 45 of 151 clerkship directors (30.0%) had grading committees. Exactly half of the clerkship directors (87/174 [50.0%]) reported requiring unconscious bias training for faculty who assess students. In addition, some responded that students from groups underrepresented in medicine (50/173 [28.9%]) and introverted students (105/173 [60.7%]) received lower evaluations. Finally, 65 of 173 clerkship directors (37.6%) agreed that grades should be pass or fail.

CONCLUSION

Considerable heterogeneity exists in obstetrics and gynecology clerkship directors' practices and perspectives in grading. Strategies to mitigate inequities and improve the reliability of grading include the elimination of a subject examination score threshold to achieve the highest grade and the implementation of both unconscious bias training and grading committees.

"My Assessments Are Biased!" Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education

Assessing learners is foundational to their training and developmental growth throughout the medical education continuum. However, growing evidence shows the prevalence and impact of harmful bias in assessments in medical education, accelerating the urgency to identify solutions. Assessment bias presents a critical problem for all stages of learning and the broader educational system. Bias poses significant challenges to learners, disrupts the learning environment, and threatens the pathway and transition of learners into health professionals. While the topic of assessment bias has been examined within the context of measurement literature, limited guidance and solutions exist for learners in medical education, particularly in the clinical environment. This article presents an overview of assessment bias, focusing on clinical learners. A definition of bias and its manifestations in assessments are presented. Consequences of assessment bias are discussed within the contexts of validity and fairness and their impact on learners, patients/caregivers, and the broader field of medicine. Messick's unified validity framework is used to contextualize assessment bias; in addition, perspectives from sociocultural contexts are incorporated into the discussion to elaborate the nuanced implications in the clinical training environment. Discussions of these topics are conceptualized within the literature and the interventions used to date. The article concludes with practical recommendations to overcome bias and to develop an ideal assessment system. Recommendations address articulating values to guide assessment, designing assessment to foster learning and outcomes, attending to assessment procedures, promoting continuous quality improvement of assessment, and fostering equitable learning and assessment environments.

Understanding and Addressing Bias in Grading: Progress at Washington University School of Medicine

In 2020, the authors published work examining disparities in clerkship grading and found students who identify as White were more likely to receive honors grades compared with students from races/ethnicities traditionally underrepresented in medicine. Through a quality improvement approach, the authors identified the following 6 areas where improvements could mitigate grading disparities and, therefore, put processes in place to: ensure equitable access to examination preparation resources, change student assessment, develop medical student curriculum interventions, improve the learning environment, change house staff and faculty recruitment and retention practices, and provide ongoing program evaluation and continuous quality improvement processes to monitor for success. While the authors cannot yet be sure that they have achieved their goal of promoting equity in grading, they believe this evidence-based, multipronged intervention is a clear step in the right direction and encourage other schools to consider a similar approach to tackling this critically important problem at their own institutions.

Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail

Clerkship grading is a core feature of evaluation for medical students' skills as physicians and is considered by most residency program directors to be an indicator of future performance and success. With the transition of the U.S. Medical Licensing Examination Step 1 score to pass/fail, there will likely be even greater reliance on clerkship grades, which raises several important issues that need to be urgently addressed. This article details the current landscape of clerkship grading and the systemic discrepancies in assessment and allocation of honors. The authors examine not only objectivity and fairness in clerkship grading but also the reliability of clerkship grading in predicting residency performance and the potential benefits and drawbacks to adoption of a pass/fail clinical clerkship grading system. In the promotion of a more fair and equitable residency selection process, there must be standardization of grading systems with consideration of explicit grading criteria, grading committees, and/or structured education of evaluators and assessors regarding implicit bias. In addition, greater adherence and enforcement of transparency in grade distributions in the Medical Student Performance Evaluation is needed. These changes have the potential to level the playing field, foster equitable comparisons, and ultimately add more fairness to the residency selection process.

Call for Action: Undergraduate Medical Education Assessment-Advocating for a Transition From Tiered to Competency-Based Grading

Traditionally, undergraduate medical education (UME) grading has been based on a tiered system. Tier-based grading can cause anxiety as medical students are compared to their peers. Students then become overly driven by the pursuit of creating favorable impressions by supervisors as well as by high grades. Additionally, the emphasis on normative parameters appears to misalign with the goal of UME which is to not sort learners into different residency programs but to train future doctors to meet the needs of society. This commentary is a call for action to shift from utilizing a normative-based grading paradigm in UME to implementing one in which learners are being assessed on their ability to attain specific competencies. It is important that UME transitions to competency-based assessments as the graduate medical education (GME) realm has already adopted this framework.

Hawks and Doves: Perceptions and Reality of Faculty Evaluations

OBJECTIVES

Internal medicine clerkship grades are important for residency selection, but inconsistencies between evaluator ratings threaten their ability to accurately represent student performance and perceived fairness. Clerkship grading committees are recommended as best practice, but the mechanisms by which they promote accuracy and fairness are not certain. The ability of a committee to reliably assess and account for grading stringency of individual evaluators has not been previously studied.

METHODS

This is a retrospective analysis of evaluations completed by faculty considered to be stringent, lenient, or neutral graders by members of a grading committee of a single medical college. Faculty evaluations were assessed for differences in ratings on individual skills and recommendations for final grade between perceived stringency categories. Logistic regression was used to determine if actual assigned ratings varied based on perceived faculty's grading stringency category.

RESULTS

"Easy graders" consistently had the highest probability of awarding an above-average rating, and "hard graders" consistently had the lowest probability of awarding an above-average rating, though this finding only reached statistical significance only for 2 of 8 questions on the evaluation form (P = .033 and P = .001). Odds ratios of assigning a higher final suggested grade followed the expected pattern (higher for "easy" and "neutral" compared to "hard," higher for "easy" compared to "neutral") but did not reach statistical significance.

CONCLUSIONS

Perceived differences in faculty grading stringency have basis in reality for clerkship evaluation elements. However, final grades recommended by faculty perceived as "stringent" or "lenient" did not differ. Perceptions of "hawks" and "doves" are not just lore but may not have implications for students' final grades. Continued research to describe the "hawk and dove effect" will be crucial to enable assessment of local grading variation and empower local educational leadership to correct, but not overcorrect, for this effect to maintain fairness in student evaluations.

Moving toward Mastery: Changes in Student Perceptions of Clerkship Assessment with Pass/Fail Grading and Enhanced Feedback

ProblemClerkship grades contribute to a summative assessment culture in clerkships and can therefore interfere with students' learning. For example, by focusing on summative, tiered clerkship grades, students often discount accompanying feedback that could inform future learning. This case report seeks to explore whether an assessment system intervention which eliminated tiered grades and enhanced feedback was associated with changes in student perceptions of clerkship assessment and perceptions of the clinical learning environment. Intervention: In January 2019, our institution eliminated tiered clerkship grading (honors/pass/fail) for medical students during the core clerkship year and implemented pass/fail clerkship grading along with required twice weekly, work-based assessments for formative feedback. Context: In this single institution, cross-sectional survey study, we collected data from fourth-year medical students one year after an assessment system intervention. The intervention entailed changing from honors/pass/fail to pass/fail grading in all eight core clerkships and implementing an electronic system to record twice-weekly real-time formative work-based assessments. The survey queried student perceptions on the fairness and accuracy of grading and the clinical learning environment-including whether clerkships were mastery- or performance-oriented. We compared responses from students one year after the assessment intervention to those from the class one year before the intervention. Comparisons were made using unpaired, two-tailed t-tests or chi-squared tests as appropriate with Cohen's d for effect size estimation for score differences. Content analysis was used to analyze responses from two open-ended questions about feedback and grading. Impact: Survey response rates were similar before and after intervention (76% (127/168) vs. 72% (118/163), respectively) with no between-group differences in demographics. The after-intervention group showed statistically significant increases in the following factors: "grades are transparent and fair" (Cohen's d = 0.80), "students receive useful feedback" (d = 0.51), and "resident evaluation procedures are fair" (d = 0.40). After-intervention respondents perceived the clerkship learning environment to be more mastery-oriented (d = 0.52), less performance approach-oriented (d = 0.63), and less performance avoid-oriented (d = 0.49). There were no statistical differences in the factors "attending evaluation procedures are fair," "evaluations are accurate," "evaluations are biased," or "perception of stereotype threat." Open-ended questions revealed student recommendations to improve clerkship summary narratives, burden of work-based assessment, and in-person feedback. Lessons Learned: After an assessment system change to pass/fail grading with work-based assessments, we observed moderate to large improvements in student perceptions of clerkship grading and the mastery orientation of the learning environment. Our intervention did not improve perceptions around bias in assessment in clerkships. Other medical schools may consider similar interventions to begin to address student concerns with clerkship assessment and promote a more adaptive learning environment.

The effect of gender dyads on the quality of narrative assessments of general surgery trainees

BACKGROUND

Prior studies have shown that gender can influence how learners are assessed and the feedback they receive. We investigated the quality of faculty narrative comments in general surgery trainee evaluation using trainee-assessor gender dyads.

METHODS

Narrative assessments of surgical trainees at the University of British Columbia were collected and rated using the McMaster Narrative Comment Rating Scale (MNCRS). Variables from the MNCRS were inputted into a generalized linear mixed model to explore the impact of gender dyads on the quality of narrative feedback.

RESULTS

2,469 assessments were collected. Women assessors tended to give higher-quality comments (p's < 0.05) than men assessors. Comments from men assessors to women trainees were significantly more positive than comments from men assessors to men trainees (p = 0.02). Men assessors also tended to give women trainees more reinforcing than corrective comments than to men trainees (p < 0.01).

CONCLUSIONS

There are significant differences in the quality of faculty feedback to trainees by gender dyads. A range of solutions to improve and reduce differences in feedback quality are discussed.

Excellence in medical training: developing talent-not sorting it

Many medical schools have reconsidered or eliminated clerkship grades and honor society memberships. National testing organizations announced plans to eliminate numerical scoring for the United States Medical Licensing Examination Step 1 in favor of pass/fail results. These changes have led some faculty to wonder: "How will we recognize and reward excellence?" Excellence in undergraduate medical education has long been defined by high grades, top test scores, honor society memberships, and publication records. However, this model of learner excellence is misaligned with how students learn or what society values. This accolade-driven view of excellence is perpetuated by assessments that are based on gestalt impressions influenced by similarity between evaluators and students, and assessments that are often restricted to a limited number of traditional skill domains. To achieve a new model of learner excellence that values the trainee's achievement, growth, and responsiveness to feedback across multiple domains, we must envision a new model of teacher excellence. Such teachers would have a growth mindset toward assessing competencies and learning new competencies. Actualizing true learner excellence will require teachers to change from evaluators who conduct assessments of learning to coaches who do assessment for learning. Schools will also need to establish policies and structures that foster a culture that supports this change. In this new paradigm, a teacher's core duty is to develop talent rather than sort it.

From Grading to Assessment for Learning: A Qualitative Study of Student Perceptions Surrounding Elimination of Core Clerkship Grades and Enhanced Formative Feedback

Problem: Medical students perceive honors grading during core clerkships as unfair and inequitable, and negatively impacting their learning and wellbeing. Eliminating honors grading, a powerful extrinsic motivator, and emphasizing formative feedback may address these problems and promote intrinsic motivation and learning. However, it is unknown how transitioning from honors to pass/fail grading with enhanced formative feedback in the core clerkship year may affect student learning experiences, wellbeing, and perceptions of the learning environment. Intervention: Core clerkship grading was transitioned from honors/pass/fail to pass/fail at one US medical school. In addition, the requirement for students to obtain formative supervisor feedback was formalized to twice per week. Context: This qualitative study utilized semi-structured interviews to explore the perceptions among core clerkship students of learning and assessment. Interview questions addressed motivation, wellbeing, learning behaviors, team dynamics, feedback, and student and supervisor attitudes regarding assessment changes. The authors analyzed data inductively using thematic analysis informed by sensitizing concepts related to theories of motivation (goal orientation theory and self-determination theory). Impact: Eighteen students participated, including five with experience in both honors-eligible and pass/fail clerkships. The authors identified three major themes in students' descriptions of the change in approach to assessment: student engagement in clerkships, wellbeing, and recognition of learning context. Student engagement subthemes included intrinsic motivation for patient care rather than performing; sense of agency over learning, including ability to set learning priorities, seek and receive feedback, take learning risks, and disagree with supervisors, and collaborative relationships with peers and team members. Positive wellbeing was characterized by low stress, sense of authenticity with team members, prioritized physical health, and attention to personal life. Learning context subthemes included recognition of variability of clerkship contexts with pass/fail grading mitigating fairness and equity concerns, support of the grading change from residents and some attendings, and implications surrounding future stress and residency selection. Lessons Learned: Students perceive a transition from honors grading to pass/fail with increased feedback as supporting their engagement in learning, intrinsic motivation, and wellbeing. Drivers of wellbeing appear to include students' feelings of control, achieved through the ability to seek learning opportunities, teaching, and constructive feedback without the perceived need to focus on impressing others. Ongoing evaluation of the consequences of this shift in assessment is needed.

Determining Grades in the Internal Medicine Clerkship: Results of a National Survey of Clerkship Directors

PURPOSE

Trust in and comparability of assessments are essential in clerkships in undergraduate medical education for many reasons, including ensuring competency in clinical skills and application of knowledge important for the transition to residency and throughout students' careers. The authors examined how assessments are used to determine internal medicine (IM) core clerkship grades across U.S. medical schools.

METHODS

A multisection web-based survey of core IM clerkship directors at 134 U.S. medical schools with membership in the Clerkship Directors in Internal Medicine was conducted in October through November 2018. The survey included a section on assessment practices to characterize current grading scales used, who determines students' final clerkship grades, the nature/type of summative assessments, and how assessments are weighted. Respondents were asked about perceptions of the influence of the National Board of Medical Examiners (NBME) Medicine Subject Examination (MSE) on students' priorities during the clerkship.

RESULTS

The response rate was 82.1% (110/134). There was considerable variability in the summative assessments and their weighting in determining final grades. The NBME MSE (91.8%), clinical performance (90.9%), professionalism (70.9%), and written notes (60.0%) were the most commonly used assessments. Clinical performance assessments and the NBME MSE accounted for the largest percentage of the total grade (on average 52.8% and 23.5%, respectively). Eighty-seven percent of respondents were concerned that students' focus on the NBME MSE performance detracted from patient care learning.

CONCLUSIONS

There was considerable variability in what IM clerkships assessed and how those assessments were translated into grades. The NBME MSE was a major contributor to the final grade despite concerns about the impact on patient care learning. These findings underscore the difficulty in comparing learners across institutions and serve to advance discussions for how to improve accuracy and comparability of grading in the clinical environment.

Medical Education's Wicked Problem: Achieving Equity in Assessment for Medical Learners

Despite a lack of intent to discriminate, physicians educated in U.S. medical schools and residency programs often take actions that systematically disadvantage minority patients. The approach to assessment of learner performance in medical education can similarly disadvantage minority learners. The adoption of holistic admissions strategies to increase the diversity of medical training programs has not been accompanied by increases in diversity in honor societies, selective residency programs, medical specialties, and medical school faculty. These observations prompt justified concerns about structural and interpersonal bias in assessment. This manuscript characterizes equity in assessment as a "wicked problem" with inherent conflicts, uncertainty, dynamic tensions, and susceptibility to contextual influences. The authors review the underlying individual and structural causes of inequity in assessment. Using an organizational model, they propose strategies to achieve equity in assessment and drive institutional and systemic improvement based on clearly articulated principles. This model addresses the culture, systems, and assessment tools necessary to achieve equitable results that reflect stated principles. Three components of equity in assessment that can be measured and evaluated to confirm success include intrinsic equity (selection and design of assessment tools), contextual equity (the learning environment in which assessment occurs), and instrumental equity (uses of assessment data for learner advancement and selection and program evaluation). A research agenda to address these challenges and controversies and demonstrate reduction in bias and discrimination in medical education is presented.