"My Assessments Are Biased!" Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education

Recommendations to address and research systemic bias in assessment: perspectives from directors of research in medical education

INTRODUCTION

Addressing systemic bias in medical school assessment is an urgent task for medical education. This paper outlines recommendations on topic areas for further research on systemic bias, developed from a workshop discussion at the 2023 annual meeting of the Society of Directors of Research in Medical Education.

MATERIALS AND METHODS

During the workshop, directors engaged in small-group discussions on guidelines to address bias in assessment practices following a proposed categorization of 'Do's,' 'Don'ts,' and 'Don't knows' and listed their insights using anonymous sticky notes, which were shared and discussed with the larger group of participants. The authors performed a content analysis of the notes through deductive and inductive coding. We reviewed and discussed our analysis to reach consensus.

RESULTS

The workshop included 31 participants from 28 institutions across the US and Canada, generating 51 unique notes. Participants identified 23 research areas in need of further study. The inductive analysis of proposed research areas revealed four main topics: 1) The role of interventions, including pre-medical academic interventions, medical-education interventions, assessment approaches, and wellness interventions; 2) Professional development, including the definition and assessment of professionalism and professional identity formation; 3) Context, including patient care and systemic influences; and 4) Research approaches.

DISCUSSION

While limited to data from a single workshop, the results offered perspectives about areas for further research shared by a group of directors of medical education research units from diverse backgrounds. The workshop produced valuable insights into the need for more evidence-based interventions that promote more equitable assessment practices grounded in real-world situations and that attenuate the effects of bias.

Assessing supervisor versus trainee viewpoints of entrustment through cognitive and affective lenses: an artificial intelligence investigation of bias in feedback

The entrustment framework redirects assessment from considering only trainees' competence to decision-making about their readiness to perform clinical tasks independently. Since trainees and supervisors both contribute to entrustment decisions, we examined the cognitive and affective factors that underly their negotiation of trust, and whether trainee demographic characteristics may bias them. Using a document analysis approach, we adapted large language models (LLMs) to examine feedback dialogs (N = 24,187, each with an associated entrustment rating) between medical student trainees and their clinical supervisors. We compared how trainees and supervisors differentially documented feedback dialogs about similar tasks by identifying qualitative themes and quantitatively assessing their correlation with entrustment ratings. Supervisors' themes predominantly reflected skills related to patient presentations, while trainees' themes were broader-including clinical performance and personal qualities. To examine affect, we trained an LLM to measure feedback sentiment. On average, trainees used more negative language (5.3% lower probability of positive sentiment, p < 0.05) compared to supervisors, while documenting higher entrustment ratings (+ 0.08 on a 1-4 scale, p < 0.05). We also found biases tied to demographic characteristics: trainees' documentation reflected more positive sentiment in the case of male trainees (+ 1.3%, p < 0.05) and of trainees underrepresented in medicine (UIM) (+ 1.3%, p < 0.05). Entrustment ratings did not appear to reflect these biases, neither when documented by trainee nor supervisor. As such, bias appeared to influence the emotive language trainees used to document entrustment more than the degree of entrustment they experienced. Mitigating these biases is nonetheless important because they may affect trainees' assimilation into their roles and formation of trusting relationships.

Advancing Equity in Assessment in the Surgical Learning Environment

IMPORTANCE

Assessment is a fundamental part of teaching and learning that provides the basis for making inferences about the development of learners. Inequity in assessment disproportionately impacts underrepresented in medicine learners and can limit their opportunities for achievement.

OBSERVATIONS

Unfortunately, inequity in assessment is prevalent in the surgical learning environment due to systemic and individual factors. The Antideficit Achievement framework can be effectively applied to address sources of inequity.

CONCLUSIONS AND RELEVANCE

This paper explores sources of inequity in assessment in the surgical learning environment and illuminates them with a descriptive case study. Recommendations created using the Antideficit Achievement framework provide effective, practical ways to begin to advance equity in assessment in the surgical learning environment.

"I have established this support network": How Chosen Kin Support Women Medical Students During their First Two Years in Medical School

Women medical students experience unique stressors and challenges during medical school related to inherent structural androcentric norms. Through a longitudinal qualitative study of 17 women medical students in their first two years of medical school, we sought to investigate how they navigated their medical school experience. We used a critical lens and narrative inquiry to understand their experiences within the powerful and marginalizing culture of medical school. Our participants identified two essential support groups: those relationships made within, and those sustained outside, medical school. These findings invoked a kinship framework-one where women medical students have a network of chosen kin who provide essential support for them during their first 2 years. The participants' chosen kin within medical school provided support through recognition of one another, belonging by not belonging, being encouraged to reach out, and creating long-term relationships. The chosen kin outside medical school provided support by reminding the student who they are and creating stability. Integrating models of kinship into medical school as practiced by women medical students may have immense value in providing essential supports for medical students, preventing burnout, and changing the culture of care for future physicians that would align recognition and practice of self-care with patient care.

Exploring Unprofessional Behaviors and Biased Perceptions in the Clinical Environment: Students' Perspectives

Professionalism stands as a fundamental cornerstone within the realm of physician training, representing a core competency that holds significant importance. It entails creating workplaces that are physically and psychologically safe within the health care system. Positive role modeling from health professionals is important in creating a learning environment that fosters diversity, equity, and inclusion for all. Using the mixed-methods sequential design, this study investigated unprofessional behaviors and biased perceptions demonstrated by health professionals as perceived and experienced by medical students during their clinical rotations at one medical school. Seventy-three M3 students and 37 M4 students responded to the survey, followed by three focus groups (n = 11) to further examine unprofessionalism and biases as well as impacts on performance, learning opportunities, and well-being. The results from both the quantitative and qualitative data demonstrate the presence of unprofessionalism and biases within the current medical education environment. These issues include a lack of respect and compassion, a lack of commitment to professional duties, malfunctioning teamwork, and a lack of sensitivity towards individuals regardless of their group affiliations. The unprofessional behaviors and biased perceptions have detrimental impacts on students' performance, learning, and well-being. The reasons behind unprofessionalism and bias are multifaceted, influenced by societal and local environmental factors that extend beyond individual beliefs and values. By collecting empirical data on students' experiences and perceptions, the study sheds light on the areas that need improvement and offers insights into adopting strategies to decrease unprofessional conducts and foster a respectful and inclusive learning environment.

Inclusive assessment in health professions education: Balancing global goals and local contexts

CONTEXT

In this article, we draw upon diverse and contextually different experiences of working on inclusive assessment, with the aim of bridging and enhancing practices of inclusive assessments for health professions education (HPE) within universities. Instead of juxtaposing our views from three countries, we combine our perspectives to advocate for inclusive assessment.

DISCUSSION

Creating an inclusive assessment culture is important for equitable education, even if priorities for inclusion might differ between contexts. We recognise challenges in the enactment of inclusive assessment, namely, the notion of lowering standards, harming reliability and robustness of assessment design and inclusion as a poorly defined and catchall term. Importantly, the lack of awareness that inclusion means recognising intersectionality is a barrier for well-designed inclusive assessments. This is why we offer considerations for HPE practitioners that can guide towards a unified direction of travel for inclusive assessments. This article highlights the importance of contextual prioritisation and initiatives to be considered at the global level to national, institutional, programme and the individual level. Utilising experience and literature from undergraduate, higher education contexts, we offer considerations with applicability across the assessment continuum.

CONTEXT

In this state of science paper, we were set the challenge of providing cross-cultural viewpoints on inclusive assessment. In this discursive article, we focus on inclusive assessment within undergraduate health professions education whilst looking to the wider higher education literature, since institutional policies and procedures frequently drive assessment decisions and influence the environment in which they occur. We explore our experiences of working in inclusive assessment, with the aim of bridging and enhancing practices of inclusive assessments for HPE. Unlike other articles that juxtapose views, we all come from the perspective of supporting inclusive assessment. We begin with a discussion on what inclusive assessment is and then describe our contexts as a basis for understanding differences and broadening conversations. We work in the United Kingdom, Australia and Malaysia, having undertaken research, facilitated workshops and seminars on inclusive assessment nationally and internationally. We recognise our perspectives will differ as a consequence of our global context, institutional culture, individual characteristics and educational experiences. (Note that individual characteristics are also known as protected characteristics in some countries). Then, we outline challenges and opportunities associated with inclusive assessment, drawing on evidence within our contexts, acknowledging that our understanding of inclusive assessment research is limited to publications in English and currently tilted to publications from the Global North. In the final section, we then offer recommendations for championing inclusion, focussing firstly on assessment designs, and then broader considerations to organise collective action. Our article is unapologetically practical; the deliberate divergence from a theoretical piece is with the intent that anyone who reads this paper might enact even one small change progressing towards more inclusive assessment practices within their context.

Comparing Peer and Faculty Feedback for Asynchronous Laparoscopic Skill Acquisition

OBJECTIVE

Traditionally, expert surgeons have provided surgical trainees with feedback about their simulation performance, including for asynchronous practice. Unfortunately, innumerable time demands may limit experts' ability to provide feedback. It is unknown whether and how peer feedback is an effective mechanism to help residents acquire laparoscopic skill in an asynchronous setting. As such, we aimed to assess the effect of peer feedback on laparoscopic performance and determine how residents perceive giving and receiving peer feedback.

DESIGN

We conducted a convergent mixed methods study. In the quantitative component, we randomized residents to receive feedback on home laparoscopic tasks from peers or faculty. We then held an end-of-curriculum, in-person laparoscopic assessment with members from both groups and compared performance on the in-person assessment between the groups. In the qualitative component, we conducted interviews with resident participants to explore experiences with feedback and performance. Three authors coded and rigorously reviewed interview data using a directed content analysis.

SETTING

We performed this study at a single tertiary academic institution: the University of California, San Francisco.

PARTICIPANTS

We invited 47 junior residents in general surgery, obstetrics-gynecology, and urology to participate, of whom 37 (79%) participated in the home curriculum and 25 (53%) participated in the end-of-curriculum assessment.

RESULTS

Residents in the peer feedback group scored similarly on the final assessment (mean 70.7%; SD 16.1%) as residents in the faculty feedback group (mean 71.8%; SD 11.9%) (p = 0.86). Through qualitative analysis of interviews with 13 residents, we identified key reasons for peer feedback's efficacy: shared mental models, the ability to brainstorm and appreciate new approaches, and a low-stakes learning environment.

CONCLUSIONS

We found that peer and faculty feedback led to similar performance in basic laparoscopy and that residents engaged positively with peer feedback, suggesting that peer feedback can be used when residents learn basic laparoscopy.

Through the Looking Glass: Comparing Hospitalists' and Internal Medicine Residents' Perceptions of Feedback

INTRODUCTION

Feedback is critical for resident growth and is most effective when the relationship between residents and attendings is collaborative, with shared expectations for the purpose, timing, and manner of communication for feedback. Within internal medicine, there is limited work exploring the resident and hospitalist perspectives on whether key elements are included in feedback sessions.

METHODS

We surveyed internal medicine residents and supervising hospitalists at a large urban training program about their perspectives on four components of effective feedback: specificity,timeliness, respectful communication, and actionability.

RESULTS

We received surveys from 130/184 internal medicine residents and 74/129 hospitalists (71% and 57% response rate, respectively). Residents and hospitalists differed in their perspectives about specificity and timeliness: 54% (70/129) of residents reported they did not receive specific feedback while 90% (65/72) of hospitalists reported they delivered specific feedback (p<0.01), and 33% (43/129) of residents compared with 82% (59/72) of hospitalists perceived feedback as timely (p<0.01). Internal medicine residents and hospitalists reported concordant rates of feedback sessions consisting of a two-way conversation (84%, 109/129; 89%, 64/72, respectively, p=0.82) and that communication was delivered in a respectful manner (95%, 122/129; 97%, 70/72, respectively, p=0.57).

CONCLUSIONS

We observed discordance between internal medicine residents and supervising hospitalist perspectives on the inclusion of two critical components of feedback: specificity and timing. The hospitalist cohort reported delivering more components of effective feedback than the resident cohort reported receiving. The etiology of this discordance is likely multifactorial and requires further investigation.

Ten tips for effective use and quality assurance of multiple-choice questions in knowledge-based assessments

Multiple-choice questions (MCQs) are the most popular type of items used in knowledge-based assessments in undergraduate and postgraduate healthcare education. MCQs allow assessment of candidates' knowledge on a broad range of knowledge-based learning outcomes in a single assessment. Single-best-answer (SBA) MCQs are the most versatile and commonly used format. Although writing MCQs may seem straight-forward, producing decent-quality MCQs is challenging and warrants a range of quality checks before an item is deemed suitable for inclusion in an assessment. Like all assessments, MCQ-based examinations must be aligned with the learning outcomes and learning opportunities provided to the students. This paper provides evidence-based guidance on the effective use of MCQs in student assessments, not only to make decisions regarding student progression but also to build an academic environment that promotes assessment as a driver for learning. Practical tips are provided to the readers to produce authentic MCQ items, along with appropriate pre- and post-assessment reviews, the use of standard setting and psychometric evaluation of assessments based on MCQs. Institutions need to develop an academic culture that fosters transparency, openness, equality and inclusivity. In line with contemporary educational principles, teamwork amongst teaching faculty, administrators and students is essential to establish effective learning and assessment practices.

What the hell is water? Changing medical education's ideology through validity

@Midwest_MedPeds and @DrDanSchumacher call for a "What the hell is water?" reckoning with #MedEd using academic excellence as a central selection metric

The Next Era of Assessment: Building a Trustworthy Assessment System

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

WR, Farrell SE, Hundert EM, Dalrymple JL, Goldhamer MEJ. A pilot clinical skills coaching program to reimagine remediation: a cohort study

Background New approaches are needed to improve and destigmatize remediation in undergraduate medical education (UME).  The COVID-19 pandemic magnified the need to support struggling learners to ensure competency and readiness for graduate medical education (GME).  Clinical skills (CS) coaching is an underutilized approach that may mitigate the stigma of remedial learning. Methods A six-month CS coaching pilot was conducted at Harvard Medical School (HMS) as a destigmatized remedial learning environment for clerkship and post-clerkship students identified as 'at risk' based on objective structured clinical examinations (OSCE).  The pilot entailed individual and group coaching with five faculty, direct bedside observation of CS, and standardized patient encounters with video review. Strengths-based coaching principles and appreciative inquiry were emphasized.  Results Twenty-three students participated in the pilot: 14 clerkship students (cohort 1) and 9 post-clerkship students (cohort 2).  All clerkship students (cohort 1) demonstrated sustained improvement in CS across three OSCEs compared to baseline: at pilot close, at 6-months post pilot, and at 21-24 months post-pilot all currently graduating students (10/10, 100%) passed the summative OSCE, an HMS graduation requirement. All post-clerkship students (cohort 2) passed the HMS graduation OSCE (9/9,100%). Feedback survey results included clerkship students (9/14; 64%) and post-clerkship students (7/9; 78%); all respondents unanimously agreed that individual coaching was "impactful to my clinical learning and practice". Faculty and leadership fully supported the pilot as a destigmatized and effective approach to remediation.  Conclusion Remediation has an essential and growing role in medical schools.  CS coaching for remedial learning can reduce stigma, foster a growth mindset, and support sustained progress for 'at risk' early clerkship through final year students. An "implementation template" with suggested tools and timelines can be locally adapted to guide CS coaching for UME remediation. The CS coaching pilot model is feasible and can be generalized to many UME programs.