Comparing the Quality of Narrative Comments by Rotation Setting

A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles

Introduction

Competency-based education requires high-quality feedback to guide students' acquisition of competencies. Sound assessment and feedback systems, such as ePortfolios, are needed to facilitate seeking and giving feedback during clinical placements. However, it is unclear whether the written feedback comments in ePortfolios are of high quality and aligned with the current competency focus. Therefore, this study investigates the quality of written feedback comments in ePortfolios of healthcare students, as well as how these feedback comments align with the CanMEDS roles.

Methods

A qualitative textual analysis was conducted. 2,349 written feedback comments retrieved from the ePortfolios of 149 healthcare students (specialist medicine, general practice, occupational therapy, speech therapy and midwifery) were analysed retrospectively using deductive content analysis. Two structured categorisation matrices, one based on four literature-derived feedback quality criteria (performance, judgment, elaboration and improvement) and another one on the seven CanMEDS roles (Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar and Professional), guided the analysis.

Results

The minority of the feedback comments (n = 352; 14.9%) could be considered of high quality because they met all four quality criteria. Most feedback comments were of moderate quality and met only two to three quality criteria. Regarding the CanMEDS roles, the Medical Expert role was most frequently represented in the feedback comments, as opposed to the roles Leader and Health Advocate.

Discussion

The results highlighted that providing high-quality feedback is challenging. To respond to these challenges, it is recommended to set up individual and continuous feedback training.

Predictors of faculty narrative evaluation quality in medical school clerkships

INTRODUCTION

Narrative approaches to assessment provide meaningful and valid representations of trainee performance. Yet, narratives are frequently perceived as vague, nonspecific and low quality. To date, there is little research examining factors associated with narrative evaluation quality, particularly in undergraduate medical education. The purpose of this study was to examine associations of faculty- and student-level characteristics with the quality of faculty member's narrative evaluations of clerkship students.

METHODS

The authors reviewed faculty narrative evaluations of 50 students' clinical performance in their inpatient medicine and neurology clerkships, resulting in 165 and 87 unique evaluations in the respective clerkships. The authors evaluated narrative quality using the Narrative Evaluation Quality Instrument (NEQI). The authors used linear mixed effects modelling to predict total NEQI score. Explanatory covariates included the following: time to evaluation completion, number of weeks spent with student, faculty total weeks on service per year, total faculty years in clinical education, student gender, faculty gender, and an interaction term between student and faculty gender.

RESULTS

Significantly higher narrative evaluation quality was associated with a shorter time to evaluation completion, with NEQI scores decreasing by approximately 0.3 points every 10 days following students' rotations (p = .004). Additionally, women faculty had statistically higher quality narrative evaluations with NEQI scores 1.92 points greater than men faculty (p = .012). All other covariates were not significant.

CONCLUSIONS

The quality of faculty members' narrative evaluations of medical students was associated with time to evaluation completion and faculty gender but not faculty experience in clinical education, faculty weeks on service, or the amount of time spent with students. Findings advance understanding on ways to improve the quality of narrative evaluations which are imperative given assessment models that will increase the volume and reliance on narratives.

The effect of gender dyads on the quality of narrative assessments of general surgery trainees

BACKGROUND

Prior studies have shown that gender can influence how learners are assessed and the feedback they receive. We investigated the quality of faculty narrative comments in general surgery trainee evaluation using trainee-assessor gender dyads.

METHODS

Narrative assessments of surgical trainees at the University of British Columbia were collected and rated using the McMaster Narrative Comment Rating Scale (MNCRS). Variables from the MNCRS were inputted into a generalized linear mixed model to explore the impact of gender dyads on the quality of narrative feedback.

RESULTS

2,469 assessments were collected. Women assessors tended to give higher-quality comments (p's < 0.05) than men assessors. Comments from men assessors to women trainees were significantly more positive than comments from men assessors to men trainees (p = 0.02). Men assessors also tended to give women trainees more reinforcing than corrective comments than to men trainees (p < 0.01).

CONCLUSIONS

There are significant differences in the quality of faculty feedback to trainees by gender dyads. A range of solutions to improve and reduce differences in feedback quality are discussed.