Can a reflective rubric be applied consistently with raters globally? A study across three countries

Mental effort in the assessment of critical reflection: Implications for assessment quality and scoring

INTRODUCTION

Critical reflection is a mainstay in the training of health professionals, yet assessment of reflection is commonly described as difficult, taxing, and resulting in inconsistent scoring across assessors. At the same time, there is evidence from experiential and simulation settings that assessors' mental effort may explain assessor variability, which could be a target for simplifications in assessment design. Assessors' mental effort for assessment of reflection is currently unknown. This study aimed to determine reliability of rubric scoring of critical reflection, variation in pass-fail rates, and the relationship between reflection scores and assessors' perceived mental effort.

METHODS

Eleven assessors were recruited to assess six reflection assignments using a published rubric. Mental effort was measured using the Paas scale for each assignment assessed and was correlated with rubric scores for each assignment.

RESULTS

Findings showed inconsistency in scoring between assessors, resulting in varying pass rates for each assignment (55-100%). All assignments demonstrated negative correlations between rubric scores and perceived mental effort (r = -0.115 to -0.649).

CONCLUSIONS

Findings support the notion that more work should be done to optimize assessment of critical reflection. Future studies should focus on disentangling the influence on mental effort of scoring tools, assignment structures, and writing quality.

Reflective Practice: Co-Creating Reflective Activities for Pharmacy Students

Reflective practice is important in pharmacy education to support skill development for decision-making, critical thinking, problem-solving, and in continuing professional development and beyond. Despite the importance of reflective practice in higher education and professional practice, reflection assignments are not universally embraced by students. This project was initiated due to recent interest in the co-creation of pharmacy curriculum by students and faculty. The purpose of this project was to develop reflection assignments for pharmacy students. The principles of the analysis, design, development, implementation, evaluation (ADDIE) instructional design framework guided the development of reflection assignment templates for three focus areas: personal development, professional development, and professional identity formation. Templates included background and definitions for these specific focus areas as well as objectives, instructions, guiding questions, assessment methods, and submission requirements. A previously tested assessment rubric was adopted for reflection assignments. Development involved target audience and expert reviews and a trial implementation was held in a year 3 patient care skills course. The co-creation process enriched the experiences of students and faculty involved in it. Future co-creation projects including groups of students, formal evaluation of outcomes, and impact on the program will further support integration of reflective practice in the pharmacy curriculum.

Grading reflective essays: the reliability of a newly developed tool- GRE-9

BACKGROUND

The main objective of this study is the development of a short reliable easy-to-use assessment tool in the aim of providing feedback to the reflective writings of medical students and residents.

METHODS

This study took place in a major tertiary academic medical center in Beirut, Lebanon. Seventy-seven reflective essays written by 18 residents in the department of Family Medicine at the American University of Beirut Medical Center (AUBMC) were graded by 3 raters using the newly developed scale to assess the scale reliability. Following a comprehensive search and analysis of the literature, and based on their experience in reflective grading, the authors developed a concise 9-item scale to grade reflective essays through repeated cycles of development and analysis as well as the determination of the inter-rater reliability (IRR) using intra-class correlation coefficients (ICC) and Krippendorff's Alpha.

RESULTS

The inter-rater reliability of the new scale ranges from moderate to substantial with ICC of 0.78, 95% CI 0.64-0.86, p < 0.01 and Krippendorff's Alpha was 0.49.

CONCLUSIONS

The newly developed scale, GRE-9, is a short, concise, easy-to-use reliable grading tool for reflective essays that has demonstrated moderate to substantial inter-rater reliability. This will enable raters to objectively grade reflective essays and provide informed feedback to residents and students.