Teaching and assessing intra-operative consultations in competency-based medical education: development of a workplace-based assessment instrument

"Rater training" re-imagined for work-based assessment in medical education

In this perspective, the authors critically examine "rater training" as it has been conceptualized and used in medical education. By "rater training," they mean the educational events intended to improve rater performance and contributions during assessment events. Historically, rater training programs have focused on modifying faculty behaviours to achieve psychometric ideals (e.g., reliability, inter-rater reliability, accuracy). The authors argue these ideals may now be poorly aligned with contemporary research informing work-based assessment, introducing a compatibility threat, with no clear direction on how to proceed. To address this issue, the authors provide a brief historical review of "rater training" and provide an analysis of the literature examining the effectiveness of rater training programs. They focus mainly on what has served to define effectiveness or improvements. They then draw on philosophical and conceptual shifts in assessment to demonstrate why the function, effectiveness aims, and structure of rater training requires reimagining. These include shifting competencies for assessors, viewing assessment as a complex cognitive task enacted in a social context, evolving views on biases, and reprioritizing which validity evidence should be most sought in medical education. The authors aim to advance the discussion on rater training by challenging implicit incompatibility issues and stimulating ways to overcome them. They propose that "rater training" (a moniker they suggest be reserved for strong psychometric aims) be augmented with "assessor readiness" programs that link to contemporary assessment science and enact the principle of compatibility between that science and ways of engaging with advances in real-world faculty-learner contexts.

From principles to practice: Implementation of entrustable professional activities (EPAs) for surgical pathology residency education in a large academic hospital

Over the past decade, competency-based medical education (CBME) has gained momentum in the United States to develop trainees into independent and confident physicians by the end of their training. Entrustable professional activities (EPAs) are an established methodology for assessing trainee development through an outcomes-driven rather than a time-based model. While EPAs have been utilized as an assessment tool for CBME in Europe and Canada, their validation and implementation in some medical specialties has occurred more recently in the United States. Pediatrics was the first specialty in the US to conduct a large-scale UME-GME pilot. Pathology Residency EPAs were published in 2018; however, implementation in training programs has been slow. We have piloted EPAs in our residency program's surgical pathology rotation and propose a unique set of 4 surgical pathology EPAs to track trainee preparedness for independent practice.

Validity and reliability of the Spanish version of the ACE tool for assessing competencies in evidence-based medicine in medical students

OBJECTIVE

To translate and culturally adapt the tool 'Assessing Competency in evidence-based medicine (EBM)' (ACE) to Spanish and to implement it in a cohort of medical students for the evaluation of the instrument's psychometric properties.

DESIGN

Bilingual translators produced a translation and backtranslation of the original instrument, with interim consensus in each stage with oversight and input by a group of experts. We then performed cognitive interviews to adapt the wording of the tool culturally. Finally, we implemented the final version in a cohort of medical students on a virtual general practice course with EBM modules.

SETTING

Medical School in Buenos Aires, Argentina. Due to restrictions to in-person teaching during the COVID-19 pandemic, we conducted this study in the context of virtual learning.

PARTICIPANTS

We included 125 fourth and fifth-year medical students.

MAIN OUTCOME MEASURES

We measured internal consistency with the Kuder-Richardson coefficient (>0.6 as a threshold for reliability) and construct validity through a Pearson's correlation between the examinations carried out with the translated instrument and the results of the regular examinations of EBM in the same students (expected values of at least 0.3 to 0.7). We also compared the total score of the instrument of fifth-year students to fourth-year students.

RESULTS

As for measurements for internal consistency, the coefficient Kuder-Richardson resulted in a value of 0.268, below our prespecified threshold. For construct validity, the Pearson correlation between the sum of the items and regular examinations was 0.139, also below our prespecified threshold. However, fifth-year students averaged 0.94 points more than fourth-year students (95% CI 0.24 more to 1.65 more).

CONCLUSION

The translated and cross-culturally adapted version of the ACE tool into Spanish had low reliability and validity in an MBE course taught and evaluated in a virtual environment.

TRIAL REGISTRATION

Not applicable.