Yeh VJH, Mukhtar F, Yudkowsky R, Baloul MS, Farley DR, Cook DA. Response Process Validity Evidence for Video Commentary Assessment in Surgery: A Qualitative Study.
JOURNAL OF SURGICAL EDUCATION 2022;
79:1270-1281. [PMID:
35688704 DOI:
10.1016/j.jsurg.2022.05.006]
[Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/30/2021] [Revised: 04/09/2022] [Accepted: 05/08/2022] [Indexed: 06/15/2023]
Abstract
OBJECTIVES
Well-developed mental representations of a task are fundamental to proficient performance. 'Video Commentary' (VC) is a novel assessment intended to measure mental representations of surgical tasks that would reflect an important aspect of task proficiency. Whether examinees' actual response processes align with this intent remains unknown. As part of ongoing validation of the assessment, we sought to understand examinees' response processes in VC.
DESIGN
Grounded theory qualitative study. In 2019, residents were interviewed about their understanding of and approach to VC. Using grounded theory, we created a theoretical model explaining relationships among factors that influence residents' response processes and performance. Residents' perceived purpose of VC was also explored using Likert-type questions.
SETTING
Academic surgical residency program.
PARTICIPANTS
Forty-eight surgical residents (PGY-1 to PGY-5).
RESULTS
Analysis of narrative comments indicated that residents' perceived purposes of VC generally align with the educator's intent. Resident response processes are influenced by test characteristics, residents' perception and understanding of VC, and residents' personal characteristics. Four strategies seem to guide how residents respond, namely a focus on speed, points, logic, and relevance. Quantitative results indicated residents believe VC scores reflect their ability to speak quickly, ability to think quickly, and knowledge of anatomy (mean = 5.0, 4.5, and 4.4 respectively [1 = strongly disagree, 6 = strongly agree]). PGY-1 and PGY-2 residents tend to focus on naming facts whereas PGY-4 and PGY-5 residents focus on providing comprehensive descriptions.
CONCLUSIONS
Residents generally have an accurate understanding of the purpose of VC. However, their use of different approaches could represent a threat to validity. The response strategies of speed, points, logic, and relevance may inform other clinical skills assessments.
Collapse