Making EPAs a 59 minute objective measure for surgical trainees - A pilot study

A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

Response Process Validity Evidence for Video Commentary Assessment in Surgery: A Qualitative Study

OBJECTIVES

Well-developed mental representations of a task are fundamental to proficient performance. 'Video Commentary' (VC) is a novel assessment intended to measure mental representations of surgical tasks that would reflect an important aspect of task proficiency. Whether examinees' actual response processes align with this intent remains unknown. As part of ongoing validation of the assessment, we sought to understand examinees' response processes in VC.

DESIGN

Grounded theory qualitative study. In 2019, residents were interviewed about their understanding of and approach to VC. Using grounded theory, we created a theoretical model explaining relationships among factors that influence residents' response processes and performance. Residents' perceived purpose of VC was also explored using Likert-type questions.

SETTING

Academic surgical residency program.

PARTICIPANTS

Forty-eight surgical residents (PGY-1 to PGY-5).

RESULTS

Analysis of narrative comments indicated that residents' perceived purposes of VC generally align with the educator's intent. Resident response processes are influenced by test characteristics, residents' perception and understanding of VC, and residents' personal characteristics. Four strategies seem to guide how residents respond, namely a focus on speed, points, logic, and relevance. Quantitative results indicated residents believe VC scores reflect their ability to speak quickly, ability to think quickly, and knowledge of anatomy (mean = 5.0, 4.5, and 4.4 respectively [1 = strongly disagree, 6 = strongly agree]). PGY-1 and PGY-2 residents tend to focus on naming facts whereas PGY-4 and PGY-5 residents focus on providing comprehensive descriptions.

CONCLUSIONS

Residents generally have an accurate understanding of the purpose of VC. However, their use of different approaches could represent a threat to validity. The response strategies of speed, points, logic, and relevance may inform other clinical skills assessments.

Conducting Virtual Simulated Skills Multiple Mini-Interviews for General Surgery Residency Interviews

OBJECTIVE

We describe the feasibility of transitioning simulated skills assessments during general surgery interviews from an in-person to virtual format.

DESIGN/SETTING

Technical and nontechnical skill multiple mini-interviews (MMIs) were performed virtually and assessed for 109 applicants during virtual general surgery interviews over 2 days at a tertiary academic medical center.

RESULTS

We demonstrate the feasibility of virtually assessing general surgery residency applicants' technical and non-technical skills. Using a virtual MMI format during general surgery interviews, we assessed communication, emotional intelligence, anatomical knowledge, interpretation of medical tests, knot tying, and suturing. Four tasks (communication, emotional intelligence , anatomical knowledge, and interpretation of tests) were assessed synchronously by trained general surgery interns. Applicants submitted a recording of themselves performing knot tying and suturing tasks, which were asynchronously assessed after the interview day. Applicants rated the MMI experience highly (4.3/5) via postinterview day survey and the majority of applicants felt that station objectives were met in the virtual format.

CONCLUSIONS

We report a successful experience implementing technical and nontechnical virtual MMIs with capacity for 120 applicants during general surgery residency interviews. In the midst of a COVID-19 pandemic, the ability to assess surgical leaners virtually is essential. Virtual skills assessments may provide a more comprehensive picture of applicants and enable residency programs to better assess residents when gathering in person is not feasible.

Video Commentary: Rapid-Fire Assessment of Surgical Cognitive Skills

OBJECTIVE

We describe the use of "Video Commentary", an assessment using a set of operative videos, to assess trainees' surgical cognitive skills (operative knowledge, spatial awareness, and surgical insight).

DESIGN & SETTING

The Video Commentary assessment has been routinely administered to Postgraduate Year (PGY) 1-5 general surgery residents since 2014 as part of a biannual multistation, OSCE-type exam at Mayo Clinic, Rochester, MN. Video Commentary is a rapid-fire, 6-minute assessment, where trainees watch a series of 20 to 30 second operative video clips and comment on them as they play. Each clip varies in procedure, approach, difficulty, and complexity. The combination of video clips differs according to trainees' PGY level except for a few videos that overlap among PGY groups. The name of the procedure is provided at the beginning of each clip with a countdown timer showing in the corner of the screen. A comprehensive checklist is used to score trainees' performance in real-time.

DISCUSSION

Assessment of trainees of different levels and staff surgeons show a positive correlation with the experience level (p = 0.0001). The assessment provides a safe alternative to assess trainees in the operating room and encourages them to become more effective communicators. With the use of technology, large video databases can be created to provide just-in-time tailored feedback to the trainees.

CONCLUSIONS

Video Commentary can serve as a time and resource-efficient assessment of trainees' surgical cognitive skills and insight. The use and demand of real-time commentary on operative videos may provide a viable approach to help surgeon educators determine trainees' baseline, progression, and readiness to advance.