An American Board of Surgery Pilot of Video Assessment of Surgeon Technical Performance in Surgery

Feasibility of Wearable Sensors to Assess Cognitive Load During Clinical Performance: Lessons Learned and Blueprint for Success

INTRODUCTION

Cognitive load (CogL) is increasingly recognized as an important resource underlying operative performance. Current innovations in surgery aim to develop objective performance metrics via physiological monitoring from wearable digital sensors. Surgeons have access to consumer technology that could measure CogL but need guidance regarding device selection and implementation. To realize the benefits of surgical performance improvement these methods must be feasible, incorporating human factors usability and design principles. This paper aims to evaluate the feasibility of using wearable sensors to assess CogL, identify the benefits and challenges of implementing devices, and develop guidance for surgeons planning to implement wearable devices in their research or practice.

METHODS

We examined the feasibility of wearable sensors from a series of empirical studies that measured aspects of clinical performance relating to CogL. Across four studies, 84 participants and five sensors were involved in the following clinical settings: (i) real intraoperative surgery; (ii) simulated laparoscopic surgery; and (iii) medical team performance outside the hospital.

RESULTS

Wearable devices worn on the wrist and chest were found to be comfortable. After a learning curve, electrodermal activity data were easily and reliably collected. Devices using photoplethysmography to determine heart rate variability were significantly limited by movement artifact. There was variable success with electroencephalography devices regarding connectivity, comfort, and usability.

CONCLUSIONS

It is feasible to use wearable sensors across various clinical settings, including surgery. There are some limitations, and their implementation is context and device dependent. To scale sensor use in clinical research, surgeons must embrace human factors principles to optimize wearability, usability, reliability, and data security.

Correcting for Rater Effects in Operating Room Surgical Skills Assessment

OBJECTIVE

To estimate and adjust for rater effects in operating room surgical skills assessment performed using a structured rating scale for nasal septoplasty.

METHODS

We analyzed survey responses from attending surgeons (raters) who supervised residents and fellows (trainees) performing nasal septoplasty in a prospective cohort study. We fit a structural equation model with the rubric item scores regressed on a latent component of skill and then fit a second model including the rating surgeon as a random effect to model a rater-effects-adjusted latent surgical skill. We validated this model against conventional measures including the level of expertise and post-graduation year (PGY) commensurate with the trainee's performance, the actual PGY of the trainee, and whether the surgical goals were achieved.

RESULTS

Our dataset included 188 assessments by 7 raters and 41 trainees. The model with one latent construct for surgical skill and the rater as a random effect was the best. Rubric scores depended on how severe or lenient the rater was, sometimes almost as much as they depended on trainee skill. Rater-adjusted latent skill scores increased with attending-estimated skill levels and PGY of trainees, increased with the actual PGY, and appeared constant over different levels of achievement of surgical goals.

CONCLUSION

Our work provides a method to obtain rater effect adjusted surgical skill assessments in the operating room using structured rating scales. Our method allows for the creation of standardized (i.e., rater-effects-adjusted) quantitative surgical skill benchmarks using national-level databases on trainee assessments.

LEVEL OF EVIDENCE

N/A Laryngoscope, 134:3548-3554, 2024.

Putting the coach in the game: Current and future state of surgical coaching in pediatric urology

BACKGROUND

Surgical coaching has been proposed as a mechanism to fill gaps in proficiency and encourage continued growth following formal surgical training. Coaching benefits have been demonstrated in other surgical fields; however, have not been evaluated within pediatric urology. The aims of this study were to survey members of The Societies for Pediatric Urology (SPU) to assess the current understanding and utilization of surgical coaching while gauging interest, potential barriers and personal goals for participation in a coaching program.

METHODS

Following IRB approval, members of the SPU were invited to electronically complete an anonymous survey which assessed 4 domains: 1) understanding of surgical coaching principles, 2) current utilization, 3) interest and potential barriers to participation, and 4) personal surgical goals. To evaluate understanding, questions with predefined correct answers on the key principles of coaching were posed either in multiple choice or True/False format to the SPU membership.

RESULTS

Of the 674 pediatric urologists invited, 146 completed the survey (22%). Of those, 46% correctly responded the definition of surgical coaching. Coaching utilization was reported in 27% of respondents currently or having previously participated in a surgical coaching program. Despite current participation rates, only 6 surgeons (4%) have completed training in surgical coaching, despite 79% expressing interest to participate in a surgical coaching program. The most influential barrier to participating in a coaching program was time commitment. Respondents largely prioritized technical and cognitive skill improvement as their primary goals for coaching (see figure below).

CONCLUSIONS

While interest in surgical coaching is high among pediatric urologists, the principles of surgical coaching were not universally understood. Furthermore, formal coach training is markedly deficient, representing a gap in our profession and an opportunity for significant avenues for improvement, especially for technical and cognitive skills. Development of a coaching model based on these results would best suit the needs of pediatric urologists providing that the time commitment barrier for these endeavors can be mitigated and/or reconciled.

Effect of video-based self-reflection on intraoperative skills: A pilot randomized controlled trial

BACKGROUND

The value of video-based self-assessment in enhancing surgical skills is uncertain. This study investigates the feasibility and estimates sample size for a full-scale randomized controlled trial to evaluate the effectiveness of video-based self-assessment to improve surgical performance of laparoscopic cholecystectomy in trainees.

METHODS

This parallel pilot randomized controlled trial included general surgery trainees performing supervised laparoscopic cholecystectomy randomized 1:1 to control (traditional intraoperative teaching) or intervention group (traditional teaching plus video-based self-assessment). Operative performance was measured by the attending surgeon blinded to group assignment at the time of surgery using standardized assessment tools (Global Operative Assessment of Laparoscopic Skills and Operative Performance Rating System). The intervention group had access to their video recordings on a web-based platform for review and self-assessment using the same instruments. The primary outcome for the estimation of sample size was the difference in faculty-assessed final operative performance (third submitted case). Feasibility criteria included >85% participation, >85% adherence to case submission and >85% completion of self-assessment.

RESULTS

Of 37 eligible trainees approached, 32 consented and were randomized (86%). There were 16 in the intervention group, 15 in the control group (55% male, 55% junior trainees), and 1 was excluded for protocol violation. Twenty-four (75%) of participants submitted 3 cases. Thirteen trainees (81%) accessed the platform and completed 26 (63.2%) case self-assessments. Fifty-five trainees per arm will be needed to power a full-scale laparoscopic cholecystectomy with Global Operative Assessment of Laparoscopic Skills and 130 trainees per arm with Operative Performance Rating System as the assessment tool.

CONCLUSION

This pilot study contributes important data to inform the design of an adequately powered randomized controlled trial of video-based self-assessment to improve trainee performance of laparoscopic cholecystectomy. Although a priori trial feasibility criteria were not achieved, automated video capture and storage could significantly improve adherence in future trials.

A systematic review of video-based educational interventions in robotic surgical training

As robotic surgical procedures become more prevalent in practice, there is a demand for effective and efficient educational strategies in robotic surgery. Video has been used in open and laparoscopic surgery to instruct trainees in the acquisition of operative knowledge and surgical skill. Robotic surgery is an ideal application of video-based technology given the access of video recording directly from the console. This review will present the evidence base for video-based educational tools in robotic surgery to guide the development of future educational interventions using this technology. A systematic review of the literature was performed using the key words "video" "robotic surgery" and "education". From a total of 538 results, 15 full text articles were screened. Inclusion criteria were the presentation of an educational intervention using video and the application of this intervention to robotic surgery. The results of 10 publications are presented in this review. Analysis of the key concepts presented in these publications revealed three themes: video as technology, video as instruction, video as feedback. All studies showed a video-based learning had a positive effect on educational outcomes. There are limited published studies looking specifically at the use of video as an educational intervention in robotic surgical training. Existing studies primarily focus on the use of video as a review tool for skill development. There is scope to expand the use of robotic video as a teaching tool through adaptation of novel technology such as 3D headsets and concepts of cognitive simulation including guided mental imagery and verbalisation.