Video-analysis for the assessment of practical skill

Assessing operative competence in core surgical training: A reliability analysis

BACKGROUND

This study quantifies the number of observations required to reliably assess the operative competence of Core Surgical Trainees (CSTs) in Ireland, using the Supervised Structured Assessment of Operative Performance (SSAOP) tool.

METHODS

SSAOPs (April 2016-February 2021) were analysed across a mix of undifferentiated procedures, as well as for three commonly performed general surgery procedures in CST: appendicectomy, abdominal wall hernia repair, and skin/subcutaneous lesion excision. Generalizability and Decision studies determined the number of observations required to achieve dependability indices ≥0.8, appropriate for use in high-stakes assessment.

RESULTS

A total of 2,294 SSAOPs were analysed. Four assessors, each observing 10 cases, can generate scores sufficiently reliable for use in high-stakes assessments. Focusing on a selection of core procedures yields more favourable reliability indices.

CONCLUSION

Trainers should conduct repeated assessments across a smaller number of procedures to improve reliability. Programs should increase the assessor mix to yield sufficient dependability indices for high-stakes assessment.

Electroencephalography can provide advance warning of technical errors during laparoscopic surgery

BACKGROUND

Intraoperative adverse events lead to patient injury and death, and are increasing. Early warning systems (EWSs) have been used to detect patient deterioration and save lives. However, few studies have used EWSs to monitor surgical performance and caution about imminent technical errors. Previous (non-surgical) research has investigated neural activity to predict future motor errors using electroencephalography (EEG). The present proof-of-concept cohort study investigates whether EEG could predict technical errors in surgery.

METHODS

In a large academic hospital, three surgical fellows performed 12 elective laparoscopic general surgeries. Audiovisual data of the operating room and the surgeon's neural activity were recorded. Technical errors and epochs of good surgical performance were coded into events. Neural activity was observed 40 s prior and 10 s after errors and good events to determine how far in advance errors were detected. A hierarchical regression model was used to account for possible clustering within surgeons. This prospective, proof-of-concept, cohort study was conducted from July to November 2021, with a pilot period from February to March 2020 used to optimize the technique of data capture and included participants who were blinded from study hypotheses.

RESULTS

Forty-five technical errors, mainly due to too little force or distance (n = 39), and 27 good surgical events were coded during grasping and dissection. Neural activity representing error monitoring (p = .008) and motor uncertainty (p = .034) was detected 17 s prior to errors, but not prior to good surgical performance.

CONCLUSIONS

These results show that distinct neural signatures are predictive of technical error in laparoscopic surgery. If replicated with low false-alarm rates, an EEG-based EWS of technical errors could be used to improve individualized surgical training by flagging imminent unsafe actions-before errors occur and cause patient harm.

Development and Validation of Task-Specific Metrics for the Assessment of Linear Stapler-Based Small Bowel Anastomosis

INTRODUCTION

Task-specific metrics facilitate the assessment of surgeon performance. This 3-phased study was designed to (1) develop task-specific metrics for stapled small bowel anastomosis, (2) obtain expert consensus on the appropriateness of the developed metrics, and (3) establish its discriminant validity.

METHODS

In Phase I, a hierarchical task analysis was used to develop the metrics. In Phase II, a survey of expert colorectal surgeons established the importance of the developed metrics. In Phase III, to establish discriminant validity, surgical trainees and surgeons, divided into novice and experienced groups, constructed a side-to-side anastomosis on porcine small bowel using a linear cutting stapler. The participants' performances were videotaped and rated by 2 independent observers. Partial least squares regression was used to compute the weights for the task-specific metrics to obtain weighted total score.

RESULTS

In Phase II, a total of 45 colorectal surgeons were surveyed: 28 with more than 15 years, 13 with 5 to 15 years, and 4 with less than 5 years of experience. The consensus was obtained on all the task-specific metrics in the more experienced groups. In Phase III, 20 subjects participated equally in both groups. The experienced group performed better than the novice group regardless of the rating scale used: global rating scale (p = 0.009) and the task-specific metrics (p = 0.012). After partial least squares regression, the weighted task-specific metric score continued to show that the experienced group performed better (p < 0.001).

CONCLUSION

Task-specific metric items were developed based on expert consensus and showed good discriminant validity compared with a global rating scale between experienced and novice operators. These items can be used for evaluating technical skills in a stapled small bowel anastomosis model.

Microsurgical Anastomosis Rating Scale (MARS10): A Final Product Scoring System for Initial Microsurgical Training

Background

High cost and ethical controversy of using living models in microsurgical training made non-living models more popular. However, non-living models don't provide appropriate feedback of microsurgical performance. Currently existing Global Rating Scales used for advanced microsurgical skills validation are difficult to apply on non-living model. This study presents a simple instrument for basic assessment of microsurgical anastomosis on non-living model.

Methods

Seventy medical students were divided into 2 groups depending on their prior microsurgical experience. Each participant performed 3 end-to-end anastomoses on chicken femoral artery model. Anastomoses were reviewed by 3 blinded experts and then photographed. Evaluation included a patency tests, longitudinal cut of anastomosis, and the newly proposed tool 10 Point Microsurgical Anastomosis Rating Scale (MARS10). Presented scale consists of 5 factors important for anastomosis closure (anastomosis closure, suture spacing, bites size, knot tying, and cut ends length), graded on 3 point scale (0-2 points). Results were analyzed with analysis of variance, Spearman correlation, and t Student test.

Results

Anastomoses evaluated by experts as patent significantly correlated with a high summary score in MARS10 scale (r = 0.73 P < .0001). There was a significant difference in MARS10 score between groups (P < .0001). There were no significant inter-rater differences in scoring among all 3 evaluators (p > .05).

Conclusions

10 Point Microsurgical Anastomosis Rating Scale is a quick, valid, and reliable tool to assess microsurgical end-to-end arterial anastomoses on non-living model.

Video-Based Coaching: Current Status and Role in Surgical Practice (Part 1) From the Society for Surgery of the Alimentary Tract, Health Care Quality and Outcomes Committee

Patient safety and outcomes are directly related to surgical performance. Surgical training emphasizes the importance of the surgeon in determining these outcomes. After training is complete, there is a lack of structured programs for surgeons to audit their skills and continue their individual development. There is a significant linear relationship between surgeon technical skill and surgical outcomes; however, measuring technical performance is difficult. Video-based coaching matches an individual surgeon in practice with a surgical colleague who has been trained in the core principles of coaching for individualizing instruction. It can provide objective assessment for teaching higher-level concepts, such as technical skills, cognitive skills, and decision-making. There are many benefits to video-based coaching. While the concept is gaining acceptance as a method of surgical education, it is still novel in clinical practice. As more surgeons look towards video-based coaching for quality improvement, a consistent definition of the program, goals, and metrics for assessment will be critical. This paper is a review on the status of the video-based coaching as it applies to practicing surgeons.

What lessons can surgeons learn from sport? The reflections of a retired athlete

Surgery often looks to other domains of high performance such as the airline industry for ideas on how to improve surgical performance however little is written about what surgeons might learn from high performance sport. In this paper I offer some observations and ideas from my experience as an Olympic swimmer which I feel may be applicable to surgery and some thoughts on how these concepts might be introduced.

Expert Consensus Recommendations for Robotic Surgery Credentialing

OBJECTIVE

To define criteria for robotic credentialing using expert consensus.

BACKGROUND

A recent review of institutional robotic credentialing policies identified significant variability and determined current policies are largely inadequate to ensure surgeon proficiency and may threaten patient safety.

METHODS

28 national robotic surgery experts were invited to participate in a consensus conference. After review of available institutional policies and discussion, the group developed a 91 proposed criteria. Using a modified Delphi process the experts were asked to indicate their agreement with the proposed criteria in three electronic survey rounds after the conference. Criteria that achieved 80% or more in agreement (consensus) in all rounds were included in the final list.

RESULTS

All experts agreed that there is a need for standardized robotic surgery credentialing criteria across institutions that promote surgeon proficiency. 49 items reached consensus in the first round, 19 in the second, and 8 in the third for a total of 76 final items. Experts agreed that privileges should be granted based on video review of surgical performance and attainment of clearly defined objective proficiency benchmarks. Parameters for ongoing outcome monitoring were determined and recommendations for technical skills training, proctoring, and performance assessment were defined.

CONCLUSIONS

Using a systematic approach, detailed credentialing criteria for robotic surgery were defined. Implementation of these criteria uniformly across institutions will promote proficiency of robotic surgeons and has the potential to positively impact patient outcomes.

What can surgery learn from other high-performance disciplines?

High-performance disciplines have always been concerned with safety and exceptional performance. They have established a culture of vigilance and accepted that human error is both inevitable and ubiquitous. These disciplines, therefore, have all implemented a 'systems approach' to error by focusing on predicting, preventing, rescuing and reporting errors that occur so that they can constantly adapt and improve. Given the complexity of surgery, and the error-prone environment within which it takes place, extracting positive behaviours from other high-performance disciplines will serve to improve performance and enhance patient safety. Surgery is being practiced in an ever-changing environment. Currently, there is less available operative experience for surgical trainees; multi-morbidity in patients is growing and rapidly evolving technology means that more high-tech equipment is being used in procedures. This article evaluates the effectiveness of current surgical protocol in reducing errors and possible modifications that can be made to fit the new environment that surgery is now being practiced in. It will then describe how three different high-performance disciplines: aviation, professional sport and Formula 1, have developed in their approaches to safety and excellence, which will serve as the basis for a discussion about what more can be learnt from these disciplines so that the surgical profession can continue to excel in the face of change.