Classification of Intermediate and Novice Surgeons' Skill Assessment Through Performance Metrics

A systematic review on classification and assessment of surgical skill levels for simulation-based training programs

BACKGROUND

Nowadays, advances in medical informatics have made minimally invasive surgery (MIS) procedures the preferred choice. However, there are several problems with the education programs in terms of surgical skill acquisition. For instance, defining and objectively measuring surgical skill levels is a challenging process. Accordingly, the aim of this study is to conduct a literature review for an investigation of the current approaches for classifying the surgical skill levels and for identifying the skill training tools and measurement methods.

MATERIALS AND METHODS

In this research, a search is conducted and a corpus is created. Exclusion and inclusion criteria are applied by limiting the number of articles based on surgical education, training approximations, hand movements, and endoscopic or laparoscopic operations. To satisfy these criteria, 57 articles are included in the corpus of this study.

RESULTS

Currently used surgical skill assessment approaches have been summarized. Results show that various classification approaches for the surgical skill level definitions are being used. Besides, many studies are conducted by omitting particularly important skill levels in between. Additionally, some inconsistencies are also identified across the skill level classification studies.

CONCLUSION

In order to improve the benefits of simulation-based training programs, a standardized interdisciplinary approach should be developed. For this reason, specific to each surgical procedure, the required skills should be identified. Additionally, appropriate measures for assessing these skills, which can be defined in simulation-based MIS training environments, should be refined. Finally, the skill levels gained during the developmental stages of these skills, with their threshold values referencing the identified measures, should be redefined in a standardized manner.

Machine learning for technical skill assessment in surgery: a systematic review

Accurate and objective performance assessment is essential for both trainees and certified surgeons. However, existing methods can be time consuming, labor intensive, and subject to bias. Machine learning (ML) has the potential to provide rapid, automated, and reproducible feedback without the need for expert reviewers. We aimed to systematically review the literature and determine the ML techniques used for technical surgical skill assessment and identify challenges and barriers in the field. A systematic literature search, in accordance with the PRISMA statement, was performed to identify studies detailing the use of ML for technical skill assessment in surgery. Of the 1896 studies that were retrieved, 66 studies were included. The most common ML methods used were Hidden Markov Models (HMM, 14/66), Support Vector Machines (SVM, 17/66), and Artificial Neural Networks (ANN, 17/66). 40/66 studies used kinematic data, 19/66 used video or image data, and 7/66 used both. Studies assessed the performance of benchtop tasks (48/66), simulator tasks (10/66), and real-life surgery (8/66). Accuracy rates of over 80% were achieved, although tasks and participants varied between studies. Barriers to progress in the field included a focus on basic tasks, lack of standardization between studies, and lack of datasets. ML has the potential to produce accurate and objective surgical skill assessment through the use of methods including HMM, SVM, and ANN. Future ML-based assessment tools should move beyond the assessment of basic tasks and towards real-life surgery and provide interpretable feedback with clinical value for the surgeon.

PROSPERO: CRD42020226071

Distinguishing Intermediate and Novice Surgeons by Eye Movements

Surgical skill-level assessment is key to collecting the required feedback and adapting the educational programs accordingly. Currently, these assessments for the minimal invasive surgery programs are primarily based on subjective methods, and there is no consensus on skill level classifications. One of the most detailed of these classifications categorize skill levels as beginner, novice, intermediate, sub-expert, and expert. To properly integrate skill assessment into minimal invasive surgical education programs and provide skill-based training alternatives, it is necessary to classify the skill levels in as detailed a way as possible and identify the differences between all skill levels in an objective manner. Yet, despite the existence of very encouraging results in the literature, most of the studies have been conducted to better understand the differences between novice and expert surgical skill levels leaving out the other crucial skill levels between them. Additionally, there are very limited studies by considering the eye-movement behaviors of surgical residents. To this end, the present study attempted to distinguish novice- and intermediate-level surgical residents based on their eye movements. The eye-movement data was recorded from 23 volunteer surgical residents while they were performing four computer-based simulated surgical tasks under different hand conditions. The data was analyzed using logistic regression to estimate the skill levels of both groups. The best results of the estimation revealing a 91.3% recognition rate of predicting novice and intermediate surgical residents on one scenario were selected from four under the dominant hand condition. These results show that the eye-movements can be potentially used to identify surgeons with intermediate and novice skills. However, the results also indicate that the order in which the scenarios are provided, and the design of the scenario, the tasks, and their appropriateness with the skill levels of the participants are all critical factors to be considered in improving the estimation ratio, and hence require thorough assessment for future research.