Objective and automated assessment of surgical technical skills with IoT systems: A systematic literature review

Deep learning-based video-analysis of instrument motion in microvascular anastomosis training

PURPOSE

Attaining sufficient microsurgical skills is paramount for neurosurgical trainees. Kinematic analysis of surgical instruments using video offers the potential for an objective assessment of microsurgical proficiency, thereby enhancing surgical training and patient safety. The purposes of this study were to develop a deep-learning-based automated instrument tip-detection algorithm, and to validate its performance in microvascular anastomosis training.

METHODS

An automated instrument tip-tracking algorithm was developed and trained using YOLOv2, based on clinical microsurgical videos and microvascular anastomosis practice videos. With this model, we measured motion economy (procedural time and path distance) and motion smoothness (normalized jerk index) during the task of suturing artificial blood vessels for end-to-side anastomosis. These parameters were validated using traditional criteria-based rating scales and were compared across surgeons with varying microsurgical experience (novice, intermediate, and expert). The suturing task was deconstructed into four distinct phases, and parameters within each phase were compared between novice and expert surgeons.

RESULTS

The high accuracy of the developed model was indicated by a mean Dice similarity coefficient of 0.87. Deep learning-based parameters (procedural time, path distance, and normalized jerk index) exhibited correlations with traditional criteria-based rating scales and surgeons' years of experience. Experts completed the suturing task faster than novices. The total path distance for the right (dominant) side instrument movement was shorter for experts compared to novices. However, for the left (non-dominant) side, differences between the two groups were observed only in specific phases. The normalized jerk index for both the right and left sides was significantly lower in the expert than in the novice groups, and receiver operating characteristic analysis showed strong discriminative ability.

CONCLUSION

The deep learning-based kinematic analytic approach for surgical instruments proves beneficial in assessing performance in microvascular anastomosis. Moreover, this methodology can be adapted for use in clinical settings.

Tissue Acceleration as a Novel Metric for Surgical Performance During Carotid Endarterectomy

BACKGROUND AND OBJECTIVES

Gentle tissue handling to avoid excessive motion of affected fragile vessels during surgical dissection is essential for both surgeon proficiency and patient safety during carotid endarterectomy (CEA). However, a void remains in the quantification of these aspects during surgery. The video-based measurement of tissue acceleration is presented as a novel metric for the objective assessment of surgical performance. This study aimed to evaluate whether such metrics correlate with both surgeons' skill proficiency and adverse events during CEA.

METHODS

In a retrospective study including 117 patients who underwent CEA, acceleration of the carotid artery was measured during exposure through a video-based analysis. Tissue acceleration values and threshold violation error frequencies were analyzed and compared among the surgeon groups with different surgical experience (3 groups: novice , intermediate , and expert ). Multiple patient-related variables, surgeon groups, and video-based surgical performance parameters were compared between the patients with and without adverse events during CEA.

RESULTS

Eleven patients (9.4%) experienced adverse events after CEA, and the rate of adverse events significantly correlated with the surgeon group. The mean maximum tissue acceleration and number of errors during surgical tasks significantly decreased from novice, to intermediate, to expert surgeons, and stepwise discriminant analysis showed that the combined use of surgical performance factors could accurately discriminate between surgeon groups. The multivariate logistic regression analysis revealed that the number of errors and vulnerable carotid plaques were associated with adverse events.

CONCLUSION

Tissue acceleration profiles can be a novel metric for the objective assessment of surgical performance and the prediction of adverse events during surgery. Thus, this concept can be introduced into futuristic computer-aided surgeries for both surgical education and patient safety.

An explainable machine learning method for assessing surgical skill in liposuction surgery

PURPOSE

Surgical skill assessment has received growing interest in surgery training and quality control due to its essential role in competency assessment and trainee feedback. However, the current assessment methods rarely provide corresponding feedback guidance while giving ability evaluation. We aim to validate an explainable surgical skill assessment method that automatically evaluates the trainee performance of liposuction surgery and provides visual postoperative and real-time feedback.

METHODS

In this study, machine learning using a model-agnostic interpretable method based on stroke segmentation was introduced to objectively evaluate surgical skills. We evaluated the method on liposuction surgery datasets that consisted of motion and force data for classification tasks.

RESULTS

Our classifier achieved optimistic accuracy in clinical and imitation liposuction surgery models, ranging from 89 to 94%. With the help of SHapley Additive exPlanations (SHAP), we deeply explore the potential rules of liposuction operation between surgeons with variant experiences and provide real-time feedback based on the ML model to surgeons with undesirable skills.

CONCLUSION

Our results demonstrate the strong abilities of explainable machine learning methods in objective surgical skill assessment. We believe that the machine learning model based on interpretive methods proposed in this article can improve the evaluation and training of liposuction surgery and provide objective assessment and training guidance for other surgeries.

Continuous monitoring of surgical bimanual expertise using deep neural networks in virtual reality simulation

In procedural-based medicine, the technical ability can be a critical determinant of patient outcomes. Psychomotor performance occurs in real-time, hence a continuous assessment is necessary to provide action-oriented feedback and error avoidance guidance. We outline a deep learning application, the Intelligent Continuous Expertise Monitoring System (ICEMS), to assess surgical bimanual performance at 0.2-s intervals. A long-short term memory network was built using neurosurgeon and student performance in 156 virtually simulated tumor resection tasks. Algorithm predictive ability was tested separately on 144 procedures by scoring the performance of neurosurgical trainees who are at different training stages. The ICEMS successfully differentiated between neurosurgeons, senior trainees, junior trainees, and students. Trainee average performance score correlated with the year of training in neurosurgery. Furthermore, coaching and risk assessment for critical metrics were demonstrated. This work presents a comprehensive technical skill monitoring system with predictive validation throughout surgical residency training, with the ability to detect errors.

A cost-effective IoT learning environment for the training and assessment of surgical technical skills with visual learning analytics

BACKGROUND

Surgeons need to train and certify their technical skills. This is usually done with the intervention of experts who monitor and assess trainees. Nevertheless, this is a time-consuming task that is subject to variations among evaluators. In recent decades, subjectivity has been significantly reduced through 1) the introduction of standard curricula, such as the Fundamentals of Laparoscopic Surgery (FLS) program, which measures students' performance in specific exercises, and 2) rubrics, which are widely accepted in the literature and serve to provide feedback about the overall technical skills of the trainees. Although these two elements reduce subjectivity, they do not, however, eliminate the figure of the expert evaluator, and so the process remains time consuming.

OBJECTIVES

The objective of this work is to automate those parts of the work of the expert evaluator that the technology can measure objectively, using sensors to collect evidence, and visualizations to provide feedback. We designed and developed 1) a cost-effective IoT (Internet of Things) learning environment for the training and assessment of surgical technical skills and 2) visualizations supported by the literature on visual learning analytics (VLA) to provide feedback about the exercises (in real time) and overall performance (at the end of the training) of the trainee.

METHODS

A hybrid approach was followed based on previous research for the design of the sensor based IoT learning environment. Previous studies were used as the basis for getting best practices on the tracking of surgical instruments and on the detection of the force applied to the tissue, with a focus on reducing the costs of data collection. The monitoring of the specific exercises required the design of sensors and collection mechanisms from scratch as there is little existing research on this subject. Moreover, it was necessary to design the overall architecture to collect, process, synchronize and communicate the data coming from the different sensors to provide high-level information relevant to the end user. The information to be presented was already validated by the literature and the focus was on how to visualize this information and the optimal time for its presentation to end users. The visualizations were validated with 18 VLA experts assessing the technical aspects of the visualizations and 4 medical experts assessing their functional aspects.

RESULTS

This IoT learning environment amplifies the evaluation mechanisms already validated by the literature, allowing automatic data collection. First, it uses IoT sensors to automatically correct two of the exercises defined in the FLS (peg transfer and precision cutting), providing real-time visualizations. Second it monitors the movement of the surgical instruments and the force applied to the tissues during the exercise, computing 6 of the high-level indicators used by expert evaluators in their rubrics (efficiency, economy of movement, hand tremor, depth perception, bimanual dexterity, and respect for tissue), providing feedback about the technical skills of the trainee using a radar chart with these six indicators at the end of the training (summative visualizations).

CONCLUSIONS

The proposed IoT learning environment is a promising and cost-effective alternative to help in the training and assessment of surgical technical skills. The system shows the trainees' progress and presents new indicators about the correctness of each specific exercise through real-time visualizations, as well as their general technical skills through summative visualizations, aligned with the 6 more frequent indicators in standardized scales. Early results suggest that although both types of visualizations are useful, it is necessary to reduce the cognitive load of the graphs presented in real time during training. Nevertheless, an additional evaluation is needed to confirm these results.