Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): improving the objective assessment

Can Computerized Simulation be Used to Assess Surgical Proficiency in Laparoscopic Colorectal Surgeries? A Systematic Review

INTRODUCTION

Computerized simulation (CS) of surgery in virtual reality (VR), augmented reality (AR) and mixed reality (MR) settings are used to teach foundational skills, but its applicability in advanced training is to be determined. This review aims to summarize the types of CS available for laparoscopic colorectal surgery (CRS) and its utility in assessment of proficiency.

METHODS

A systematic review of CS in laparoscopic CRS was done on PubMed, Embase, Scopus and Cochrane Library databases.

RESULTS

Eleven relevant observational studies were identified. The most common procedure simulated was laparoscopic colectomy. Assessment using performance metrics measured by the simulator such as path length moved by laparoscopic tools, procedure time and number of discrete movements had the most consistent differentiating ability between expert and non-expert cohorts. Surgeons fared similarly in proficiency scores in assessment with CS compared to assessment with traditional cadaveric or porcine models.

CONCLUSION

CS of laparoscopic CRS may be used in assessment of proficiency using performance metrics measuring economy of movement. CS may be a viable assessment tool in advanced surgical training, but further studies should assess utility of incorporating it as a formal assessment tool in training programs.

A systematic review on classification and assessment of surgical skill levels for simulation-based training programs

BACKGROUND

Nowadays, advances in medical informatics have made minimally invasive surgery (MIS) procedures the preferred choice. However, there are several problems with the education programs in terms of surgical skill acquisition. For instance, defining and objectively measuring surgical skill levels is a challenging process. Accordingly, the aim of this study is to conduct a literature review for an investigation of the current approaches for classifying the surgical skill levels and for identifying the skill training tools and measurement methods.

MATERIALS AND METHODS

In this research, a search is conducted and a corpus is created. Exclusion and inclusion criteria are applied by limiting the number of articles based on surgical education, training approximations, hand movements, and endoscopic or laparoscopic operations. To satisfy these criteria, 57 articles are included in the corpus of this study.

RESULTS

Currently used surgical skill assessment approaches have been summarized. Results show that various classification approaches for the surgical skill level definitions are being used. Besides, many studies are conducted by omitting particularly important skill levels in between. Additionally, some inconsistencies are also identified across the skill level classification studies.

CONCLUSION

In order to improve the benefits of simulation-based training programs, a standardized interdisciplinary approach should be developed. For this reason, specific to each surgical procedure, the required skills should be identified. Additionally, appropriate measures for assessing these skills, which can be defined in simulation-based MIS training environments, should be refined. Finally, the skill levels gained during the developmental stages of these skills, with their threshold values referencing the identified measures, should be redefined in a standardized manner.

Tracking and evaluating motion skills in laparoscopy with inertial sensors

BACKGROUND

Analysis of surgical instrument motion is applicable in surgical skill assessment and monitoring of the learning progress in laparoscopy. Current commercial instrument tracking technology (optical or electromagnetic) has specific limitations and is expensive. Therefore, in this study, we apply inexpensive, off-the-shelf inertial sensors to track laparoscopic instruments in a training scenario.

METHODS

We calibrated two laparoscopic instruments to the inertial sensor and investigated its accuracy on a 3D-printed phantom. In a user study during a one-week laparoscopy training course with medical students and physicians, we then documented and compared the training effect in laparoscopic tasks on a commercially available laparoscopy trainer (Laparo Analytic, Laparo Medical Simulators, Wilcza, Poland) and the newly developed tracking setup.

RESULTS

Eighteen participants (twelve medical students and six physicians) participated in the study. The student subgroup showed significantly poorer results for the count of swings (CS) and count of rotations (CR) at the beginning of the training compared to the physician subgroup (p = 0.012 and p = 0.042). After training, the student subgroup showed significant improvements in the rotatory angle sum, CS, and CR (p = 0.025, p = 0.004 and p = 0.024). After training, there were no significant differences between medical students and physicians. There was a strong correlation between the measured learning success (LS) from the data of our inertial measurement unit system (LS_IMU) and the Laparo Analytic (LS_Lap) (Pearson's r = 0.79).

CONCLUSION

In the current study, we observed a good and valid performance of inertial measurement units as a possible tool for instrument tracking and surgical skill assessment. Moreover, we conclude that the sensor can meaningfully examine the learning progress of medical students in an ex-vivo setting.

Autonomous sequential surgical skills assessment for the peg transfer task in a laparoscopic box-trainer system with three cameras

In laparoscopic surgery, surgeons should develop several manual laparoscopic skills before carrying out real operative procedures using a low-cost box trainer. The Fundamentals of Laparoscopic Surgery (FLS) program was developed as a program to assess fundamental knowledge and surgical skills, required for basic laparoscopic surgery. The peg transfer task is a hands-on exam in the FLS program that assists a trainee to understand the relative minimum amount of grasping force necessary to move the pegs from one place to another place without dropping them. In this paper, an autonomous, sequential assessment algorithm based on deep learning, a multi-object detection method, and, several sequential If-Then conditional statements have been developed to monitor each step of a surgeon’s performance. Images from three different cameras are used to assess whether the surgeon executes the peg transfer task correctly and to display a notification on any errors on the monitor immediately. This algorithm improves the performance of a laparoscopic box-trainer system using top, side, and front cameras and removes the need for any human monitoring during a peg transfer task. The developed algorithm can detect each object and its status during a peg transfer task and notifies the resident about the correct or failed outcome. In addition, this system can correctly determine the peg transfer execution time, and the move, carry, and dropped states for each object by the top, side, and front-mounted cameras. Based on the experimental results, the proposed surgical skill assessment system can identify each object at a high score of fidelity, and the train-validation total loss for the single-shot detector (SSD) ResNet50 v1 was about 0.05. Also, the mean average precision (mAP) and Intersection over Union (IoU) of this detection system were 0.741, and 0.75, respectively. This project is a collaborative research effort between the Department of Electrical and Computer Engineering and the Department of Surgery, at Western Michigan University.

Scene-dependent, feedforward eye gaze metrics can differentiate technical skill levels of trainees in laparoscopic surgery

INTRODUCTION

In laparoscopic surgery, looking in the target areas is an indicator of proficiency. However, gaze behaviors revealing feedforward control (i.e., looking ahead) and their importance have been under-investigated in surgery. This study aims to establish the sensitivity and relative importance of different scene-dependent gaze and motion metrics for estimating trainee proficiency levels in surgical skills.

METHODS

Medical students performed the Fundamentals of Laparoscopic Surgery peg transfer task while recording their gaze on the monitor and tool activities inside the trainer box. Using computer vision and fixation algorithms, five scene-dependent gaze metrics and one tool speed metric were computed for 499 practice trials. Cluster analysis on the six metrics was used to group the trials into different clusters/proficiency levels, and ANOVAs were conducted to test differences between proficiency levels. A Random Forest model was trained to study metric importance at predicting proficiency levels.

RESULTS

Three clusters were identified, corresponding to three proficiency levels. The correspondence between the clusters and proficiency levels was confirmed by differences between completion times (F2,488 = 38.94, p < .001). Further, ANOVAs revealed significant differences between the three levels for all six metrics. The Random Forest model predicted proficiency level with 99% out-of-bag accuracy and revealed that scene-dependent gaze metrics reflecting feedforward behaviors were more important for prediction than the ones reflecting feedback behaviors.

CONCLUSION

Scene-dependent gaze metrics revealed skill levels of trainees more precisely than between experts and novices as suggested in the literature. Further, feedforward gaze metrics appeared to be more important than feedback ones at predicting proficiency.

Comparing Simulator Metrics and Rater Assessment of Laparoscopic Suturing Skills

BACKGROUND

Laparoscopic intracorporeal suturing is important to master and competence should be ensured using an optimal method in a simulated environment before proceeding to real operations. The objectives of this study were to gather validity evidence for two tools for assessing laparoscopic intracorporeal knot tying and compare the rater-based assessment of laparoscopic intracorporeal suturing with the assessment based on simulator metrics.

METHODS

Twenty-eight novices and 19 experienced surgeons performed four laparoscopic sutures on a Simball Box simulator twice. Two surgeons used the Intracorporeal Suturing Assessment Tool (ISAT) for blinded video rating.

RESULTS

Composite Simulator Score (CSS) had higher test-retest reliability than the ISAT. The correlation between the number performed procedures including suturing and ISAT score was 0.51, p<0.001, and 0.59 p<0.001 for CSS. We found an inter-rater reliability (0.72, p<0.001 for test 1 and 0.53 p<0.001 for test 2). The pass/fail rates for ISAT and CSS were similar.

CONCLUSION

CSS and ISAT provide similar results for assessing laparoscopic suturing but assess different aspects of performance. Using simulator metrics and raters' assessments in combination should be considered for a more comprehensive evaluation of laparoscopic knot-tying competency.

Fuzzy logic supervisor –A surgical skills assessment system using multi-class detection of laparoscopic box-trainer instruments

Recent developments in deep learning can be used in skill assessments for laparoscopic surgeons. In Minimally Invasive Surgery (MIS), surgeons should acquire many skills before carrying out a real operation. The Laparoscopic Surgical Box-Trainer allows surgery residents to train on specific skills that are not traditionally taught to them. This study aims to automatically detect the tips of laparoscopic instruments, localize a point, evaluate the detection accuracy to provide valuable assessment and expedite the development of surgery skills and assess the trainees’ performance using a Multi-Input-Single-Output Fuzzy Logic Supervisor system. The output of the fuzzy logic assessment is the performance evaluation for the surgeon, and it is quantified in percentages. Based on the experimental results, the trained SSD Mobilenet V2 FPN can identify each instrument at a score of 70% fidelity. On the other hand, the trained SSD ResNet50 V1 FPN can detect each instrument at the score of 90% fidelity, in each location within a region of interest, and determine their relative distance with over 65% and 80% reliability, respectively. This method can be applied in different types of laparoscopic tooltip detection. Because there were a few instances when the detection failed, and the system was designed to generate pass-fail assessment, we recommend improving the measurement algorithm and the performance assessment by adding a camera to the system and measuring the distance from multiple perspectives.

Subjective vs Computerized Assessment of Surgeon Skill Level During Mastoidectomy

This pilot study examines the use of surgical instrument tracking and motion analysis in objectively measuring surgical performance. Accuracy of objective measures in distinguishing between surgeons of different levels was compared to that of subjective assessments. Twenty-four intraoperative video clips of mastoidectomies performed by junior residents (n = 12), senior residents (n = 8), and faculty (n = 4) were sent to otolaryngology programs via survey, yielding 708 subjective ratings of surgical experience level. Tracking software captured the total distance traveled by the drill, suction irrigator, and patient's head. Measurements were used to predict surgeon level of training, and accuracy was estimated via area under the curve (AUC) of receiver operating characteristic curves. Key objective metrics proved more accurate than subjective evaluations in determining both faculty vs resident level and senior vs junior resident level. The findings of this study suggest that objective analysis using computer software has the potential to improve the accuracy of surgical skill assessment.

Evaluation of Surgical Skills during Robotic Surgery by Deep Learning-Based Multiple Surgical Instrument Tracking in Training and Actual Operations

As the number of robotic surgery procedures has increased, so has the importance of evaluating surgical skills in these techniques. It is difficult, however, to automatically and quantitatively evaluate surgical skills during robotic surgery, as these skills are primarily associated with the movement of surgical instruments. This study proposes a deep learning-based surgical instrument tracking algorithm to evaluate surgeons’ skills in performing procedures by robotic surgery. This method overcame two main drawbacks: occlusion and maintenance of the identity of the surgical instruments. In addition, surgical skill prediction models were developed using motion metrics calculated from the motion of the instruments. The tracking method was applied to 54 video segments and evaluated by root mean squared error (RMSE), area under the curve (AUC), and Pearson correlation analysis. The RMSE was 3.52 mm, the AUC of 1 mm, 2 mm, and 5 mm were 0.7, 0.78, and 0.86, respectively, and Pearson’s correlation coefficients were 0.9 on the x-axis and 0.87 on the y-axis. The surgical skill prediction models showed an accuracy of 83% with Objective Structured Assessment of Technical Skill (OSATS) and Global Evaluative Assessment of Robotic Surgery (GEARS). The proposed method was able to track instruments during robotic surgery, suggesting that the current method of surgical skill assessment by surgeons can be replaced by the proposed automatic and quantitative evaluation method.

Simulation platforms to assess laparoscopic suturing skills: a scoping review

BACKGROUND

Laparoscopic suturing (LS) has become a common technique used in a variety of advanced laparoscopic procedures. However, LS is a challenging skill to master, and many trainees may not be competent in performing LS at the end of their training. The purpose of this review is to identify simulation platforms available for assessment of LS skills, and determine the characteristics of the platforms and the LS skills that are targeted.

METHODS

A scoping review was conducted between January 1997 and October 2018 for full-text articles. The search was done in various databases. Only articles written in English or French were included. Additional studies were identified through reference lists. The search terms included "laparoscopic suturing" and "clinical competence."

RESULTS

Sixty-two studies were selected. The majority of the simulation platforms were box trainers with inanimate tissue, and targeted basic suturing and intracorporeal knot-tying techniques. Most of the validation came from internal structure (rater reliability) and relationship to other variables (compare training levels/case experience, and various metrics). Consequences were not addressed in any of the studies.

CONCLUSION

We identified many types of simulation platforms that were used for assessing LS skills, with most being for assessment of basic skills. Platforms assessing the competence of trainees for advanced LS skills were limited. Therefore, future research should focus on development of LS tasks that better reflect the needs of the trainees.

Distance Measures for Surgical Process Models

Background: The development of new resources, such as surgical techniques and approaches, results in continuous modification of surgery. To assess these modifications, it is necessary to use measures that quantify the impact of resources on surgical processes.

Objectives: The objective of this work is to introduce and evaluate distance measurements that are able to represent differences in the courses of surgical interventions as processes.

Methods: Hence, we present four different distance measures for surgical processes: the Jaccard distance, Levenshtein distance, Adjacency distance, and Graph matching distance. These measures are formally introduced and evaluated by applying them to clinical data sets from laparoscopic training in pediatric surgery.

Results: We analyzed the distances of 450 surgical processes using these four measures with a focus on the difference in surgical processes performed by novices and by experienced surgeons. The Levenshtein and Adjacency distances were best suited to measure distances between surgical processes.

Conclusion: The measurement of distances between surgical processes is necessary to estimate the benefit of new surgical techniques and strategies.

Outcome quality assessment by surgical process compliance measures in laparoscopic surgery

OBJECTIVE

The effective and efficient assessment, management, and evolution of surgical processes are intrinsic to excellent patient care. Hence, in addition to economic interests, the quality of the outcome is of great importance. Process benchmarking examines the compliance of an intraoperative surgical process to another process that is considered as best practice. The objective of this work is to assess the relationship between the course and the outcome of surgical processes of the study.

MATERIALS AND METHODS

By assessing 450 skill practices on rapid prototyping models in minimally invasive surgery training, we extracted descriptions of surgical processes and examined the hypothesis that a significant relationship exists between the course of a surgical process and the quality of its outcome.

RESULTS

The results showed a significant correlation with Person correlation coefficients >0.05 between the quality of process outcome and process compliance for simple and complex suturing tasks in the study.

CONCLUSIONS

We conclude that high process compliance supports good quality outcomes and, therefore, excellent patient care. We also showed that a deviation from best training processes led to a decreased outcome quality. This is relevant for identifying requirements for surgical processes, for generating feedback for the surgeon with regard to human factors and for inducing changes in the workflow in order to improve the outcome quality.