Systems and technologies for objective evaluation of technical skills in laparoscopic surgery

Application of an Optical Tracking System for Motor Skill Assessment in Laparoscopic Surgery

Objective

Motion analysis of surgical instruments can be used to evaluate laparoscopic surgical skills, and this study assessed the validity of an optical tracking system for the assessment of laparoscopic surgical motor skills.

Methods

Ten experienced surgeons and ten novices were recruited to complete the transferring tasks on a laparoscopic simulator. An optical tracking system, Micron Tracker, was used to capture the marker points on each instrument and to obtain the coordinates of the marker points and the corresponding instrument tip coordinates. The data are processed to create a coordinate system based on the laparoscopic simulator and to calculate the movement parameters of the instruments, such as operating time, path length, speed, acceleration, and smoothness. At the same time, the range of motion of the instrument (insertion depth and pivoting angle) is also calculated.

Results

The position that the tip of the instrument can reach is a small, irregularly shaped spatial area. Significant differences (p < 0.05) were found between the surgeon and novice groups in parameters such as operating time, path length, mean speed, mean acceleration, and mean smoothness. The range of insertion depth of the instruments was approximately 150 mm to 240 mm, and the pivoting angles of the left and right instruments were 30.9° and 46.6° up and down and 28.0° and 35.0° left and right, respectively.

Conclusions

The optical tracking system was effective in subjectively evaluating laparoscopic surgical skills, with significant differences between the surgeon and novice groups in terms of movement parameters, but not in terms of range of motion.

A cost-effective IoT learning environment for the training and assessment of surgical technical skills with visual learning analytics

BACKGROUND

Surgeons need to train and certify their technical skills. This is usually done with the intervention of experts who monitor and assess trainees. Nevertheless, this is a time-consuming task that is subject to variations among evaluators. In recent decades, subjectivity has been significantly reduced through 1) the introduction of standard curricula, such as the Fundamentals of Laparoscopic Surgery (FLS) program, which measures students' performance in specific exercises, and 2) rubrics, which are widely accepted in the literature and serve to provide feedback about the overall technical skills of the trainees. Although these two elements reduce subjectivity, they do not, however, eliminate the figure of the expert evaluator, and so the process remains time consuming.

OBJECTIVES

The objective of this work is to automate those parts of the work of the expert evaluator that the technology can measure objectively, using sensors to collect evidence, and visualizations to provide feedback. We designed and developed 1) a cost-effective IoT (Internet of Things) learning environment for the training and assessment of surgical technical skills and 2) visualizations supported by the literature on visual learning analytics (VLA) to provide feedback about the exercises (in real time) and overall performance (at the end of the training) of the trainee.

METHODS

A hybrid approach was followed based on previous research for the design of the sensor based IoT learning environment. Previous studies were used as the basis for getting best practices on the tracking of surgical instruments and on the detection of the force applied to the tissue, with a focus on reducing the costs of data collection. The monitoring of the specific exercises required the design of sensors and collection mechanisms from scratch as there is little existing research on this subject. Moreover, it was necessary to design the overall architecture to collect, process, synchronize and communicate the data coming from the different sensors to provide high-level information relevant to the end user. The information to be presented was already validated by the literature and the focus was on how to visualize this information and the optimal time for its presentation to end users. The visualizations were validated with 18 VLA experts assessing the technical aspects of the visualizations and 4 medical experts assessing their functional aspects.

RESULTS

This IoT learning environment amplifies the evaluation mechanisms already validated by the literature, allowing automatic data collection. First, it uses IoT sensors to automatically correct two of the exercises defined in the FLS (peg transfer and precision cutting), providing real-time visualizations. Second it monitors the movement of the surgical instruments and the force applied to the tissues during the exercise, computing 6 of the high-level indicators used by expert evaluators in their rubrics (efficiency, economy of movement, hand tremor, depth perception, bimanual dexterity, and respect for tissue), providing feedback about the technical skills of the trainee using a radar chart with these six indicators at the end of the training (summative visualizations).

CONCLUSIONS

The proposed IoT learning environment is a promising and cost-effective alternative to help in the training and assessment of surgical technical skills. The system shows the trainees' progress and presents new indicators about the correctness of each specific exercise through real-time visualizations, as well as their general technical skills through summative visualizations, aligned with the 6 more frequent indicators in standardized scales. Early results suggest that although both types of visualizations are useful, it is necessary to reduce the cognitive load of the graphs presented in real time during training. Nevertheless, an additional evaluation is needed to confirm these results.

A new basic thoracoscopic surgical skill training and assessment system using automatic scoring techniques

PURPOSE

We report a new thoracoscopic surgical skill training and assessment system with automatic scoring techniques, the Huaxi Intelligent Thoracoscopic Skill Training and Assessment (HITSTA) system. We also evaluated the discriminative ability of this system compared to our conventional scoring method at our institution.

METHODS

We retrospectively collected training data of thoracic board-certified thoracic surgeons at West China Hospital, Sichuan University from January 1, 2018 to January 1, 2019. Surgeons were assessed by HITSTA system and human examiners simultaneously. Total scores were summed from 3 tasks (grasping with delivery, pattern cutting, and suture with knot). Bland-Altman analysis was used to test agreement of scores made by HITSTA system (automatic scoring) and human examiners (manual scoring). Differentiation ability was also compared between the two scoring methods.

RESULTS

Thirty-nine surgeons were recruited. Scores made by HITSTA system and human examiners were not consistent. For suture with knot, automatic scoring method could detect the score differences between different training status (trained: 26.92 ± 12.04, untrained: 19.85 ± 11.12; p = 0.026) and training duration (< 10 h: 20.67 ± 15.23,  ≥ 10 h: 31.92 ± 5.56; p = 0.003). For total scores, automatic scoring approach could discriminate between different training status (trained: 71.90 ± 12.63; untrained: 61.41 ± 13.87; p = 0.016) and training duration (< 10 h: 65.23 ± 15.31;  ≥ 10 h 77.23 ± 6.94; p = 0.046).

CONCLUSION

HITSTA system could discriminate the different levels of thoracoscopic surgical skills better than the traditional manual scoring method. Larger prospective studies are warranted to validate the differentiation ability of HITSTA system.

Utilization of Flexible-Wearable Sensors to Describe the Kinematics of Surgical Proficiency

BACKGROUND

Traditional assessment (e.g., checklists, videotaping) for surgical proficiency may lead to subjectivity and does not predict performance in the clinical setting. Hand motion analysis is evolving as an objective tool for grading technical dexterity; however, most devices accompany with technical limitations or discomfort. We purpose the use of flexible wearable sensors to evaluate the kinematics of surgical proficiency.

METHODS

Surgeons were recruited and performed a vascular anastomosis task in a single institution. A modified objective structured assessment of technical skills (mOSATS) was used for technical qualification. Flexible wearable sensors (BioStamp RCTM, mc10 Inc., Lexington, MA) were placed on the dorsum of the dominant hand (DH) and nondominant hand (nDH) to measure kinematic parameters: path length (T_path), mean (V_mean) and peak (V_peak) velocity, number of hand movements (N_move), ratio of DH to nDH movements (rMov), and time of task (tTask) and further compared with the mOSATS score.

RESULTS

Participants were categorized as experts (n = 12) and novices (n = 8) based on a cutoff mean mOSATS score. Significant differences for tTask (P = 0.02), rMov (P = 0.07), DH T_path (P = 0.04), V_mean (P = 0.07), V_peak (P = 0.04), and nDH N_move (P = 0.02) were in favor of the experts. Overall, mOSATS had significant correlation with tTask (r = -0.69, P = 0.001), N_move of DH (r = -0.44, P = 0.047) and nDH (r = -0.66, P = 0.001), and rMov (r = 0.52, P = 0.017).

CONCLUSIONS

Hand motion analysis evaluated by flexible wearable sensors is feasible and informative. Experts utilize coordinated two-handed motion, whereas novices perform one-handed tasks in a hastily jerky manner. These tendencies create opportunity for improvement in surgical proficiency among trainees.

Objective and automated assessment of surgical technical skills with IoT systems: A systematic literature review

The assessment of surgical technical skills to be acquired by novice surgeons has been traditionally done by an expert surgeon and is therefore of a subjective nature. Nevertheless, the recent advances on IoT (Internet of Things), the possibility of incorporating sensors into objects and environments in order to collect large amounts of data, and the progress on machine learning are facilitating a more objective and automated assessment of surgical technical skills. This paper presents a systematic literature review of papers published after 2013 discussing the objective and automated assessment of surgical technical skills. 101 out of an initial list of 537 papers were analyzed to identify: 1) the sensors used; 2) the data collected by these sensors and the relationship between these data, surgical technical skills and surgeons' levels of expertise; 3) the statistical methods and algorithms used to process these data; and 4) the feedback provided based on the outputs of these statistical methods and algorithms. Particularly, 1) mechanical and electromagnetic sensors are widely used for tool tracking, while inertial measurement units are widely used for body tracking; 2) path length, number of sub-movements, smoothness, fixation, saccade and total time are the main indicators obtained from raw data and serve to assess surgical technical skills such as economy, efficiency, hand tremor, or mind control, and distinguish between two or three levels of expertise (novice/intermediate/advanced surgeons); 3) SVM (Support Vector Machines) and Neural Networks are the preferred statistical methods and algorithms for processing the data collected, while new opportunities are opened up to combine various algorithms and use deep learning; and 4) feedback is provided by matching performance indicators and a lexicon of words and visualizations, although there is considerable room for research in the context of feedback and visualizations, taking, for example, ideas from learning analytics.

Application of multidirectional stitching technology in a laparoscopic suturing instructional program: a randomized controlled trial

BACKGROUND

Surgeon suturing technology plays a pivotal role in patient recovery after laparoscopic surgery. Intracorporal suturing and knot tying in minimally invasive surgery are particularly challenging and represent a key skill for advanced procedures. In this study, we compared the application of multidirectional stitching technology with application of the traditional method in a laparoscopic suturing instructional program.

METHODS

We selected forty residents within two years of graduation to assess the specialized teaching of laparoscopic suturing with laparoscopic simulators. The forty students were randomly divided into two groups, a control group and an experimental group, with twenty students in each group. The control group was scheduled to learn the traditional suture method, and the experimental group applied multidirectional stitching technology. The grades for suturing time, thread length, accuracy of needle entry, stability of the knot, tissue integrity, and tightness of the tissue before and after the training program were calculated.

RESULTS

There was no significant difference between the two groups before the learning intervention. After the program, both groups significantly improved in each subject. There were significant differences between the control group and the experimental group in suture time (P = 0.001), accuracy of needle entry and exit (P = 0.035), and whether the suture tissue had cracks (P = 0.030). However, the two groups showed non-significant differences in thread length (P = 0.093), stablity of the knot (P = 0.241), or tightness of the tissue (P = 0.367).

CONCLUSIONS

Multidirectional stitching technology improves the efficiency and effectiveness of traditional laparoscopic suture instructional programs. It might be a practicable, novel training method for acquiring proficiency in manual laparoscopic skills in a training setting.

Development of a novel training system for laparoscopic inguinal hernia repair

Background: Laparoscopic inguinal hernia repair is considered technically difficult. Although a simulation-based curriculum has been developed to help trainees, access to simulation training outside North America is limited. This study aimed to develop an educational system for the transabdominal preperitoneal (TAPP) approach based on the TAPP checklist, an assessment tool we had developed and validated earlier. Material and methods: Consensus within the TAPP education working group consisting of local TAPP experts, hernia experts and a surgical educator to develop educational tools and the educational system based on the TAPP checklist. The system was piloted in several institutes, and participants were surveyed on its efficacy. Results: We systematically developed an educational video and training manual explaining each item of the TAPP checklist and conveying basic knowledge of the procedure. We integrated the training tool with evaluation and feedback to develop an educational system. In a pilot study, seven trainees at five hospitals were trained using this system and found it very useful for mastering the TAPP procedure. Conclusions: We have developed a training system for TAPP procedure and successfully implemented it in several hospitals. Further research will focus on the educational value of this tool.

Assessment potential of a new suture simulator in laparoscopic surgical skills training

BACKGROUND

The skills necessary for performing effective laparoscopic suturing are difficult to acquire; as a result, simulators for learning these skills are rapidly becoming integrated into surgical training. The aim of the study was to verify whether a new hybrid simulator has the potential to measure skill improvement in young, less experienced gastroenterological surgeons.

MATERIAL AND METHODS

The study included 12 surgeons (median age, 29 (27-38)] years; 11 men (91.7%), one woman (8.3%)) who participated in a two-day laparoscopic training seminar. We used the new simulator before and after the program to evaluate individual performance. Skills were evaluated using five criteria: volume of air pressure leakage, number of full-thickness sutures, suture tension, wound area, and performance time.

RESULTS

Air pressure leakage was significantly higher after than before the training (p = .027). The number of full-thickness sutures was significantly higher post-training (p < .01). Suture tension was significantly less post-training (p = .011). Wound opening areas were significantly smaller post-training (p = .018). Performance time was significantly shorter post-training (p = .032).

CONCLUSIONS

Our study demonstrated the assessment quality of this new laparoscopic suture simulator.

Achieving Interface and Environment Fidelity in the Virtual Basic Laparoscopic Surgical Trainer

Virtual reality trainers are educational tools with great potential for laparoscopic surgery. They can provide basic skills training in a controlled environment and free of risks for patients. They can also offer objective performance assessment without the need for proctors. However, designing effective user interfaces that allow the acquisition of the appropriate technical skills on these systems remains a challenge. This paper aims to examine a process for achieving interface and environment fidelity during the development of the Virtual Basic Laparoscopic Surgical Trainer (VBLaST). Two iterations of the design process were conducted and evaluated. For that purpose, a total of 42 subjects participated in two experimental studies in which two versions of the VBLaST were compared to the accepted standard in the surgical community for training and assessing basic laparoscopic skills in North America, the FLS box-trainer. Participants performed 10 trials of the peg transfer task on each trainer. The assessment of task performance was based on the validated FLS scoring method. Moreover, a subjective evaluation questionnaire was used to assess the fidelity aspects of the VBLaST relative to the FLS trainer. Finally, a focus group session with expert surgeons was conducted as a comparative situated evaluation after the first design iteration. This session aimed to assess the fidelity aspects of the early VBLaST prototype as compared to the FLS trainer. The results indicate that user performance on the earlier version of the VBLaST resulting from the first design iteration was significantly lower than the performance on the standard FLS box-trainer. The comparative situated evaluation with domain experts permitted us to identify some issues related to the visual, haptic and interface fidelity on this early prototype. Results of the second experiment indicate that the performance on the second generation VBLaST was significantly improved as compared to the first generation and not significantly different from that of the standard FLS box-trainer. Furthermore, the subjects rated the fidelity features of the modified VBLaST version higher than the early version. These findings demonstrate the value of the comparative situated evaluation sessions entailing hands on reflection by domain experts to achieve the environment and interface fidelity and training objectives when designing a virtual reality laparoscopic trainer. This suggests that this method could be used successfully in the future to enhance the value of VR systems as an alternative to physical trainers for laparoscopic surgery skills. Some recommendations on how to use this method to achieve the environment and interface fidelity of a VR laparoscopic surgical trainer are identified.

Ergonomic comparison of laparoscopic hand instruments in a single site surgery simulator with novices

BACKGROUND

Single-site surgery improves cosmesis but increases procedural difficulty. Enhanced instruments could improve procedural efficiency leading to better patient outcomes.

MATERIAL AND METHODS

One pair of non-articulating (straight) and two different pairs of articulating laparoscopic instruments were evaluated using a peg-transfer surgical task simulator by premedical college students. The instruments were comparatively tested using task performance measures, ergonomic measures, and participant questionnaires.

RESULTS

The straight instrument produced significantly higher task performance scores and lower task times compared to both articulating instruments (p < 0.05). The straight instrument required less muscle activation and less wrist deviation than the articulating instruments to perform the same task. Participants rated the straight instrument significantly easier to use and less difficult to complete the task than with either articulating instrument (p < 0.05 for both).

CONCLUSIONS

This exploratory study suggests that novices have difficulty using articulating instruments and perform better using straight laparoscopic instruments when first attempting LESS surgical tasks. Although a study with post-graduate medical trainees is needed to confirm these results, trainees should initially practice LESS with non-articulating instruments to gain proficiency at basic laparoscopic tasks. Additionally, redesigning articulating instruments to specifically address the spatial constraints and learning curve of LESS may also improve trainee performance and instrument usability.