Pediatric Thoracoscopic Surgical Simulation Using a Rapid-Prototyped Chest Model and Motion Sensors Can Better Identify Skilled Surgeons Than a Conventional Box Trainer

Validated simulation models in pediatric surgery: A review

INTRODUCTION

This review evaluates the validation and availability of simulation models in the field of pediatric surgery that can be used for training purposes.

METHODS

MEDLINE and EMBASE were searched for studies describing a simulation models in pediatric surgery. Articles were included if face, content and/or construct validity was described. Additionally, the costs and availability were assessed. Validation scores for each model were depicted as percentage (0-100), based on the reported data, to compare the outcomes. A score of >70% was considered adequate.

RESULTS

Forty-three studies were identified, describing the validation process of 38 simulation models. Face validity was evaluated in 33 articles, content in 36 and construct in 19. Twenty-two models received adequate validation scores (>70%). The majority (27/38, 70%) was strictly inanimate. Five models were available for purchase and eleven models were replicable based on the article.

CONCLUSION

The number of validated inanimate simulation models for pediatric surgery procedures is growing, however, few are replicable or available for widespread training purposes.

LEVEL OF EVIDENCE

Level II.

Evaluation of minimally invasive surgical skills training: comparing a neonatal esophageal atresia/tracheoesophageal fistula model with a dry box

BACKGROUND

Pediatric surgeons require highly advanced minimally invasive surgical skills to perform rare and complex surgeries in a very vulnerable population. We developed a neonatal esophageal atresia (EA) model to improve thoracoscopic surgical skills. This study aimed to evaluate the concurrent validity of the model by undertaking pre- and post-training skills assessments in two groups of students with no prior experience performing minimally invasive surgery, using the EA model and a dry box (DB).

METHODS

A pilot study was performed. The participants were randomly divided into two groups: one trained using the DB and one trained using the EA model. Both groups practiced a minimally invasive surgical suture task. The task completion time, 29-point checklist score, modified suturing error sheet score, and three-dimensional forceps movement in both groups were compared pre-and post-training by video analysis.

RESULTS

The EA model task was significantly more difficult than that of the DB. Both groups showed significant improvement in the task time, 29-point checklist score, and modified suturing error sheet score; however, the EA model training was more efficient in improving each error item. Regarding forceps movement, the EA model training significantly decreased wasted motion, whereas the DB was limited in this regard.

CONCLUSIONS

Short-term training on the EA model, which was more technically demanding than the DB, decreased technical error and wasted motion, and allowed learners to acquire surgical skills more efficiently than training with the DB model. These facts revealed the concurrent validity of the EA model.

Face and construct validity assessment of training models for intestinal anastomosis in low-birth-weight infants

PURPOSE

It is difficult to perform intestinal anastomosis in low-birth-weight infants because the intestinal diameter is small and the discrepancy in diameter of the proximal and distal intestines is often large, but there has been no optimal-sized training model. Therefore, we developed a new intestinal anastomosis training model that imitated the size of the intestine in low-birth-weight infants, and evaluated its face and construct validity.

METHODS

Two intestinal models were developed with crossMedical, Inc. using a hydrophilic acrylic material (wet model) or a polyurethane soft resin (dry model). The inner diameter of the simulated intestinal tract was 15 mm on the oral end and 6 mm on the anal end. Thirteen pediatric surgeons performed anastomosis and responded to the questionnaire.

RESULTS

In the questionnaire, the wet model had significantly higher scores than the dry model in "appearance", "softness" and "usefulness for training". In the anastomotic results of the wet model, the anastomosis leak pressure was significantly correlated with the number of intestinal anastomotic experiences in low-birth-weight infants (correlation coefficient = 0.64, P = 0.035).

CONCLUSIONS

The wet-type intestinal anastomosis model showed good face validity. Its leak pressure had a significant correlation with clinical experience; thus, construct validity was demonstrated.

A cost-effective IoT learning environment for the training and assessment of surgical technical skills with visual learning analytics

BACKGROUND

Surgeons need to train and certify their technical skills. This is usually done with the intervention of experts who monitor and assess trainees. Nevertheless, this is a time-consuming task that is subject to variations among evaluators. In recent decades, subjectivity has been significantly reduced through 1) the introduction of standard curricula, such as the Fundamentals of Laparoscopic Surgery (FLS) program, which measures students' performance in specific exercises, and 2) rubrics, which are widely accepted in the literature and serve to provide feedback about the overall technical skills of the trainees. Although these two elements reduce subjectivity, they do not, however, eliminate the figure of the expert evaluator, and so the process remains time consuming.

OBJECTIVES

The objective of this work is to automate those parts of the work of the expert evaluator that the technology can measure objectively, using sensors to collect evidence, and visualizations to provide feedback. We designed and developed 1) a cost-effective IoT (Internet of Things) learning environment for the training and assessment of surgical technical skills and 2) visualizations supported by the literature on visual learning analytics (VLA) to provide feedback about the exercises (in real time) and overall performance (at the end of the training) of the trainee.

METHODS

A hybrid approach was followed based on previous research for the design of the sensor based IoT learning environment. Previous studies were used as the basis for getting best practices on the tracking of surgical instruments and on the detection of the force applied to the tissue, with a focus on reducing the costs of data collection. The monitoring of the specific exercises required the design of sensors and collection mechanisms from scratch as there is little existing research on this subject. Moreover, it was necessary to design the overall architecture to collect, process, synchronize and communicate the data coming from the different sensors to provide high-level information relevant to the end user. The information to be presented was already validated by the literature and the focus was on how to visualize this information and the optimal time for its presentation to end users. The visualizations were validated with 18 VLA experts assessing the technical aspects of the visualizations and 4 medical experts assessing their functional aspects.

RESULTS

This IoT learning environment amplifies the evaluation mechanisms already validated by the literature, allowing automatic data collection. First, it uses IoT sensors to automatically correct two of the exercises defined in the FLS (peg transfer and precision cutting), providing real-time visualizations. Second it monitors the movement of the surgical instruments and the force applied to the tissues during the exercise, computing 6 of the high-level indicators used by expert evaluators in their rubrics (efficiency, economy of movement, hand tremor, depth perception, bimanual dexterity, and respect for tissue), providing feedback about the technical skills of the trainee using a radar chart with these six indicators at the end of the training (summative visualizations).

CONCLUSIONS

The proposed IoT learning environment is a promising and cost-effective alternative to help in the training and assessment of surgical technical skills. The system shows the trainees' progress and presents new indicators about the correctness of each specific exercise through real-time visualizations, as well as their general technical skills through summative visualizations, aligned with the 6 more frequent indicators in standardized scales. Early results suggest that although both types of visualizations are useful, it is necessary to reduce the cognitive load of the graphs presented in real time during training. Nevertheless, an additional evaluation is needed to confirm these results.

Construct validation of a 3D printed neonatal thoracoscopic simulator: Can it measure expertise?

BACKGROUND

acquiring technical expertise for neonatal thoracoscopy is challenging. To address this, we designed a fully synthetic thoracoscopic simulator for which we established its construct validity.

METHODS

three thoracoscopic tasks were assessed: ring transfer, needle pass and incision of a blind upper esophageal pouch (EA cut). Participants watched instructional videos with accompanying written instructions for each task before having their attempt video recorded. All tasks were marked by three blinded pediatric surgeons using a modified Objective Structured Assessment of Technical Skills (OSATS). Scores were assessed using appropriate statistical analysis and inter-rater reliability was analyzed by interclass correlation coefficient (ICC).

RESULTS

23 participants completed the ring transfer and needle pass and 21 the EA cut: 5 experts (consultant surgeons), 5 intermediate (registrars on a training program) and 13 novices (medical students, house surgeons or non-training registrars). All three tasks distinguished between novice and intermediate/expert (ring transfer p = 0.00001, needle pass p = 0.0004 and EA cut p = 0.0014, respectively). Interrater reliability was good for ring transfer and needle pass but poor for EA cut.

CONCLUSION

the tasks distinguished between novice and intermediate/expert but not between expert and intermediate. In needle pass and EA cut, there was a trend for the experts to score higher than intermediate participants. Ring transfer and needle pass tasks achieved construct validity, had good interrater reliability and were found to be useful in assessing a novice surgeon's progression towards the intermediate level. Distinguishing between intermediate and expert may require assessment of more complex tasks such as intracorporeal suturing and tying.

LEVEL OF EVIDENCE

II.

Objective and automated assessment of surgical technical skills with IoT systems: A systematic literature review

The assessment of surgical technical skills to be acquired by novice surgeons has been traditionally done by an expert surgeon and is therefore of a subjective nature. Nevertheless, the recent advances on IoT (Internet of Things), the possibility of incorporating sensors into objects and environments in order to collect large amounts of data, and the progress on machine learning are facilitating a more objective and automated assessment of surgical technical skills. This paper presents a systematic literature review of papers published after 2013 discussing the objective and automated assessment of surgical technical skills. 101 out of an initial list of 537 papers were analyzed to identify: 1) the sensors used; 2) the data collected by these sensors and the relationship between these data, surgical technical skills and surgeons' levels of expertise; 3) the statistical methods and algorithms used to process these data; and 4) the feedback provided based on the outputs of these statistical methods and algorithms. Particularly, 1) mechanical and electromagnetic sensors are widely used for tool tracking, while inertial measurement units are widely used for body tracking; 2) path length, number of sub-movements, smoothness, fixation, saccade and total time are the main indicators obtained from raw data and serve to assess surgical technical skills such as economy, efficiency, hand tremor, or mind control, and distinguish between two or three levels of expertise (novice/intermediate/advanced surgeons); 3) SVM (Support Vector Machines) and Neural Networks are the preferred statistical methods and algorithms for processing the data collected, while new opportunities are opened up to combine various algorithms and use deep learning; and 4) feedback is provided by matching performance indicators and a lexicon of words and visualizations, although there is considerable room for research in the context of feedback and visualizations, taking, for example, ideas from learning analytics.

Evaluation methods and impact of simulation-based training in pediatric surgery: a systematic review

PURPOSE

The aim of this study was to identify (1) the type of skill evaluation methods and (2) how the effect of training was evaluated in simulation-based training (SBT) in pediatric surgery.

METHODS

Databases of PubMed, Cochrane Library, and Web of Science were searched for articles published from January 2000 to January 2017. Search concepts of Medical Subject Heading terms were "surgery," "pediatrics," "simulation," and "training, evaluation."

RESULTS

Of 5858 publications identified, 43 were included. Twenty papers described simulators as assessment tools used to evaluate technical skills. Reviewers differentiated between experts and trainees using a scoring system (45%) and/or a checklist (25%). Simulators as training tools were described in 23 papers. While the training's effectiveness was measured using performance assessment scales (52%) and/or surveys (43%), no study investigated the improvement of the clinical outcomes after SBT.

CONCLUSION

Scoring, time, and motion analysis methods were used for the evaluation of basic techniques of laparoscopic skills. Only a few SBT in pediatric surgery have definite goals with clinical effect. Future research needs to demonstrate the educational effect of simulators as assessment or training tools on SBT in pediatric surgery.

Comparison of Three- and Two-Dimensional Laparoscopy in Pediatric Nissen Fundoplication

Aim: This study aimed to evaluate the usefulness of three-dimensional (3D) versus conventional two-dimensional (2D) vision in pediatric laparoscopic Nissen fundoplication. Materials and Methods: Medical records and procedure videos of patients 18 years old or younger who underwent laparoscopic Nissen fundoplication at a single institution between January 2015 and August 2018 were retrospectively reviewed. The total pneumoperitoneum and wrapping times were measured. The number of errors during the first stitch of the wrapping was counted. Three-dimensional laparoscopy was introduced in January 2017, and data were compared between the previous 2D and current 3D groups. A questionnaire was provided to each surgeon who performed the 3D procedure. Results: Laparoscopic fundoplication was performed under 2D vision in 32 patients and under 3D vision in 19 patients. Mean ages and weights at surgery were similar between the groups. There was no significant differences in pneumoperitoneum and wrapping times between the two groups. Significantly more errors were counted in the 2D versus 3D procedure, but all patients recovered without any complications related to intraoperative errors. On the questionnaires, all 8 surgeons desired a decreased scope diameter and multidirectional visual axis. Four surgeons (50%) experienced eyestrain and 1 experienced double vision. Conclusions: This study showed no reduction in surgical time with 3D versus 2D scope use. Three-dimensional vision decreased the technical errors in creating the wrap, but its clinical merit remains unclear. Decreasing the scope diameter and adding multidirectional vision capabilities are needed for the 3D scope to gain more widespread approval and use in pediatric surgery.

Current status of simulation-based training in pediatric surgery: A systematic review

BACKGROUND

Simulation based training enables pediatric surgical trainees to attain proficiency in surgical skills. This study aims to identify the currently available simulators for pediatric surgery, assess their validation and strength of evidence supporting each model.

METHODS

Both Medline and EMBASE were searched for English language articles either describing or validating simulation models for pediatric surgery. A level of evidence (LoE) followed by a level of recommendation (LoR) was assigned to each validation study and simulator, based on a modified Oxford Centre for Evidence-Based Medicine classification for educational studies.

RESULTS

Forty-nine articles were identified describing 44 training models and courses. Of these articles, 44 were validation studies. Face validity was evaluated by 20 studies, 28 for content, 24 demonstrated construct validity and 1 showed predictive validity. Of the validated models, 3 were given an LoR of 2, 21 an LoR of 3 and 12 an LoR of 4. None reached the highest LoR.

CONCLUSIONS

There are a growing number of simulators specific to pediatric surgery. However, these simulators have limited LoE and LoR in current studies. The lack of NoTSS training is also apparent. We advocate more randomized trials to validate these models, and attempts to determine predictive validity.

TYPE OF STUDY

Original / systematic review.

LEVEL OF EVIDENCE

1.

Evaluation of Surgical Devices Using an Artificial Pediatric Thoracic Model: A Comparison Between Robot-Assisted Thoracoscopic Suturing Versus Conventional Video-Assisted Thoracoscopic Suturing

BACKGROUND

Pediatric robot-assisted surgery is increasingly being performed, but it is difficult to perform this procedure in infants. A pediatric thoracoscopic model of a 1-year-old patient was developed in our previous study, and this model was used to evaluate the use of a surgical robot for infant surgery.

METHODS

Eight pediatric surgeons performed an intracorporeal suturing and knot-tying task using the da Vinci Xi Robotic Surgical System. The task completion time, number of needle manipulations, and force applied during suturing of the robot-assisted thoracoscopic surgery (RATS) group were compared with those of the video-assisted thoracoscopic surgery (VATS) group whose data had been collected from the same 8 surgeons in our previous study.

RESULTS

The RATS group showed a significantly shorter completion time than the VATS group in the knot-tying phase (P = .016) and in the total phase (P = .0078). The RATS group showed a significantly smaller number of manipulations than the VATS group in the total phase (P = .039). The RATS group showed a significantly smaller pushing force index than the VATS group in the suturing phase (P = .031), knot-tying phase (P = .031), and in the total phase (P = .031). A seventh rib in the model was dislocated in all RATS group cases.

CONCLUSIONS

The da Vinci Surgical System might be useful in infants because of fast movement and small pushing force. However, the robotic 8 mm instruments were too large for use in the thoracic cavity of the 1-year-old infant.

Automatically rating trainee skill at a pediatric laparoscopic suturing task

BACKGROUND

Minimally invasive surgeons must acquire complex technical skills while minimizing patient risk, a challenge that is magnified in pediatric surgery. Trainees need realistic practice with frequent detailed feedback, but human grading is tedious and subjective. We aim to validate a novel motion-tracking system and algorithms that automatically evaluate trainee performance of a pediatric laparoscopic suturing task.

METHODS

Subjects (n = 32) ranging from medical students to fellows performed two trials of intracorporeal suturing in a custom pediatric laparoscopic box trainer after watching a video of ideal performance. The motions of the tools and endoscope were recorded over time using a magnetic sensing system, and both tool grip angles were recorded using handle-mounted flex sensors. An expert rated the 63 trial videos on five domains from the Objective Structured Assessment of Technical Skill (OSATS), yielding summed scores from 5 to 20. Motion data from each trial were processed to calculate 280 features. We used regularized least squares regression to identify the most predictive features from different subsets of the motion data and then built six regression tree models that predict summed OSATS score. Model accuracy was evaluated via leave-one-subject-out cross-validation.

RESULTS

The model that used all sensor data streams performed best, achieving 71% accuracy at predicting summed scores within 2 points, 89% accuracy within 4, and a correlation of 0.85 with human ratings. 59% of the rounded average OSATS score predictions were perfect, and 100% were within 1 point. This model employed 87 features, including none based on completion time, 77 from tool tip motion, 3 from tool tip visibility, and 7 from grip angle.

CONCLUSIONS

Our novel hardware and software automatically rated previously unseen trials with summed OSATS scores that closely match human expert ratings. Such a system facilitates more feedback-intensive surgical training and may yield insights into the fundamental components of surgical skill.