Validity of a cross-specialty test in basic laparoscopic techniques (TABLT)

Laparoscopy training of novices with complex curved instruments using 2D- and 3D-visualization

PURPOSE

Beside many advantages, disadvantages such as reduced degrees of freedom and poorer depth perception are still apparent in laparoscopic surgery. 3D visualization and the development of complex instruments are intended to counteract the disadvantages. We want to find out whether the use of complex instruments and 3D visualization has an influence on the performance of novices.

METHODS

48 medical students with no experience in laparoscopic surgery or simulator-based laparoscopy training were included. They were randomized in four groups according to a stratification assessment. During a structured training period they completed the FLS-Tasks "PEG Transfer", "Pattern Cut" and "Intracorporeal Suture" and a transfer task based on these three. Two groups used conventional laparoscopic instruments with 3D or 2D visualization, two groups used complex curved instruments. The groups were compared in terms of their performance.

RESULTS

In 2D laparoscopy there was a better performance with straight instruments vs. curved instruments in PEG Transfer and Intracorporeal Suture. In the transfer task, fewer errors were made with straight instruments. In 2D vs. 3D laparoscopy when using complex curved instruments there was an advantage in Intracorporeal Suture and PEG Transfer for 3D visualization. Regarding the transfer exercise, a better performance was observed and fewer errors were made in 3D group.

CONCLUSION

We could show that learning laparoscopic techniques with complex curved instruments is more difficult with standard 2D visualization and can be overcome using 3D optics. The use of curved instruments under 3D vision seems to be advantageous when working on more difficult tasks.

Using Self-Rated Examinations to Ensure Competence in Laparoscopy

INTRODUCTION

Asking participants to rate their own performance during unsupervised training in laparoscopy is reliable and may be cost-effective. The objective of the study was to explore the reliability of self-rated examinations where participants rate their own performance and decide themselves when they have passed tasks in basic laparoscopic skills.

METHODS

This prospective observational study was conducted at the Copenhagen Academy for Medical Education and Simulation where simulation-based laparoscopic skill training is offered. Here, participants taking part in a basic laparoscopic skills course were asked to rate their own performance and decide when they had passed the Training and Assessment of Basic Laparoscopic Techniques test. To explore reliability, all examinations were video recorded and rated by a blinded rater after the end of the course.

RESULTS

Thirty-two surgical trainees participated in the course, and 28 completed the study. We found a high reliability when comparing self-rated scores and blinded ratings with an intraclass correlation coefficient of 0.89 ( P < 0.001); self-rated scores compared with blinded ratings were not significantly different (mean = 451 vs. 455, P = 0.28), and the participants did not underestimate nor overestimate their performance.

CONCLUSIONS

Ratings from self-rated examinations in a basic laparoscopic skills course are reliable, and participants neither underestimate nor overestimate their performance. Self-rated examinations may also be beneficial because they also can offer a cost-effective approach to assessment of surgical trainees.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.

Telementoring of in-home real-time laparoscopy using whatsapp messenger: An innovative teaching tool during the COVID-19 pandemic. A cohort study

Background

COVID-19 pandemic has decreased worldwide surgical activity, thus impairing the training of surgical residents. As a coping strategy, we opted for laparoscopic simulation with endotrainer; however, due to social distancing rules, a face-to-face tutoring training method is unfeasible. In this context telementoring using WhatsApp® Messenger (owned by Facebook Inc., Menlo Park, CA) with a Smartphone aims to train surgical residents in laparoscopy from their homes during the pandemic.

Methods

Trainers and surgical residents who were at home participated in this study. They used endotrainers with Smartphone cameras and were trained by video calls using WhatsApp. The baseline and final score of intracorporeal knotting was measured, according to the Fundamentals of Laparoscopic Surgery (FLS) methodology, measuring the difference in means with the paired t-test. Recommendations for virtual surgical tutoring were followed and user satisfaction was measured using the Likert scale.

Results

40 surgical residents participated. The average distance between the mentor and the trainee was 12.87 km. Average bandwidth was 32.5 Megabits per second (Mbits/s), latency was 424.5 ms (ms), good quality image with immediate feedback was achieved. The statistically significant difference between basal and final measurement was 23.7 points (p < 0.001). All participants found telementoring a good educational tool.

Conclusion

Smartphone telementoring using WhatsApp® Messenger is an effective educational tool for acquiring laparoscopic skills with a statistically significant difference. This type of training represents a new teaching resource in the light of the recommendations of social distancing during a pandemic.

Development and Validation of a Homemade, Low-Cost Laparoscopic Simulator for Resident Surgeons (LABOT)

Several studies have demonstrated that training with a laparoscopic simulator improves laparoscopic technical skills. We describe how to build a homemade, low-cost laparoscopic training simulator (LABOT) and its validation as a training instrument. First, sixty surgeons filled out a survey characterized by 12 closed-answer questions about realism, ergonomics, and usefulness for surgical training (global scores ranged from 1—very insufficient to 5—very good). The results of the questionnaires showed a mean (±SD) rating score of 4.18 ± 0.65 for all users. Then, 15 students (group S) and 15 residents (group R) completed 3 different tasks (T1, T2, T3), which were repeated twice to evaluate the execution time and the number of users’ procedural errors. For T1, the R group had a lower mean execution time and a lower rate of procedural errors than the S group; for T2, the R and S groups had a similar mean execution time, but the R group had a lower rate of errors; and for T3, the R and S groups had a similar mean execution time and rate of errors. On a second attempt, all the participants tended to improve their results in doing these surgical tasks; nevertheless, after subgroup analysis of the T1 results, the S group had a better improvement of both parameters. Our laparoscopic simulator is simple to build, low-cost, easy to use, and seems to be a suitable resource for improving laparoscopic skills. In the future, further studies should evaluate the potential of this laparoscopic box on long-term surgical training with more complex tasks and simulation attempts.

The Implementation Gap in Laparoscopic Simulation Training

BACKGROUND AND AIMS

Simulation-based training in laparoscopy can improve patient safety and efficiency of care, but it depends on how it is used. Research in medical education has moved from demonstrating transferability of simulation training to the operating room to how to best implement it. This study aims to investigate how simulation-based training in laparoscopy has been implemented Scandinavia.

MATERIAL AND METHODS

An online survey was sent out to medical doctors at surgical, gynecological, and urological departments at 138 hospitals in Denmark, Norway, and Sweden. The questionnaire included questions on respondents' baseline characteristics, opinions, access, and actual use of simulation-based training in laparoscopy.

RESULTS

In total, 738 respondents completed the survey. Of these, 636 (86.2%) of respondents agreed or strongly agreed that simulation-based training in laparoscopy should be mandatory. A total of 602 (81.6%) had access to simulation-based training in laparoscopy. Of the total 738 respondents, 141 (19.1%) were offered structured training courses, 129 (17.5%) were required to reach a predefined level of competency, and 66 (8.9%) had mandatory courses in laparoscopy. In all, 72 (9.8%) had never used simulation-based training in laparoscopy.

CONCLUSION

An implementation gap in laparoscopic simulation-based training still exists in Scandinavia. Simulation equipment is generally available, but there is a lack of structured simulation-based training.

A radiation-free mixed-reality training environment and assessment concept for C-arm-based surgery

PURPOSE

The discrepancy of continuously decreasing opportunities for clinical training and assessment and the increasing complexity of interventions in surgery has led to the development of different training and assessment options like anatomical models, computer-based simulators or cadaver trainings. However, trainees, following training, assessment and ultimately performing patient treatment, still face a steep learning curve.

METHODS

To address this problem for C-arm-based surgery, we introduce a realistic radiation-free simulation system that combines patient-based 3D printed anatomy and simulated X-ray imaging using a physical C-arm. To explore the fidelity and usefulness of the proposed mixed-reality system for training and assessment, we conducted a user study with six surgical experts performing a facet joint injection on the simulator.

RESULTS

In a technical evaluation, we show that our system simulates X-ray images accurately with an RMSE of 1.85 mm compared to real X-ray imaging. The participants expressed agreement with the overall realism of the simulation, the usefulness of the system for assessment and strong agreement with the usefulness of such a mixed-reality system for training of novices and experts. In a quantitative analysis, we furthermore evaluated the suitability of the system for the assessment of surgical skills and gather preliminary evidence for validity.

CONCLUSION

The proposed mixed-reality simulation system facilitates a transition to C-arm-based surgery and has the potential to complement or even replace large parts of cadaver training, to provide a safe assessment environment and to reduce the risk for errors when proceeding to patient treatment. We propose an assessment concept and outline the steps necessary to expand the system into a test instrument that provides reliable and justified assessments scores indicative of surgical proficiency with sufficient evidence for validity.

Contrasting groups' standard setting for consequences analysis in validity studies: reporting considerations

Background

The contrasting groups’ standard setting method is commonly used for consequences analysis in validity studies for performance in medicine and surgery. The method identifies a pass/fail cut-off score, from which it is possible to determine false positives and false negatives based on observed numbers in each group. Since groups in validity studies are often small, e.g., due to a limited number of experts, these analyses are sensitive to outliers on the normal distribution curve.

Methods

We propose that these shortcomings can be addressed in a simple manner using the cumulative distribution function.

Results

We demonstrate considerable absolute differences between the observed false positives/negatives and the theoretical false positives/negatives. In addition, several important examples are given.

Conclusions

We propose that a better reporting strategy is to report theoretical false positives and false negatives together with the observed false positives and negatives, and we have developed an Excel sheet to facilitate such calculations.

Trial registration

Not relevant.

Electronic supplementary material

The online version of this article (10.1186/s41077-018-0064-7) contains supplementary material, which is available to authorized users.

Development of a reliable simulation-based test for diagnostic abdominal ultrasound with a pass/fail standard usable for mastery learning

BACKGROUND

This study aimed to develop a test with validity evidence for abdominal diagnostic ultrasound with a pass/fail-standard to facilitate mastery learning.

METHOD

The simulator had 150 real-life patient abdominal scans of which 15 cases with 44 findings were selected, representing level 1 from The European Federation of Societies for Ultrasound in Medicine and Biology. Four groups of experience levels were constructed: Novices (medical students), trainees (first-year radiology residents), intermediates (third- to fourth-year radiology residents) and advanced (physicians with ultrasound fellowship). Participants were tested in a standardized setup and scored by two blinded reviewers prior to an item analysis.

RESULTS

The item analysis excluded 14 diagnoses. Both internal consistency (Cronbach's alpha 0.96) and inter-rater reliability (0.99) were good and there were statistically significant differences (p < 0.001) between all four groups, except the intermediate and advanced groups (p = 1.0). There was a statistically significant correlation between experience and test scores (Pearson's r = 0.82, p < 0.001). The pass/fail-standard failed all novices (no false positives) and passed all advanced (no false negatives). All intermediate participants and six out of 14 trainees passed.

CONCLUSION

We developed a test for diagnostic abdominal ultrasound with solid validity evidence and a pass/fail-standard without any false-positive or false-negative scores.

KEY POINTS

• Ultrasound training can benefit from competency-based education based on reliable tests. • This simulation-based test can differentiate between competency levels of ultrasound examiners. • This test is suitable for competency-based education, e.g. mastery learning. • We provide a pass/fail standard without false-negative or false-positive scores.

Ensuring Competency of Novice Laparoscopic Surgeons-Exploring Standard Setting Methods and their Consequences

OBJECTIVE

Simulation-based assessment tools have been developed to allow for proficiency-based simulator training in laparoscopy. However, few studies have examined the consequences of different standard setting methods or examined what level of proficiency is considered adequate for trainees. The objectives of the present study were to explore the consequences of different standard setting methods and to examine the proficiency level that surgical trainees are expected to reach, before performing supervised surgery on patients.

DESIGN

Study participants undertook the Training and Assessment of Basic Laparoscopic Techniques test. The tests were video-recorded and rated using a simple scoring system based on number of errors and time. Participants were then asked to assess how high a score a novice should reach before performing supervised surgery on a patient. We then compared 3 methods of standard setting: expert performance level, contrasting groups method, and a modified Angoff method.

SETTING

The study was conducted at the Copenhagen Academy for Medical Education and Simulation. The academy provides surgical simulation training in laparoscopy for trainees at the hospitals in the Capital Region and the Zealand Region of Denmark.

PARTICIPANTS

Participants were recruited among surgical trainees in their first year of specialty training from surgery, gynecology, and urology departments. A total of 40 participants were included and completed the trial.

RESULTS

The different standard setting methods resulted in different pass/fail levels. At the expert performance level, the pass/fail level was 474 points-the contrasting groups method resulted in 358 points and the modified Angoff method resulted in 311 points among experienced surgeons, and 386 points among trainees. The different proficiency levels resulted in a failure rate of 0% to 50% of experienced surgeons and a pass rate of 0% to 25% of novices. Novice laparoscopic surgeons set a higher pass/fail level than experienced surgeons did (p = 0.008).

CONCLUSION

Required proficiency levels varies depending on the standard setting method used, which highlights the importance of using an established standard setting method to set the pass/fail level.

Systematic review to establish absolute standards for technical performance in surgery

BACKGROUND

Standard setting allows educators to create benchmarks that distinguish between those who pass and those who fail an assessment. It can also be used to create standards in clinical and simulated procedural skill. The objective of this review was to perform a systematic review of the literature using absolute standard-setting methodology to create benchmarks in technical performance.

METHODS

A systematic review was conducted by searching MEDLINE, Embase, PsycINFO and the Cochrane Database of Systematic Reviews. Abstracts of retrieved studies were reviewed and those meeting the inclusion criteria were selected for full-text review. The quality of evidence presented in the included studies was assessed using the Medical Education Research Study Quality Instrument (MERSQI), where a score of 14 or more of 18 indicates high-quality evidence.

RESULTS

Of 1809 studies identified, 37 used standard-setting methodology for assessment of procedural skill. Of these, 24 used participant-centred and 13 employed item-centred methods. Thirty studies took place in a simulated environment, and seven in a clinical setting. The included studies assessed residents (26 of 37), fellows (6 of 37) and staff physicians (17 of 37). Seventeen articles achieved a MERSQI score of 14 or more of 18, whereas 20 did not meet this mark.

CONCLUSION

Absolute standard-setting methodologies can be used to establish cut-offs for procedural skill assessments.

Take-home training in a simulation-based laparoscopy course

BACKGROUND

Simulation training can prepare trainees for clinical practice in laparoscopic surgery. Training on box trainers allows for simulation training at home, which studies have shown to be a feasible method of training. However, little research has been conducted into how to make it a more efficient method of training. Our aim was to investigate how box trainers are used in take-home training to help guide the design of take-home training courses.

METHODS

This study was designed using a mixed methods approach. Junior doctors participating in a laparoscopy curriculum, which included practising at home on box trainers, were invited. Quantitative data on training patterns was collected from logbooks. Qualitative data on the use of box trainers was retrieved from focus groups and individual interviews.

RESULTS

From logbooks, we found that 14 out of 18 junior doctors mixed their training modalities, and four practised first on box trainers then on virtual reality simulators. Twelve practised only at home, while five practised at both places and one practised solely at the simulation centre. After a delayed start, most practised for some time, then had a period without training and then started training again towards the end of the course. We found that the themes of the interviews were: training method, training pattern, feedback and self-regulation. Participants identified the lack of feedback as challenging but described how self-rating provided direction during unsupervised training. Mandatory training elements affected when and how much participants practised.

CONCLUSIONS

When participants practised at home, they took an individualised approach to training. They mixed their training at home with training at the simulation centre. Participants practised at the beginning and towards the end of the course. Self-rating helped to guide unsupervised training where feedback was not accessible. Curricular requirements and testing determined when and how much participants practised.

Laser visual guidance versus two-dimensional vision in laparoscopy: a randomized trial

Background

During laparoscopy, the surgeon’s loss of depth perception and spatial orientation is problematic. Laser visual guidance (LVG) is an innovative technology that improves depth perception to enhance the visual field. In this trial, we examined the effect of LVG on surgical novices’ motor skills, quality of task performance, and cognitive workload.

Methods

We designed a randomized controlled trial following the CONSORT statement. Thirty-two surgical novices completed the Training and Assessment of Basic Laparoscopic Techniques (TABLT) test. The first attempt allowed participants to familiarize themselves with the exercises. We then randomized the participants, and they completed a test session using either LVG or conventional two-dimensional vision.

Results

We found no significant difference between using the LVG tool and conventional 2D vision; however, both the mean completion time and movements used were less in the LVG group: Mean time used in the LVG group was 1288 s (95 % CI 1188–1388) versus 1354 s (95 % CI 1190–1518) (p = 0.45); mean angular path length used in the LVG group was 24,049° (95 % CI 20,761–27,336) versus 26,014° (95 % CI 22,059–29,970) (p = 0.42); mean path length in the LVG group was 4560 cm (95 % CI 3971–5,149 cm) versus 5062 cm (95 % CI 4328–5797), (p = 0.26). Moreover, the mean TABLT performance score was higher in the LVG group compared with the 2D group, although not significant: 379 (95 % CI 352–405) versus 338 (95 % CI 288–387) (p = 0.14). No significant difference was found between the groups’ cognitive workloads.

Conclusion

We found no significant improvement of laparoscopic motor skills when using LVG, although a tendency toward improved performance was seen. LVG could have the potential to help novice surgeons acquire basic laparoscopic; however, further development of the concept and validation is needed to confirm this.