The bibliometric analysis of extended reality in surgical training: Global and Chinese perspective

Objectives

The prospect of extended reality (XR) being integrated with surgical training curriculum has attracted scholars. However, there is a lack of bibliometric analysis to help them better understand this field. Our aim is to analyze relevant literature focusing on development trajectory and research directions since the 21st century to provide valuable insights.

Methods

Papers were retrieved from the Web of Science Core Collection. Microsoft Excel, VOSviewer, and CiteSpace were used for bibliometric analysis.

Results

Of the 3337 papers published worldwide, China contributed 204, ranking fifth. The world's enthusiasm for this field has been growing since 2000, whereas China has been gradually entering since 2001. Although China had a late start, its growth has accelerated since around 2016 due to the reform of the medical postgraduate education system and the rapid development of Chinese information technology, despite no research explosive period has been yet noted. International institutions, notably the University of Toronto, worked closely with others, while Chinese institutions lacked of international and domestic cooperation. Sixteen stable cooperation clusters of international scholars were formed, while the collaboration between Chinese scholars was not yet stable. XR has been primarily applied in orthopedic surgery, cataract surgery, laparoscopic training and intraoperative use in neurosurgery worldwide.

Conclusions

There is strong enthusiasm and cooperation in the international research on the XR-based surgical training. Chinese scholars are making steady progress and have great potential in this area. There has not been noted an explosive research phase yet in the Chinese pace. The research on several surgical specialties has been summarized at the very first time. AR will gradually to be more involved and take important role of the research.

Fluoroscopic image-based behavior analysis can objectively explain subjective expert assessment of wire navigation skill

Psychomotor skill and decision-making efficiency in surgical wire navigation can be objectively evaluated by analysis of intraoperative fluoroscopic image sequences. Prior work suggests that such image-based behavior analysis of operating room (OR) performance can predict performer experience level (R2  = 0.62) and agree with expert opinion (the current standard) on the quality of a final implant construct (R2  = 0.59). However, it is unclear how objective image-based evaluation compares with expert assessments for entire technical OR performances. This study examines the relationships between three key variables: (1) objective image-based criteria, (2) expert opinions, and (3) performing surgeon experience level. A paired-comparison survey of seven experts, based upon eight OR fluoroscopic wire navigation image sequences, shows that the experts' preferences are best explained by objective metrics that reflect psychomotor and decision-making behaviors which are counter-productive to successful implant placement, like image count (R2  = 0.83) and behavior tally (R2  = 0.74). One such behavior, adjustments away from goal, uniquely correlated well with all three key variables: a fluoroscopic image-based analysis composite score (R2  = 0.40), expert consensus (R2  = 0.76), and performer experience (R2  = 0.41). These results confirm that experts view less efficient technical behavior as indicative of lesser technical proficiency. While expert assessments of technical skill were reliable and consistent, neither individual nor consensus expert opinion appears to correlate with performer experience (R2  = 0.11).

Laparoscopic cholecystectomy critical view of safety (LC-CVS): a multi-national validation study of an objective, procedure-specific assessment using video-based assessment (VBA)

BACKGROUND

A novel 6-item objective, procedure-specific assessment for laparoscopic cholecystectomy incorporating the critical view of safety (LC-CVS OPSA) was developed to support trainee formative and summative assessments. The LC-CVS OPSA included two retraction items (fundus and infundibulum retraction) and four CVS items (hepatocystic triangle visualization, gallbladder-liver separation, cystic artery identification, and cystic duct identification). The scoring rubric for retraction consisted of poor (frequently outside of defined range), adequate (minimally outside of defined range) and excellent (consistently inside defined range) and for CVS items were "poor-unsafe", "adequate-safe", or "excellent-safe".

METHODS

A multi-national consortium of 12 expert LC surgeons applied the OPSA-LC CVS to 35 unique LC videos and one duplicate video. Primary outcome measure was inter-rater reliability as measured by Gwet's AC2, a weighted measure that adjusts for scales with high probability of random agreement. Analysis of the inter-rater reliability was conducted on a collapsed dichotomous scoring rubric of "poor-unsafe" vs. "adequate/excellent-safe".

RESULTS

Inter-rater reliability was high for all six items ranging from 0.76 (hepatocystic triangle visualization) to 0.86 (cystic duct identification). Intra-rater reliability for the single duplicate video was substantially higher across the six items ranging from 0.91 to 1.00.

CONCLUSIONS

The novel 6-item OPSA LC CVS demonstrated high inter-rater reliability when tested with a multi-national consortium of LC expert surgeons. This brief instrument focused on safe surgical practice was designed to support the implementation of entrustable professional activities into busy surgical training programs. Instrument use coupled with video-based assessments creates novel datasets with the potential for artificial intelligence development including computer vision to drive assessment automation.

Telestration with augmented reality improves surgical performance through gaze guidance

BACKGROUND

In minimally invasive surgery (MIS), trainees need to learn how to interpret the operative field displayed on the laparoscopic screen. Experts currently guide trainees mainly verbally during laparoscopic procedures. A newly developed telestration system with augmented reality (iSurgeon) allows the instructor to display hand gestures in real-time on the laparoscopic screen in augmented reality to provide visual expert guidance (telestration). This study analysed the effect of telestration guided instructions on gaze behaviour during MIS training.

METHODS

In a randomized-controlled crossover study, 40 MIS naive medical students performed 8 laparoscopic tasks with telestration or with verbal instructions only. Pupil Core eye-tracking glasses were used to capture the instructor's and trainees' gazes. Gaze behaviour measures for tasks 1-7 were gaze latency, gaze convergence and collaborative gaze convergence. Performance measures included the number of errors in tasks 1-7 and trainee's ratings in structured and standardized performance scores in task 8 (ex vivo porcine laparoscopic cholecystectomy).

RESULTS

There was a significant improvement 1-7 on gaze latency [F(1,39) = 762.5, p < 0.01, ηp2 = 0.95], gaze convergence [F(1,39) = 482.8, p < 0.01, ηp2 = 0.93] and collaborative gaze convergence [F(1,39) = 408.4, p < 0.01, ηp2 = 0.91] upon instruction with iSurgeon. The number of errors was significantly lower in tasks 1-7 (0.18 ± 0.56 vs. 1.94 ± 1.80, p < 0.01) and the score ratings for laparoscopic cholecystectomy were significantly higher with telestration (global OSATS: 29 ± 2.5 vs. 25 ± 5.5, p < 0.01; task-specific OSATS: 60 ± 3 vs. 50 ± 6, p < 0.01).

CONCLUSIONS

Telestration with augmented reality successfully improved surgical performance. The trainee's gaze behaviour was improved by reducing the time from instruction to fixation on targets and leading to a higher convergence of the instructor's and the trainee's gazes. Also, the convergence of trainee's gaze and target areas increased with telestration. This confirms augmented reality-based telestration works by means of gaze guidance in MIS and could be used to improve training outcomes.

Video-based learning of coping strategies for common errors improves laparoscopy training-a randomized study

AIMS

The aim of this study was to investigate whether shifting the focus to solution orientation and developing coping strategies for common errors could increase the efficiency of laparoscopic training and influence learning motivation. The concept of coping has been particularly defined by the psychologist Richard Lazarus [Lazarus and Folkman in Stress, appraisal, and coping, Springer publishing company, New York, 1984]. Based on this model, we examined the use of observational learning with a coping model for its effectiveness as a basic teaching model in laparoscopic training.

METHODS

55 laparoscopically naive medical students learned a standardized laparoscopic knot tying technique with video-based instructions. The control group was only offered a mastery video that showed the ideal technique and was free from mistakes. The intervention group was instructed on active error analysis and watched freely selectable videos of common errors including solution strategies (coping model) in addition to the mastery videos.

RESULTS

There was no statistically significant difference between the intervention and control groups for number of knot tying attempts until proficiency was reached (18.8 ± 5.5 vs. 21.3 ± 6.5, p = 0.142). However, there was a significantly higher fraction of knots achieving technical proficiency in the intervention group after first use of the coping model (0.7 ± 0.1 vs. 0.6 ± 0.2, p = 0.026). Additionally, the proportion of blinded attempts that met the criteria for technical proficiency was significantly higher for the intervention group at 60.9% vs. 38.0% in control group (p = 0.021). The motivational subscore "interest" of the validated score on current motivation (QCM) was significantly higher for the intervention group (p = 0.032), as well as subjective learning benefit (p = 0.002) and error awareness (p < 0.001).

CONCLUSION

Using video-based learning of coping strategies for common errors improves learning motivation and understanding of the technique with a significant difference in its qualitative implementation in laparoscopy training. The ability to think in a solution-oriented, independent way is necessary in surgery in order to recognize and adequately deal with technical difficulties and complications.

Serious gaming and virtual reality in the multimodal training of laparoscopic inguinal hernia repair: a randomized crossover study

BACKGROUND

The aim of this study was to assess the transferability of surgical skills for the laparoscopic hernia module between the serious game Touch Surgery™ (TS) and the virtual reality (VR) trainer Lap Mentor™. Furthermore, this study aimed to collect validity evidence and to discuss "sources of validity evidence" for the findings using the laparoscopic inguinal hernia module on TS.

METHODS

In a randomized crossover study, medical students (n = 40) in their clinical years performed laparoscopic inguinal hernia modules on TS and the VR trainer. TS group started with "Laparoscopic Inguinal Hernia Module" on TS (phase 1: Preparation, phase 2: Port Placement and Hernia Repair), performed the module first in training, then in test mode until proficiency was reached. VR group started with "Inguinal Hernia Module" on the VR trainer (task 1: Anatomy Identification, task 2: Incision and Dissection) and also performed the module until proficiency. Once proficiency reached in the first modality, the groups performed the other training modality until reaching proficiency. Primary endpoint was the number of attempts needed to achieve proficiency for each group for each task/phase.

RESULTS

Students starting with TS needed significantly less attempts to reach proficiency for task 1 on the VR trainer than students who started with the VR trainer (TS = 2.7 ± 0.6 vs. VR = 3.2 ± 0.7; p = 0.028). No significant differences for task 2 were observed between groups (TS = 2.3 ± 1.1 vs. VR = 2.1 ± 0.8; p = 0.524). For both phases on TS, no significant skill transfer from the VR trainer to TS was observed. Aspects of validity evidence for the module on TS were collected.

CONCLUSION

The results show that TS brought additional benefit to improve performances on the VR trainer for task 1 but not for task 2. Skill transfer from the VR trainer to TS could not be shown. VR and TS should thus be used in combination with TS first in multimodal training to ensure optimal training conditions.

A novel simulator-based checklist for evaluating residents' competence in cerebral angiography in China

Background

Nowadays, with the fast-increasing demand for neuro-endovascular therapy, surgeons in this field are in urgent need. Unfortunately, there is still no formal skill assessment in neuro-endovascular therapy in China.

Methods

We used a Delphi method to design a newly objective checklist for standards of cerebrovascular angiography in China and evaluated its validity and reliability. A total of 19 neuro-residents with no interventional experience and 19 neuro-endovascular surgeons from two centers (Guangzhou and Tianjin) were recruited; they were divided into two groups: residents and surgeons. Residents completed a simulation-based cerebrovascular angiography operation training before assessment. Assessments were under live and video record forms with two tools: the existing global rating scale (GRS) of endovascular performance and the new checklist.

Results

The average scores of residents were significantly increased after training in two centers (p < 0.05). There is good consistency between GRS and the checklist (p = 0.856). Intra-rater reliability (Spearman's rho) of the checklist was >0.9, and the same result was also observed in raters between different centers and different assessment forms (p < 0.001, rho > 0.9). The reliability of the checklist was higher than that of the GRS (Kendall's harmonious coefficient is 0.849, while GRS is 0.684).

Conclusion

The newly developed checklist appears reliable and valid for evaluating the technical performance of cerebral angiography and differentiating between trained and untrained trainees' performance well. For its efficiency, our method has been proven to be a feasible tool for resident angiography examination in certification nationwide.

Development of biotissue training models for anastomotic suturing in pancreatic surgery

BACKGROUND

Anastomotic suturing is the Achilles heel of pancreatic surgery. Especially in laparoscopic and robotically assisted surgery, the pancreatic anastomosis should first be trained outside the operating room. Realistic training models are therefore needed.

METHODS

Models of the pancreas, small bowel, stomach, bile duct, and a realistic training torso were developed for training of anastomoses in pancreatic surgery. Pancreas models with soft and hard textures, small and large ducts were incrementally developed and evaluated. Experienced pancreatic surgeons (n = 44) evaluated haptic realism, rigidity, fragility of tissues, and realism of suturing and knot tying.

RESULTS

In the iterative development process the pancreas models showed high haptic realism and highest realism in suturing (4.6 ± 0.7 and 4.9 ± 0.5 on 1-5 Likert scale, soft pancreas). The small bowel model showed highest haptic realism (4.8 ± 0.4) and optimal wall thickness (0.1 ± 0.4 on -2 to +2 Likert scale) and suturing behavior (0.1 ± 0.4). The bile duct models showed optimal wall thickness (0.3 ± 0.8 and 0.4 ± 0.8 on -2 to +2 Likert scale) and optimal tissue fragility (0 ± 0.9 and 0.3 ± 0.7).

CONCLUSION

The biotissue training models showed high haptic realism and realistic suturing behavior. They are suitable for realistic training of anastomoses in pancreatic surgery which may improve patient outcomes.

Telestration with augmented reality for visual presentation of intraoperative target structures in minimally invasive surgery: a randomized controlled study

AIMS

In minimally invasive surgery (MIS), intraoperative guidance has been limited to verbal communication without direct visual guidance. Communication issues and mistaken instructions in training procedures can hinder correct identification of anatomical structures on the MIS screen. The iSurgeon system was developed to provide visual guidance in the operating room by telestration with augmented reality (AR).

METHODS

Laparoscopic novices (n = 60) were randomized in two groups in a cross-over design: group 1 trained only with verbal guidance first and then with additional telestration with AR on the operative screen and vice versa for group 2. Training consisted of laparoscopic basic training and subsequently a specifically designed training course, including a porcine laparoscopic cholecystectomy (LC). Outcome included time needed for training, performance with Global Operative Assessment of Laparoscopic Skills (GOALS), and Objective Structured Assessment of Technical Skills (OSATS) score for LC, complications, and subjective workload (NASA-TLX questionnaire).

RESULTS

Telestration with AR led to significantly faster total training time (1163 ± 275 vs. 1658 ± 375 s, p < 0.001) and reduced error rates. LC on a porcine liver was performed significantly better (GOALS 21 ± 5 vs. 18 ± 4, p < 0.007 and OSATS 67 ± 11 vs. 61 ± 8, p < 0.015) and with less complications (13.3% vs. 40%, p < 0.020) with AR. Subjective workload and stress were significantly reduced during training with AR (33.6 ± 12.0 vs. 30.6 ± 12.9, p < 0.022).

CONCLUSION

Telestration with AR improves training success and safety in MIS. The next step will be the clinical application of telestration with AR and the development of a mobile version for remote guidance.

Eleven ways to get a grip on the implementation of remote administration of high-stakes assessments

The COVID-19 pandemic rushed licensure and certification institutions, as well as many university programs, to integrate Information and Communication Technologies (ICTs) in their practices to allow for remote administrations of their exams independent of distancing measures. The Black Ice covered in this manuscript is the integration of ICTs to allow remote administration of high-stakes assessments in terms of its development, administration, and monitoring with the aim to promote the validity of score interpretation.

Statistical models to preoperatively predict operative difficulty in laparoscopic cholecystectomy: A systematic review

BACKGROUND

Laparoscopic cholecystectomy operative difficulty is highly variable and influences outcomes. This systematic review analyzes the performance and clinical value of statistical models to preoperatively predict laparoscopic cholecystectomy operative difficulty.

METHODS

PRISMA guidelines were followed. PubMed, Embase, and the Cochrane Library were searched until June 2020. Primary studies developing or validating preoperative models predicting laparoscopic cholecystectomy operative difficulty in cohorts of >100 patients were included. Studies not reporting performance metrics or enough information for clinical implementation were excluded. Data were extracted according to CHARMS, and study quality was assessed using the PROBAST tool.

RESULTS

In total, 2,654 articles were identified, and 22 met eligibility criteria. Eighteen were model development, whereas 4 were validation studies. Eighteen studies were at high risk of bias. However, 11 studies showed low concern for applicability. Identified models predict 9 definitions of laparoscopic cholecystectomy operative difficulty, the most common being conversion to open surgery and operating time. The most validated models predict an intraoperative difficulty scale and procedures >90 minutes with an area under the curve of >0.70 and >0.76, respectively. Commonly used predictors include demographic variables such as age and gender (9/18 models) and ultrasound findings such as gallbladder wall thickness (11/18). Clinical implementation was never studied.

CONCLUSION

There is a longstanding interest in estimating laparoscopic cholecystectomy operative difficulty. Models to preoperatively predict laparoscopic cholecystectomy operative difficulty have generally good performance and seem applicable. However, an unambiguous definition of operative difficulty, validations, and clinical studies are needed to implement patients' stratification in laparoscopic cholecystectomy.

Procedure-based assessment for laparoscopic cholecystectomy can replace global rating scales

INTRODUCTION

Global rating scales (GRSs) such as the Objective Structured Assessment of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Surgery (GOALS) are assessment methods for surgical procedures. The aim of this study was to establish construct validity of Procedure-Based Assessment (PBA) and to compare PBA with GRSs for laparoscopic cholecystectomy.

MATERIAL AND METHODS

OSATS and GOALS GRSs were compared with PBA in their ability to discriminate between levels of performance between trainees who can perform the procedure independently and those who cannot. Three groups were formed based on the number of procedures performed by the trainee: novice (1-10), intermediate (11-20) and experienced (>20). Differences between groups were assessed using the Kruskal-Wallis and Mann-Whitney U tests.

RESULTS

Increasing experience correlated significantly with higher GRSs and PBA scores (all p < .001). Scores of novice and intermediate groups overlapped substantially on the OSATS (p = .1) and GOALS (p = .1), while the PBA discriminated between these groups (p = .03). The median score in the experienced group was higher with less dispersion for PBA (97.2[85.3-100]) compared to OSATS (82.1[60.7-100]) and GOALS (80[60-100]).

CONCLUSION

For assessing skill level or the capability of performing a laparoscopic cholecystectomy independently, PBA has a higher discriminative ability compared to the GRSs.

Evaluating the Effect of Surgical Skill on Outcomes for Laparoscopic Sleeve Gastrectomy: A Video-based Study

BACKGROUND

Prior studies have demonstrated a correlation between surgical skill and complication rates after laparoscopic Roux-en-Y gastric bypass. However, the impact of surgical skill on a similar but less technically challenging procedure such as sleeve gastrectomy (SG) is unknown.

METHODS

Practicing bariatric surgeons (n = 25) participating in a statewide quality improvement collaborative submitted an unedited deidentified video of a representative laparoscopic SG. Videos were obtained between 2015 and 2016 and were rated by bariatric surgeons in a blinded fashion using a validated instrument that assesses surgical skill. Overall scores were based on a 5-point Likert scale with 5 representing a "master surgeon" and 1 representing a "surgeon-in-training." Risk-adjusted 30-day complication rates, 1-year weight loss among cases performed during the study period, and operative technique were compared between surgeons rated in the top and bottom quartiles according to skill.

RESULTS

Surgeon ratings for skill varied between 2.73 and 4.60. Ratings for skill did not correlate with overall 30-day risk-adjusted complication rates (Pearson correlation coefficient, 0.213, P = 0.303). However, surgeons with higher skill ratings had lower rates of specific surgical complications, including postoperative obstruction (0.13% vs 0.3%, P = 0.017), hemorrhage (0.85% vs 1.27%, P = 0.005), and reoperation (0.24% vs 0.92%, P < 0.0001). Surgeons ranked in the top quartile for skill had faster operating times for SG (59.0 vs 82.1 min, P < 0.0001) and higher annual case volumes for both SG and any bariatric procedure (224.3 cases/yr vs 73.4 cases/yr, P = 0.009 and 244.9 cases/yr and 93.9 cases/yr, P = 0.009) when compared with surgeons in the bottom quartile. When comparing operative technique, top rated surgeons were noted to have a higher likelihood of using buttressing (83.3% vs 0%, P = 0.0041) and intraoperative endoscopy (83.3% vs 0%, P = 0.0041).

CONCLUSIONS

Peer ratings for surgical skill varied for laparoscopic sleeve gastrectomy but did not have a significant impact on overall complication rates. Top rated surgeons had lower rates of obstruction, hemorrhage, and reoperation; however, severe morbidity remained extremely low among all surgeons.

Robotic-assisted cholecystectomy is superior to laparoscopic cholecystectomy in the initial training for surgical novices in an ex vivo porcine model: a randomized crossover study

Background

Robotic-assisted surgery (RAS) potentially reduces workload and shortens the surgical learning curve compared to conventional laparoscopy (CL). The present study aimed to compare robotic-assisted cholecystectomy (RAC) to laparoscopic cholecystectomy (LC) in the initial learning phase for novices.

Methods

In a randomized crossover study, medical students (n = 40) in their clinical years performed both LC and RAC on a cadaveric porcine model. After standardized instructions and basic skill training, group 1 started with RAC and then performed LC, while group 2 started with LC and then performed RAC. The primary endpoint was surgical performance measured with Objective Structured Assessment of Technical Skills (OSATS) score, secondary endpoints included operating time, complications (liver damage, gallbladder perforations, vessel damage), force applied to tissue, and subjective workload assessment.

Results

Surgical performance was better for RAC than for LC for total OSATS (RAC = 77.4 ± 7.9 vs. LC = 73.8 ± 9.4; p = 0.025, global OSATS (RAC = 27.2 ± 1.0 vs. LC = 26.5 ± 1.6; p = 0.012, and task specific OSATS score (RAC = 50.5 ± 7.5 vs. LC = 47.1 ± 8.5; p = 0.037). There were less complications with RAC than with LC (10 (25.6%) vs. 26 (65.0%), p = 0.006) but no difference in operating times (RAC = 77.0 ± 15.3 vs. LC = 75.5 ± 15.3 min; p = 0.517). Force applied to tissue was similar. Students found RAC less physical demanding and less frustrating than LC.

Conclusions

Novices performed their first cholecystectomies with better performance and less complications with RAS than with CL, while operating time showed no differences. Students perceived less subjective workload for RAS than for CL. Unlike our expectations, the lack of haptic feedback on the robotic system did not lead to higher force application during RAC than LC and did not increase tissue damage. These results show potential advantages for RAS over CL for surgical novices while performing their first RAC and LC using an ex vivo cadaveric porcine model.

Registration number

researchregistry6029

Graphic abstract

Surgical Performance Is Not Negatively Impacted by Wearing a Commercial Full-Face Mask with Ad Hoc 3D-Printed Filter Connection as a Substitute for Personal Protective Equipment during the COVID-19 Pandemic: A Randomized Controlled Cross-Over Trial

(1) Background: During the COVID-19 pandemic, shortages in the supply of personal protective equipment (PPE) have become apparent. The idea of using commonly available full-face diving (FFD) masks as a temporary solution was quickly spread across social media. However, it was unknown whether an FFD mask would considerably impair complex surgical tasks. Thus, we aimed to assess laparoscopic surgical performance while wearing an FFD mask as PPE. (2) Methods: In a randomized-controlled cross-over trial, 40 laparoscopically naive medical students performed laparoscopic procedures while wearing an FFD mask with ad hoc 3D-printed connections to heat and moisture exchange (HME) filters vs. wearing a common surgical face mask. The performance was evaluated using global and specific Objective Structured Assessment of Technical Skills (OSATS) checklists for suturing and cholecystectomy. (3) Results: For the laparoscopic cholecystectomy, both global OSATS scores and specific OSATS scores for the quality of procedure were similar (Group 1: 25 ± 4.3 and 45.7 ± 12.9, p = 0.485, vs. Group 2: 24.1 ± 3.7 and 43.3 ± 7.6, p = 0.485). For the laparoscopic suturing task, the FFD mask group needed similar times to the surgical mask group (3009 ± 1694 s vs. 2443 ± 949 s; p = 0.200). Some participants reported impaired verbal communication while wearing the FFD mask, as it muffled the sound of speech, as well as discomfort in breathing. (4) Conclusions: FFD masks do not affect the quality of laparoscopic surgical performance, despite being uncomfortable, and may therefore be used as a substitute for conventional PPE in times of shortage—i.e., the global COVID-19 pandemic.

Assessment methods and the validity and reliability of measurement tools in online objective structured clinical examinations: a systematic scoping review

The coronavirus disease 2019 (COVID-19) pandemic has required educators to adapt the in-person objective structured clinical examination (OSCE) to online settings in order for it to remain a critical component of the multifaceted assessment of a student’s competency. This systematic scoping review aimed to summarize the assessment methods and validity and reliability of the measurement tools used in current online OSCE (hereafter, referred to as teleOSCE) approaches. A comprehensive literature review was undertaken following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. Articles were eligible if they reported any form of performance assessment, in any field of healthcare, delivered in an online format. Two reviewers independently screened the results and analyzed relevant studies. Eleven articles were included in the analysis. Pre-recorded videos were used in 3 studies, while observations by remote examiners through an online platform were used in 7 studies. Acceptability as perceived by students was reported in 2 studies. This systematic scoping review identified several insights garnered from implementing teleOSCEs, the components transferable from telemedicine, and the need for systemic research to establish the ideal teleOSCE framework. TeleOSCEs may be able to improve the accessibility and reproducibility of clinical assessments and equip students with the requisite skills to effectively practice telemedicine in the future.

Efficacy of goal-directed minimally invasive surgery simulation training with the Lübeck Toolbox-Curriculum prior to first operations on patients: Study protocol for a multi-centre randomized controlled validation trial (NOVICE)

•

Laparoscopic surgery (MIS) requires additional psychomotoric skills (basic skills)

•

Acquisition of MIS basic skills by the video box trainer Lübecker Toolbox.

•

Simulation of laparoscopic demands outside the operation room.

•

MIS training outside the operation room prior to first operations on patients.

•

Implementation of MIS training into the surgical residency curriculum.

Background

Minimally invasive surgery (MIS) procedures require special psychomotoric skills. Learning of these MIS basic skills is often performed in the operating room (OR). This is economically inefficient and could be improved in terms of patient safety. Against the background of this problem, various MIS simulators have been developed to train MIS basic skills outside the OR. Aim of this study is to evaluate to what extent MIS training programs and simulators improve the residents’ skills in performing their first MIS procedures on patients.

Method

The current multicentric RCT will be performed with surgical residents without prior active experience in MIS (n = 14). After the participants have completed their first laparoscopic cholecystectomy as baseline evaluation (CHE I), they will be randomized into two groups: 1) The intervention group will perform the Lübeck Toolbox curriculum, whereas 2) the control group will not undergo any MIS training. After 6 weeks, both groups will perform the second laparoscopic CHE (CHE II). Changes or improvements in operative performance (between CHE I and CHE II) will be analyzed and evaluated according to the Global Operative Assessment of Laparoscopic Skill (GOALS) Score (primary endpoint).

Discussion

The multicentric randomized controlled trial will help to determine the value of MIS training outside the operation room. Proof of effectiveness in practice transfer could be of considerable relevance with regard to an integration of MIS training programs into surgical education.

How do visual-spatial and psychomotor abilities influence clinical performance in periodontal plastic surgery?

AIM

We want to evaluate the relationship of self-assessed experience and proficiency, manual dexterity and visual-spatial ability with surgical performance.

MATERIAL AND METHODS

A total of 26 professionals were included in the study which consisted of four parts: (a) self-assessment by a questionnaire regarding proficiency and experience, (b) evaluation of visual-spatial ability, (c) testing of manual dexterity assessed by validated psychomotor tests and (d) evaluation of surgical performance by Objective Structured Assessment of Technical Skills (OSATS).

RESULTS

Self-assessed proficiency and experience levels did not correlate with objectively evaluated surgical performances (OSATS). However, low-level visual-spatial ability tests strongly correlated with OSATS while intermediate- and high-level tests did not. No correlation was found between psychomotor ability and clinical performance.

CONCLUSIONS

Self-assessed proficiency is not a good predictor for surgical performance as experts tend to be overconfident. To evaluate and predict surgical performance, visual-spatial ability tests seem to be more appropriate than measuring manual dexterity which failed to correlate with the surgical outcome.

Does rating with a checklist improve the effect of E-learning for cognitive and practical skills in bariatric surgery? A rater-blinded, randomized-controlled trial

BACKGROUND

Mental training of laparoscopic procedures with E-learning has been shown to translate to the operating room. The present study aims to explore whether the use of checklists during E-learning improves transfer of skills to the simulated OR on a Virtual Reality (VR) trainer for Roux-en-Y gastric bypass (RYGB).

METHODS

Laparoscopy naive medical students (n = 80) were randomized in two groups. After an E-learning introduction to RYGB, checklist group rated RYGB videos using the validated Bariatric Objective Structured Assessment of Technical Skills (BOSATS) checklist while group without checklist only observed the videos. Participants then performed RYGB on a VR-trainer twice and were evaluated by a blinded expert rater using BOSATS. A multiple choice (MC) knowledge test on RYGB was performed. Suturing on a cadaveric porcine small bowel was evaluated using objective structured assessment of technical skill (OSATS).

RESULTS

Checklist group was better in the knowledge test (A 8.3 ± 1.1 vs. B 7.1 ± 1.3; p ≤ 0.001) and there was a trend towards better VR RYGB performance (BOSATS) on the first try (85.9 ± 10.2 vs. 81.1 ± 11.5; p = 0.058), but not on the second try (92.0 ± 9.7 vs. 89.3 ± 10.5; p = 0.251). Suturing as measured by OSATS was not different (29.5 ± 3.0 vs. 29.0 ± 3.5; p = 0.472).

CONCLUSION

This study presents evidence that the use of a BOSATS checklist during E-learning helps trainees to improve their knowledge acquisition with E-learning. The transfer from mental training to the simulated OR environment seems to be partially enhanced by use of the BOSATS checklist. However, more research is required to investigate potential benefits.

One or two trainees per workplace for laparoscopic surgery training courses: results from a randomized controlled trial

BACKGROUND

There are no standards for optimal utilization of workplaces in laparoscopic training. This study aimed to define whether laparoscopy training should be done alone or in pairs (known as dyad training).

METHODS

This was a three-arm randomized controlled trial with laparoscopically naïve medical students (n = 100). Intervention groups participated alone (n = 40) or as dyad (n = 40) in a multimodality training curriculum with e-learning, basic, and procedural skills training using box and VR trainers. The control group (n = 20) had no training. Post-performance of a cadaveric porcine laparoscopic cholecystectomy (LC) was measured as the primary outcome by blinded raters using the objective structured assessment of technical skills (OSATS). Global operative assessment of laparoscopic skills (GOALS), time for LC, and VR performances were secondary outcomes.

RESULTS

There were no differences between groups for performance scores [OSATS: alone (40.2 ± 9.8) vs. dyad (39.8 ± 8.6), p = 0.995; alone vs. control (37.1 ± 7.4), p = 0.548; or dyad vs. control, p = 0.590; and GOALS score: alone (10.6 ± 3.0) vs. dyad (10.0 ± 2.7), p = 0.599; alone vs. control (10.1 ± 3.0), p = 0.748; or dyad vs. control, p = 0.998]. Dyad finished LC faster than control [median = 62.5 min (CI 58.0-73.0) vs. 76.5 min (CI 72.0-80+); p = 0.042], while there were no inter-group differences between alone vs. control [median = 69.0 min (CI 62.0-76.0) vs. control; p = 0.099] or alone vs. dyad (p = 0.840). Dyad and alone showed superior performance on the VR trainer vs. control for time, number of movements, and path length, but not for complications and application of cautery.

CONCLUSIONS

The curriculum provided trainees with the laparoscopic skills needed to perform LC safely, irrespective of the number of trainees per workplace. Dyad training reduced the operation time needed for LC. Therefore, dyad training seems to be a promising alternative, especially if training time is limited and resources must be used as efficiently as possible. Trial registration German Clinical Trials Register: DRKS00004675.

Comparison of faculty versus structured peer-feedback for acquisitions of basic and intermediate-level surgical skills

PURPOSE

Video feedback and faculty feedback has been shown to improve surgical performance; however, consistent access to faculty is challenging. We studied the utility of structured peer-feedback (PF) compared to faculty-feedback (FF) during acquisition of basic and intermediate surgical skills.

METHODOLOGY

Two randomized non-inferiority trials were conducted with 1st (n = 30) and 2nd year (n = 29) medical students learning skin-lesion excision and closure (S), and single-layer hand-sewn bowel anastomosis (B), respectively. Five attempts were performed. PF participants used an Objective Structured Assessment of Technical Skills tool to guide feedback. Blinded raters assessed video-recorded performance, time and Integrity of the completed task were also assessed.

RESULTS

For both tasks performance by PF was comparable to FF (P = 0.111). Both groups improved significantly: performance (B:P < 0.0001, S:P = 0.035), time (B:P = 0.043, S:P < 0.0001) and integrity (B:P < 0.0001, S:P < 0.032).

CONCLUSION

Structured peer-feedback is equivalent to faculty-feedback in the acquisition of basic and intermediate surgical skills, giving students freedom to practice independently.

Structured assessment of laparoscopic camera navigation skills: the SALAS score

BACKGROUND

Tools are needed to assess laparoscopic camera navigation (LCN) in the operating room. Here, we aimed to develop an objective rating scale for LCN.

STUDY DESIGN

We defined the following key aspects of LCN: operational field centering, correct angle of the horizon, correct instrument visualization, verbal commands from the operating surgeon, and manual corrections from the operating surgeon. We then developed a score based on intraoperative error evaluation from intraoperative recordings of 80 procedures. Finally, the newly developed score was validated by four different raters using video-based analysis of 20 elective laparoscopic cholecystectomies.

RESULTS

We developed and validated a tool for the structured assessment of laparoscopic assistant skills (SALAS). This score showed good internal consistency, with a Cronbach's alpha of > 0.7. Intraclass correlation revealed a low interrater variability (ICC 0.866) for the total score. Comparison of experienced and inexperienced camera assistants revealed significantly better SALAS scores for experienced assistants (p < 0.05).

CONCLUSION

Our present results show that SALAS score is valid, reliable, and practicable. This score can be used for future investigations of camera navigation efficiency and training.

Direct Observation vs. Video-Based Assessment in Flexible Cystoscopy

OBJECTIVE

Direct observation in assessment of clinical skills is prone to bias, demands the observer to be present at a certain location at a specific time, and is time-consuming. Video-based assessment could remove the risk of bias, increase flexibility, and reduce the time spent on assessment. This study investigated if video-based assessment was a reliable tool for cystoscopy and if direct observers were prone to bias compared with video-raters.

DESIGN

This study was a blinded observational trial. Twenty medical students and 9 urologists were recorded during 2 cystoscopies and rated by a direct observer and subsequently by 2 blinded video-raters on a global rating scale (GRS) for cystoscopy. Both intrarater and interrater reliability were explored. Furthermore, direct observer bias was explored by a paired samples t-test.

RESULTS

Intrarater reliability calculated by Pearson's r was 0.86. Interrater reliability was 0.74 for single measure and 0.85 for average measures. A hawk-dove effect was seen between the 2 raters. Direct observer bias was detected when comparing direct observer scores to the assessment by an independent video-rater (p < 0.001).

CONCLUSION

This study found that video-based assessment was a reliable tool for cystoscopy with 2 video-raters. There was a significant bias when comparing direct observation with blinded video-based assessment.

Comparing Real-time Versus Delayed Video Assessments for Evaluating ACGME Sub-competency Milestones in Simulated Patient Care Environments

Background

Simulation is an effective method for creating objective summative assessments of resident trainees. Real-time assessment (RTA) in simulated patient care environments is logistically challenging, especially when evaluating a large group of residents in multiple simulation scenarios. To date, there is very little data comparing RTA with delayed (hours, days, or weeks later) video-based assessment (DA) for simulation-based assessments of Accreditation Council for Graduate Medical Education (ACGME) sub-competency milestones. We hypothesized that sub-competency milestone evaluation scores obtained from DA, via audio-video recordings, are equivalent to the scores obtained from RTA.

Methods

Forty-one anesthesiology residents were evaluated in three separate simulated scenarios, representing different ACGME sub-competency milestones. All scenarios had one faculty member perform RTA and two additional faculty members perform DA. Subsequently, the scores generated by RTA were compared with the average scores generated by DA. Variance component analysis was conducted to assess the amount of variation in scores attributable to residents and raters.

Results

Paired t-tests showed no significant difference in scores between RTA and averaged DA for all cases. Cases 1, 2, and 3 showed an intraclass correlation coefficient (ICC) of 0.67, 0.85, and 0.50 for agreement between RTA scores and averaged DA scores, respectively. Analysis of variance of the scores assigned by the three raters showed a small proportion of variance attributable to raters (4% to 15%).

Conclusions

The results demonstrate that video-based delayed assessment is as reliable as real-time assessment, as both assessment methods yielded comparable scores. Based on a department’s needs or logistical constraints, our findings support the use of either real-time or delayed video evaluation for assessing milestones in a simulated patient care environment.

Impact of visual-spatial ability on laparoscopic camera navigation training

BACKGROUND

Technical limitations of minimally invasive surgery challenge both surgeons and camera assistants. Current research indicates that visual-spatial ability (VSA) has impact on learning of laparoscopic camera navigation (LCN). However, it remains unclear if complexity of LCN tasks influences the impact of VSA. The aim of this study was to examine the influence of VSA on LCN training within tasks of different complexity levels.

METHODS

The present study was conducted as a monocentric prospective trial. VSA was assessed with a cube comparison test before participants underwent LCN training. LCN training consisted of three tasks with increasing complexity. Each task was performed four times and performance was assessed each time. Correlations and multivariate regression analysis were used to assess the influence of VSA on LCN skills.

RESULTS

Seventy-one participants were included (35 males). Significant performance improvement and faster completion times were observed from the first to fourth trial of all three LCN training tasks. Significant positive correlations between VSA and performance on LCN task 3 were found (performance: r _s = 0.47; p < 0.001, time: r _s = -0.43; p < 0.001). Multivariate regression revealed that higher VSA resulted in greater reduction of time between the first trials of LCN training task 3 (B = -1.67, p = 0.031).

CONCLUSION

In the present study, all trainees improved LCN performance during the training. VSA seems to have impact on LCN performance and training progress particularly for complex LCN tasks. The relation of VSA and LCN performance was stronger for less experienced participants and in the beginning of the learning phase.

Does rating the operation videos with a checklist score improve the effect of E-learning for bariatric surgical training? Study protocol for a randomized controlled trial

BACKGROUND

Laparoscopic training has become an important part of surgical education. Laparoscopic Roux-en-Y gastric bypass (RYGB) is the most common bariatric procedure performed. Surgeons must be well trained prior to operating on a patient. Multimodality training is vital for bariatric surgery. E-learning with videos is a standard approach for training. The present study investigates whether scoring the operation videos with performance checklists improves learning effects and transfer to a simulated operation.

METHODS/DESIGN

This is a monocentric, two-arm, randomized controlled trial. The trainees are medical students from the University of Heidelberg in their clinical years with no prior laparoscopic experience. After a laparoscopic basic virtual reality (VR) training, 80 students are randomized into one of two arms in a 1:1 ratio to the checklist group (group A) and control group without a checklist (group B). After all students are given an introduction of the training center, VR trainer and laparoscopic instruments, they start with E-learning while watching explanations and videos of RYGB. Only group A will perform ratings with a modified Bariatric Objective Structured Assessment of Technical Skill (BOSATS) scale checklist for all videos watched. Group B watches the same videos without rating. Both groups will then perform an RYGB in the VR trainer as a primary endpoint and small bowel suturing as an additional test in the box trainer for evaluation.

DISCUSSION

This study aims to assess if E-learning and rating bariatric surgical videos with a modified BOSATS checklist will improve the learning curve for medical students in an RYGB VR performance. This study may help in future laparoscopic and bariatric training courses.

TRIAL REGISTRATION

German Clinical Trials Register, DRKS00010493 . Registered on 20 May 2016.

Validation of the mobile serious game application Touch Surgery™ for cognitive training and assessment of laparoscopic cholecystectomy

BACKGROUND

Touch Surgery^™ (TS) is a serious gaming application for cognitive task simulation and rehearsal of key steps in surgical procedures. The aim was to establish face, content, and construct validity of TS for laparoscopic cholecystectomy (LC). Furthermore, learning curves with TS and a virtual reality (VR) trainer were compared in a randomized trial.

METHODS

The performance of medical students and general surgeons was compared for all three modules of LC in TS to establish construct validity. Questionnaires assessed face and content validity. For analysis of learning curves, students were randomized to train on VR or TS first, and then switched to the other training modality. Performance data were recorded.

RESULTS

54 Surgeons and 51 medical students completed the validation study. Surgeons outperformed students with TS: patient preparation (students = 45.0 ± 19.1%; surgeons = 57.3 ± 15.2%; p < 0.001), access and laparoscopy (students = 70.2 ± 10.9%; surgeons = 75.9 ± 9.7%; p = 0.008) and LC (students = 69.8 ± 12.4%; surgeons = 77.7 ± 9.6%; p < 0.001). Both groups agreed that TS was a highly useful and realistic application. 46 students were randomized for learning curve analysis. It took them 2-4 attempts to reach a 100% score with TS. Training with TS first did not improve students' performance on the VR trainer; however, students who trained with VR first scored significantly higher in module 3 of TS.

CONCLUSION

TS is an accepted serious gaming application for learning cognitive aspects of LC with established construct, face, and content validity. There appeared to be a synergy between TS and the VR trainer. Therefore, the two training modalities should accompany one another in a multimodal training approach to laparoscopy.

App-based serious gaming for training of chest tube insertion: study protocol for a randomized controlled trial

Background

Chest tube insertion is a standard intervention for management of various injuries of the thorax. Quick and accurate execution facilitates efficient therapy without further complications. Here, we propose a new training concept comprised of e-learning elements as well as continuous rating using an objective structured assessment of technical skills (OSATS) tool. The study protocol is presented for a randomized trial to evaluate e-learning with app-based serious gaming for chest drain insertion.

Methods

The proposed randomized trial will be carried out at the Department of Orthopedics and Traumatology at Heidelberg University in the context of regular curricular teaching for medical students (n = 90, 3rd to 6th year). The intervention group will use e-learning with the serious gaming app Touch Surgery (TM) for chest drain insertion, whereas the control group uses serious gaming for an unrelated procedure. Primary endpoint is operative performance of chest drain insertion in a porcine cadaveric model according to OSATS.

Discussion

The randomized trial will help determine the value of e-learning with the serious gaming app Touch Surgery (TM) for chest drain insertion by using the OSATS score. The study will improve surgical training for trauma situations.

Trial registration

Trial Registration Number, DRKS00009994. Registered on 27 May 2016.

Electronic supplementary material

The online version of this article (doi:10.1186/s13063-017-1799-5) contains supplementary material, which is available to authorized users.

Face validity of the pulsatile organ perfusion trainer for laparoscopic cholecystectomy

BACKGROUND

The pulsatile organ perfusion (POP) trainer provides training of minimally invasive surgery (MIS) with real instruments and cadaveric organs. It provides training of full procedures with simulation of bleeding. Although widely used, the face validity has not yet been evaluated. This study aimed to establish face validity of the POP trainer for laparoscopic cholecystectomy (LC) and its usefulness compared with other training modalities.

MATERIALS AND METHODS

During MIS courses, the participants (n = 52) used the POP trainer to perform LC. Face validity was assessed with questionnaires for realism and usefulness on a five-point Likert scale. Participants were divided into two groups: experts (n = 15) who had performed more than 50 laparoscopic procedures and novices (n = 37) with less than 50 procedures. Secondary aims included the ranking of training modalities, as well as exploration of their specific advantages and disadvantages.

RESULTS

The POP trainer was found to be realistic (3.8 ± 0.9) and useful (4.6 ± 0.9). Differences between experts and novices were only found for "The training modality resembles reality" (3.1 ± 0.8 vs. 3.8 ± 0.7; p = 0.010), "The operation on the POP trainer is realistic" (3.4 ± 1.1 vs. 4.5 ± 0.8; p = 0.003), and "It would be desirable to have a POP trainer at my own hospital" (4.2 ± 1.1 vs. 4.8 ± 0.8; p = 0.040). In the ranking, the animal training (1.1 ± 0.3) placed first, the POP trainer (2.3 ± 0.9) second, and the VR trainer (2.8 ± 0.9) and box trainer (2.8 ± 1.1) third. The realistic simulation of animal training was named as an advantage most often, while the unrealistic simulation of the VR trainer was the most often named disadvantage.

CONCLUSIONS

The POP trainer was rated a highly realistic and useful training modality with face validity for LC. Differences between experts and novices existed concerning realism and desirability. Future studies should evaluate the POP trainer for more advanced surgical procedures. The POP trainer widens the spectrum of modalities for training of MIS in a safe environment outside the operating room.