Self-assessment in laparoscopic surgical skills training: Is it reliable?

Self-assessment, and not continuous training, improves basic open suturing skills

BACKGROUND

To develop and maintain suturing skills, clinical exposure is important. When clinical exposure cannot be guaranteed, an adequate training schedule for suturing skills is required. This study evaluates the effect of continuous training, 'reflection before practice' and self-assessment on basic open suturing skills.

METHODS

Medical students performed four basic suturing tasks on a simulation set up before ('pre-test') and after their surgical rotation ('after-test'). Participants were divided in three groups; the 'clinical exposure group' (n = 44) had clinical exposure during their rotation only, the 'continuous training group' (n = 16) completed a suturing interval training during their rotation and the 'self-assessment group' (n = 16) also completed a suturing interval training, but with the use of reflection before practice and self-assessment. Parameters measured by a tracking system during the suturing tasks and a calculated 'composite score' were compared between groups and test-moments.

RESULTS

A significantly better composite score was found at the after-test compared to the pre-test for all groups for all basic suturing tasks (0.001 ≤ p ≤ 0.049). The self-assessment group scored better at the pre-test than the other two groups for all tasks, except for 'knot tying by hand' (0.004 ≤ p ≤ 0.063). However, this group did not score better at the after-test for all tasks, compared to the other two groups. This resulted in a smaller delta of time ('transcutaneous suture', p = 0.013), distance ('Donati suture' and 'intracutaneous suture', 0.005 ≤ p ≤ 0.009) or composite score (all tasks, except for knot tying by hand, 0.007 ≤ p ≤ 0.061) in the self-assessment group.

CONCLUSION

Reflection before practice and self-assessment during continuous training of basic open suturing tasks, may improve surgical skills at the start of the learning curve.

A Study of Surgical Residents' Self-Assessment of Open Surgery Skills Using Gap Analysis

BACKGROUND

Surgical residents need structured and objective feedback to develop their skills and become capable of performing surgical procedures autonomously. A shortage of experienced surgical staff has prompted residents to seek feedback from self-assessment and peer assessments.

OBJECTIVES

We investigated whether surgical residents can reliably rate their own and their peers' basic surgical skills using the Global Rating Scale (GRS) from the Objective Structured Assessment of Technical Skills.

METHODS

The study was a prospective and descriptive study conducted using gap analysis at the Copenhagen Academy for Medical Education and Simulation (Copenhagen, Denmark) from 2016 to 2017. Surgical residents were recruited during a course in basic open surgical skills. Among 102 course participants, 53 met the inclusion criteria and 22 participated in the study.

RESULTS

We recruited surgical residents based in the Capital Region and Zealand Region of Demark, and 42% of eligible residents participated in the study. Surgical residents underestimated their own surgical performance (median, 17 [range, 15-18] vs. 20 [range, 19.75-22]; P < 0.001). They also rated their peers lower than an experienced rater did (median 10 [range, 8.75-14] vs. 15 and median 20.5 [range, 19-22] vs. 23; both P < 0.001). Gap analysis revealed residents had unrecognized strengths (ie, self-underappraisal) in most GRS domains.

CONCLUSIONS

Ratings are unreliable when surgical residents assess their own and their peers' performances using GRS. A gap analysis revealed unrecognized strengths in time and motion, instrument handling, knowledge of instruments and sutures, and knowledge of specific procedure as well as unrecognized weaknesses in flow of operation and forward planning.

Learning surgical knot tying and suturing technique - effects of different forms of training in a controlled randomized trial with dental students

Objective

The acquisition of surgical skills requires motor learning. A special form of this is intermanual transfer by transferring motor skills from the nondominant hand (NDH) to the dominant hand (DH). The purpose of this study was to determine the learning gains that can be achieved for the DH by training with the DH, the NDH, and by non-surgical alternative training (AT).

Methods

124 preclinical (n=62) and clinical (n=62) dental students completed surgical knot tying and suturing technique training with the DH, with the NDH, and an AT in a controlled randomized trial.

Results

A statistically significant learning gain in knot tying and suture technique with the DH was evident only after training with the DH when compared to training with the NDH (p<0.001 and p=0.004, respectively) and an AT (p=0.001 and p=0.010, respectively). Of those students who achieved a learning gain ≥4 OSATS points, 46.4% (n=32) benefited in their knot tying technique with the DH from training with the DH, 29.0% (n=20) from training with the NDH, and 24.6% (n=17) from an AT while 45.7% (n=32) benefited in their suturing technique with the DH from training with the DH, 31.4% (n=22) from training with the NDH, and 22, 9% (n=16) from an AT.

Conclusions

Training with the DH enabled significantly better learning gains in the surgical knot tying and suturing techniques with the DH.

Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy

INTRODUCTION

Self-review of recorded surgical procedures offers new opportunities for trainees to extend technical learning outside the operating-room. Valid tools for self-assessment are required prior to evaluating the effectiveness of video-review in enhancing technical learning. Therefore, we aimed to contribute evidence regarding the validity of intraoperative performance assessment tools for video-based self-assessment by general surgery trainees when performing laparoscopic cholecystectomies.

METHODS AND PROCEDURES

Using a web-based platform, general surgery trainees in a university-based residency program submitted recorded laparoscopic cholecystectomy procedures where they acted as the supervised primary surgeon. Attending surgeons measured operative performance at the time of surgery using general and procedure-specific assessment tools [Global Operative Assessment of Laparoscopic Skills (GOALS) and Operative Performance Rating System (OPRS), respectively] and entrustability level (O-SCORE). Trainees self-evaluated their performance from video-review using the same instruments. The validity of GOALS and OPRS for trainee self-assessment was investigated by testing the hypotheses that self-assessment scores correlate with (H1) expert assessment scores, (H2) O-SCORE, and (H3) procedure time and that (H4) self-assessment based on these instruments differentiates junior [postgraduate year (PGY) 1-3] and senior trainees (PGY 4-5), as well as (H5)simple [Visual Analogue Scale (VAS) ≤ 4] versus complex cases (VAS > 4). All hypotheses were based on previous literature, defined a priori, and were tested according to the COSMIN consensus on measurement properties.

RESULTS

A total of 35 videos were submitted (45% female and 45% senior trainees) and self-assessed. Our data supported 2 out of 5 hypotheses (H1 and H4) for GOALS and 3 out of 5 hypotheses (H1, H4 and H5) for OPRS, for trainee self-assessment.

CONCLUSIONS

OPRS, a procedure-specific assessment tool, was better able to differentiate between groups expected to have different levels of intraoperative performance, compared to GOALS, a general assessment tool. Given the interest in video-based learning, there is a need to further develop valid procedure-specific tools to support video-based self-assessment by trainees in a range of procedures.

Validation of a Newly Developed Competency Assessment Tool for the Posterior Sagittal Anorectoplasty

INTRODUCTION

 The correction of an anorectal malformation (ARM) is complex and relatively infrequent. Simulation training and subsequent assessment may result in better clinical outcomes. Assessment can be done using a competency assessment tool (CAT). This study aims to develop and validate a CAT for the posterior sagittal anorectoplasty (PSARP) on a simulation model.

MATERIALS AND METHODS

 The CAT-PSARP was developed after consultation with experts in the field. The PSARP was divided into five steps, while tissue and instrument handling were scored separately. Participants of pediatric colorectal hands-on courses in 2019 and 2020 were asked to participate. They performed one PSARP procedure on an ARM simulation model, while being assessed by two objective observers using the CAT-PSARP.

RESULTS

 A total of 82 participants were enrolled. A fair interobserver agreement was found for general skills (intraclass correlation coefficient [ICC] = 0.524, p < 0.001), a good agreement for specific skills (ICC = 0.646, p < 0.001), and overall performance (ICC = 0.669, p < 0.001). The experienced group scored higher on all steps (p < 0.001), except for "anoplasty (p = 0.540)," compared with an inexperienced group.

CONCLUSION

 The CAT-PSARP is a suitable objective assessment tool for the overall performance of the included steps of the PSARP for repair of an ARM on a simulation model.

Using Self-Rated Examinations to Ensure Competence in Laparoscopy

INTRODUCTION

Asking participants to rate their own performance during unsupervised training in laparoscopy is reliable and may be cost-effective. The objective of the study was to explore the reliability of self-rated examinations where participants rate their own performance and decide themselves when they have passed tasks in basic laparoscopic skills.

METHODS

This prospective observational study was conducted at the Copenhagen Academy for Medical Education and Simulation where simulation-based laparoscopic skill training is offered. Here, participants taking part in a basic laparoscopic skills course were asked to rate their own performance and decide when they had passed the Training and Assessment of Basic Laparoscopic Techniques test. To explore reliability, all examinations were video recorded and rated by a blinded rater after the end of the course.

RESULTS

Thirty-two surgical trainees participated in the course, and 28 completed the study. We found a high reliability when comparing self-rated scores and blinded ratings with an intraclass correlation coefficient of 0.89 ( P < 0.001); self-rated scores compared with blinded ratings were not significantly different (mean = 451 vs. 455, P = 0.28), and the participants did not underestimate nor overestimate their performance.

CONCLUSIONS

Ratings from self-rated examinations in a basic laparoscopic skills course are reliable, and participants neither underestimate nor overestimate their performance. Self-rated examinations may also be beneficial because they also can offer a cost-effective approach to assessment of surgical trainees.

Comparing control room operators' and experts' assessment of team performance using structured task-specific observation protocols and scenario replay

Operators' self-assessment has received limited interest within process control or human-system evaluation. Research on self-assessment has been criticised for poor assessment methodology, and consequently, its status is unclear. This study hypothesised that, given adequate assessment methods (such as task-specific assessment items and scenario replay), we could observe relatively accurate self-assessment results. Eighteen licensed operators and two experts assessed team performance in six nuclear control room scenarios. The results reveal an overall agreement between operators and experts, measured by the intraclass correlation coefficient, ranging from 0.60 to 0.70, which lies close to the intraclass correlation coefficient of 0.75 for the experts. This demonstrates potential for achievement of relatively accurate operator self-assessment for complex work. The agreement varied in a similar manner for both expert agreement and operator-expert agreement across eight performance dimensions. In addition, the operators' self-assessment provided additional information beyond observer assessment in identifying non-acceptable performance items.

Development and clinical implementation of a structured, simulation-based training programme in laparoscopic appendectomy: description, validation and evaluation

Background

Laparoscopic appendectomy is a common procedure in general surgery but is likely underused in structured and real-life teaching. This study describes the development, validation and evaluation of implementing a structured training programme for laparoscopic appendectomy.

Study design

A structured curriculum and simulation-based programme for trainees and trainers was developed. All general surgery trainees and trainers were involved in laparoscopic appendectomies. All trainees and trainers underwent the structured preprocedure training programme before real-life surgery evaluation. A standardised form evaluated eight technical steps (skills) of the procedure as well as an overall assessment, and nine elements of communication (feedback), and was used for bilateral evaluation by each trainee and trainer. A consecutive, observational cohort over a 12-month period was used to gauge real-life implementation.

Results

During 277 eligible real-life appendectomies, structured evaluation was performed in 173 (62%) laparoscopic appendectomies, for which 165 forms were completed by 19 trainees. Construct validity was found satisfactory. Inter-rater reliability demonstrated good correlation between trainee and trainer. The trainees’ and trainers’ stepwise and overall assessments of technical skills had an overall good reliability (intraclass correlation coefficient of 0.88). The vast majority (92.2%) of the trainees either agreed or strongly agreed that the training met their expectations.

Conclusion

Structured training for general surgery residents can be implemented for laparoscopic appendectomy. Skills assessment by trainees and trainers indicated reliable self-assessment. Overall, the trainees were satisfied with the training, including the feedback from the trainers.

The Accuracy of Health Professions Students' Self-Assessments Compared to Objective Measures of Competence

Objective. To review the literature related to potential associations between self-assessed perceptions of knowledge and/or confidence with actual competence.Findings. Twenty-two articles involving a variety of disciplines, including undergraduate studies, dentistry, medicine, and pharmacy were included following the screening process. Most studies focused on student self-reported confidence rather than competency assessed by a formative examination. Only a handful of studies were centered on pharmacy education.Summary. Educational research that evaluates student learning should employ measures of competency as the primary outcome rather than student perceptions. Using student perceptions as the primary measure of student learning should be avoided, but student perceptions may have some utility as an adjunct to competency data.

Are Self-Assessment and Peer Assessment of Added Value in Training Complex Pediatric Surgical Skills?

INTRODUCTION

 Self-assessment aids "reflection-before-practice," which is expected to result in a better understanding of one's strengths and weaknesses and consequently a better overall performance. This is, however, rarely used in surgical training. This study aims to evaluate the correlation between self-, peer-, and expert assessment on surgical skills of pediatric surgical trainees.

MATERIALS AND METHODS

 A competency assessment tool for the posterior sagittal anorectoplasty (CAT-PSARP) was previously developed and validated. During international hands-on pediatric colorectal workshops in 2019 and 2020, participants practiced the PSARP on an inanimate anorectal malformation model. They were assisted by a peer and assessed by two independent expert observers, using the CAT-PSARP. After the training session, both self- and peer assessment were completed, using the same CAT-PSARP.

RESULTS

 A total of 79 participants were included. No correlation was found between the overall CAT-PSARP scores of the expert observers and the self-assessment (r = 0.179, p = 0.116), while a weak correlation was found between experts and peer assessment (r = 0.317, p = 0.006). When comparing the self-assessment scores with peer assessment, a moderate correlation was found for the overall performance score (r = 0.495, p < 0.001). Additionally, the participants who were first to perform the procedure scored significantly better than those who trained second on the overall performance (mean 27.2 vs. 24.4, p < 0.001).

CONCLUSION

 Participants, peers, and experts seemed to have a unique view on the performance during training because there was little correlation between outcomes of the trainees and the experts. Self-assessment may be useful for reflection during training; however, expert assessment seems to be essential for assessment of surgical skills.

Evaluating the Impact of Surgeon Self-Awareness by Comparing Self vs Peer Ratings of Surgical Skill and Outcomes for Bariatric Surgery

OBJECTIVE

To evaluate variation in self vs. peer-assessments of surgical skill using surgical videos and compare surgeon-specific outcomes with bariatric surgery.

SUMMARY BACKGROUND DATA

Prior studies have demonstrated that surgeons with lower peer-reviewed ratings of surgical skill had higher complication rates after bariatric surgery.

METHODS

This is a retrospective cohort study of 25 surgeons who voluntarily submitted a video of a typical laparoscopic sleeve gastrectomy (SG) between 2015-2016. Videos were self and peer-rated using a validated instrument based on a 5-point Likert scale (5="master surgeon" and 1="surgeon-in-training"). Risk adjusted 30-day complication rates were compared between surgeons who over-rated and under-rated their skill based on data from 24,186 SG cases as well as 12,888 gastric bypass (GBP) cases.

RESULTS

Individual overall self-rating of surgical skill varied between 2.5 to 5. Surgeons in the top quartile for self:peer ratings (n=6, ratio 1.58) had lower overall mean peer-scores (2.98 vs 3.79, p = 0.0150) than surgeons in the lowest quartile (n = 6, ratio 0.94). Complication rates between top and bottom quartiles were similar after SG, however leak rates were higher with GBP among surgeons who over-rated their skill with sleeve gastrectomy (0.65 vs 0.27, p = 0.0181). Surgeon experience was similar between comparison groups.

CONCLUSIONS AND RELEVANCE

Self-perceptions of surgical skill varied widely. Surgeons who over-rated their skill had higher leak rates for more complex procedures. Video assessments can help identify surgeons with poor self-awareness who may benefit from a surgical coaching program.

Entrustable Professional Activities in General Surgery: Trends in Resident Self-Assessment

OBJECTIVE

Effective self-assessment is a cornerstone of lifelong professional development; however, evidence suggests physicians have a limited ability to self-assess. Novel strategies to improve the accuracy of learner self-assessment are needed. Our institution's surgical entrustable professional activity (EPA) implementation strategy incorporates resident self-assessment to address this issue. This study evaluates the accuracy of resident self-assessment versus faculty assessment across 5 EPAs in general surgery.

DESIGN, SETTING, PARTICIPANTS

Within a single academic general surgery residency program, assessment data for 5 surgery EPAs was prospectively collected using a mobile application. Matched assessments (resident and faculty assessments for the same clinical encounter) were identified and the remainder excluded. Assessment scores were compared using Welch's t test. Agreement was analyzed using Cohen's kappa with squared weights.

RESULTS

One thousand eight hundred and fifty-seven EPA assessments were collected in 17 months following implementation. One thousand one hundred and fifty-five (62.2%) were matched pairs. Residents under-rated their own performance relative to faculty assessments (2.36 vs 2.65, p < 0.01). This pattern held true for all subsets except for Postgraduate Year (PGY)2 residents and Inguinal Hernia EPAs. There was at least moderate agreement between matched resident and faculty EPA assessment scores (κ = 0.57). This was consistent for all EPAs except Trauma evaluations, which were completed by faculty from 2 different departments. Surgery resident self-assessments more strongly agreed with Surgery faculty assessments than Emergency Medicine faculty assessments (κ = 0.58 vs 0.36).

CONCLUSIONS

Resident EPA self-assessments are equivalent or slightly lower than faculty assessments across a wide breadth of clinical scenarios. Resident and faculty matched assessments demonstrate moderate agreement.

The effect of verbal feedback, video feedback, and self-assessment on laparoscopic intracorporeal suturing skills in novices: a randomized trial

BACKGROUND

Laparoscopic skill acquisition involves a steep learning curve and laparoscopic suturing is an exceptionally challenging task. By improving the way feedback is given, trainees can learn these skills more effectively. This study aims to establish the most effective form of structured feedback on laparoscopic suturing skill acquisition in novices, by comparing the effects of expert verbal feedback, video review with expert feedback (video feedback), and video review with self-assessment.

METHODS

A prospective randomized blinded trial comparing verbal feedback, video feedback, and self-assessment. Novices in laparoscopic surgery were tasked with performing laparoscopic suturing with intracorporeal knot tying. Time was given for practice, and pre- and post-feedback assessments were undertaken. Suturing performance was measured using a task-specific checklist and global ratings. A post-study questionnaire was used to measure participant-perceived confidence, knowledge, and experience levels.

RESULTS

Fifty-one participants were randomized and allocated equally into the three groups. Performance in all three groups improved significantly from baseline. Video feedback had the largest improvement margin with checklist and global score improvements of 17.1% (± 9.9%) and 14.7% (± 9.3%), respectively. Performance improvements between groups were statistically significant in the global components (p = 0.004) but not the checklist components (p = 0.186). Global score improvement was significantly better in the video feedback group but was statistically insignificant between the self-assessment and verbal feedback groups. Questionnaire responses demonstrated positive results in confidence, knowledge, and experience levels, across all three study groups, with no differences between the groups (p > 0.05).

CONCLUSION

Structured video feedback facilitates reflection and self-directed learning, which improves the ability to develop proficiency in surgical skills. Combining both self-assessment and video feedback may be beneficial over verbal feedback alone due to the advantages of video review. These techniques should therefore be considered for implementation into surgical education curricula.

Evidence That Female Urologists and Urology Trainees Tend to Underrate Surgical Skills on Self-Assessment

BACKGROUND

Historically low, the proportion of female urology residents now exceeds 25% in recent years. Self-assessment is a widely used tool to track progress in medical education. However, the validity of its results and gender differences may influence interpretation. Simulation of surgical skills is increasingly common in modern residency training and standardizes certain objective tasks and skills. The objective of this study was to identify gender differences in self-assessment of surgeons and trainees when using simulation of surgical skills.

METHODS

Medical students, residents, and attending and retired surgeons completed simple interrupted suturing. Assessment was self-rated using previously tested visual analog motion scales. Tasks were video recorded and rated by blinded expert surgeons using identical motion scales. Computer vision motion tracking software was used to objectively analyze the kinematics of surgical tasks.

RESULTS

Proportion of female (n = 17) and male (n = 20) participants did not differ significantly by the level of training, P = 0.76. Five expert surgeons evaluated 84 video segments of simple interrupted suturing tasks (mean 3.0 segments per task per participant). Self-assessment correlated well overall with expert rating for motion economy (Pearson correlation coefficient 0.61, P < 0.001) and motion fluidity (0.55, P = 0.002). Women underrated their performance in accordance with mean individual difference of self-assessment and expert assessment scores (Δ SAS-EAS) for both economy of motion (mean ± SEM -1.1 ± 0.38, P = 0.01) and fluidity of motion (-1.3 ± 0.39, P < 0.01). On the same measures, men tended to rate themselves in accordance with experts (-0.16 ± 0.36, P = 0.63; -0.09 ± 0.41, P = 0.82, respectively). Δ SAS-EAS did not differ significantly on any rating scale across levels of training. Expert ratings did not differ significantly by gender for any domain.

CONCLUSIONS

Female surgeons and trainees underrate some technical skills on self-assessment when compared with expert ratings, whereas male surgeon and trainee self-ratings and expert ratings were similar. Further work is needed to determine if these differences are accentuated across increasingly difficult tasks.

Virtual Operating Room Simulation Setup (VORSS) for Procedural Training in Minimally Invasive Surgery – a Pilot Study

Virtual reality (VR) training is widely used in several minimal invasive surgery (MIS) training curricula for procedural training. However, VR training in its current state lack immersive training environments, such as using head-mounted displays that is implemented in military or aviation training and even entertainment. The virtual operating room simulation setup (VORSS) is explored in this study to determine the effectiveness of immersive training in MIS. Twenty-eight surgeons and surgical trainees performed a laparoscopic cholecystectomy on the VORSS comprising of a head-mounted 360-degree realistic OR surrounding on a VR laparoscopic simulator. The VORSS replicated a full setup of instruments and surgical team-members as well as some of the distractions occurring during surgical procedures. Questionnaires were followed by semi-structured interviews to collect the data. Experts and novices found the VORSS to be intuitive and easy to use (p = 0.001). The outcome of the usability test, applying QUESI and NASA-TLX, reflected the usability of the VORSS (p < 0.05), at the cognitive level, which indicates a good sense of immersion and satisfaction, when performing the procedure within VORSS. The need for personalized experience within the setup was strongly noted from most of the participants. The VORSS for procedural training has the potential to become a useful tool to provide immersive training in MIS surgery. Further optimizing of the VORSS realism and introduction of distractors in the OR should result in an improvement of the system.

Self-Assessment of Surgical Skills: A Systematic Review

OBJECTIVE

Self-assessment is fundamental in surgical training to enhance learning in the absence of trainer feedback. The primary objective of this review was to assess the factors that influence accuracy of self-assessment at technical skills across all surgical specialties. The secondary objective was to assess whether there are any innate factors or attributes to predict those that will carry out effective self-assessment.

DESIGN

A systematic review was carried out in accordance with PRISMA guidelines. A search strategy encompassing MEDLINE, EMBASE, ERIC, WHO, and the Cochrane database was conducted to identify studies investigating self-assessment at any surgical task. Quality was assessed using the Newcastle-Ottawa scale. A summary table was created to describe specialty, participants, task, setting, assessment tool, and correlation coefficient between self and expert assessment. The review protocol was registered in PROSPERO.

RESULTS

Of 24,638 citations, 40 met inclusion criteria. In total 1753 participants performed 68 procedures. Twenty-six studies investigated skills in general surgery with the remaining 14 in various other surgical specialties. Accuracy of self-assessment is superior in those with greater experience and age, and with use of retrospective video playback. Accuracy tends to be reflected by overestimation of performance. Stressful environments reduce accuracy. There is limited evidence in the literature regarding predicting traits for those who will carry out accurate self-assessment.

CONCLUSIONS

The ability to perform accurate self-assessment is an important skill in surgical training, with accuracy being influenced by a multitude of factors. The use of self-assessment from retrospective video playback may be of benefit in surgical training curricula to enhance learning of technical skills. Further studies are required to define predictors of good self-assessment, which will strengthen recruitment and mentoring to assist trainee learning.

Validation of Motion Tracking Software for Evaluation of Surgical Performance in Laparoscopic Cholecystectomy

Motion tracking software for assessing laparoscopic surgical proficiency has been proven to be effective in differentiating between expert and novice performances. However, with several indices that can be generated from the software, there is no set threshold that can be used to benchmark performances. The aim of this study was to identify the best possible algorithm that can be used to benchmark expert, intermediate and novice performances for objective evaluation of psychomotor skills. 12 video recordings of various surgeons were collected in a blinded fashion. Data from our previous study of 6 experts and 23 novices was also included in the analysis to determine thresholds for performance. Video recording were analyzed both by the Kinovea 0.8.15 software and a blinded expert observer using the CAT form. Multiple algorithms were tested to accurately identify expert and novice performances. ½ L + [Formula: see text] A + [Formula: see text] J scoring of path length, average movement and jerk index respectively resulted in identifying 23/24 performances. Comparing the algorithm to CAT assessment yielded in a linear regression coefficient R2 of 0.844. The value of motion tracking software in providing objective clinical evaluation and retrospective analysis is evident. Given the prospective use of this tool the algorithm developed in this study proves to be effective in benchmarking performances for psychomotor skills evaluation.

Benefits and Limitations of Transurethral Resection of the Prostate Training With a Novel Virtual Reality Simulator

PURPOSE

Profound endourological skills are required for optimal postoperative outcome parameters after transurethral resection of the prostate (TURP). We investigated the Karl Storz (Tuttlingen, Germany) UroTrainer for virtual simulation training of the TURP.

MATERIALS AND METHODS

Twenty urologists underwent a virtual reality (VR) TURP training. After a needs analysis, performance scores and self-rated surgical skills were compared before and after the curriculum, the realism of the simulator was assessed, and the optimal level of experience for VR training was evaluated. Statistical testing was done with SPSS 25.

RESULTS

Forty percent of participants indicated frequent intraoperative overload during real-life TURP and 80% indicated that VR training might be beneficial for endourological skills development, underlining the need to advance classical endourological training. For the complete cohort, overall VR performance scores (P = 0.022) and completeness of resection (P < 0.001) significantly improved. Self-rated parameters including identification of anatomical structures (P = 0.046), sparing the sphincter (P = 0.002), and handling of the resectoscope (P = 0.033) became significantly better during the VR curriculum. Participants indicated progress regarding handling of the resectoscope (70%), bleeding control (55%), and finding the correct resection depth (50%). Although overall realism and handling of the resectoscope was good, virtual bleeding control and correct tissue feedback should be improved in future VR simulators. Seventy percent of participants indicated 10 to 50 virtual TURP cases to be optimal and 80% junior residents to be the key target group for VR TURP training.

CONCLUSIONS

There is a need to improve training the TURP and VR simulators might be a valuable supplement, especially for urologists beginning with the endourological desobstruction of the prostate.

Boot Camp in a Box: Initial Experience with Pretraining Skills Preparation for New Interns

PURPOSE

In order to increase selected skills at onset of training, we provided newly matched PGY-1 trainees with materials and instructions to practice these skills, as well as the opportunity to share video-recorded performance and receive feedback based on these videos.

METHODS

Knot tying and suturing kits, instruments and supplies, and video instructions for task performance were sent to newly matched trainees to our program (n = 10), with instructions to practice 4 tasks (1- and 2-handed knot tying, interrupted and running suturing) until self-assessed comfort with each task was achieved or the 8-week time point before start of training was reached. Each trainee returned a video of each task, which was graded by blinded reviewers for time and errors using an itemized evaluation instrument (12 items for suturing and five items for knot-tying). Feedback (annotations of submitted videos) was provided after grading was completed. Task performance was repeated and reassessed at the time of new intern "Boot Camp" and again 8 weeks after start of training. Performance scores were compared for the 3 time points and with scores of PGY 2-4 residents using ANOVA with posthoc tests.

RESULTS

Compliance with instruction for practice and return of video recorded tasks in the months before start of PGY-1 training was high, with only 1 of 10 failing to return knot-tying videos. A significant pattern of performance change (p < 0.05) was observed for all tasks with an initial decrease between the pre-employment practice period and the Boot Camp test followed by an increase to the highest level of performance 2 months after start of training. At that point, scores were not significantly different than those of more senior residents.

CONCLUSIONS

A high level of compliance was achieved with requested skills practice and video documentation of performance. We attribute the consistently lower scores on the tasks during Boot Camp tests to higher stress test environment which was apt to be less favorable than having the trainee choose to submit their best possible preresidency video recording of performance in a low-stress situation. Subsequent achievement of significantly higher performance even compared to more senior residents may have been helped by incentivized pretraining practice.

Competency assessment tool for laparoscopic suturing: development and reliability evaluation

BACKGROUND

Laparoscopic suturing can be technically challenging and requires extensive training to achieve competency. To date no specific and objective assessment method for laparoscopic suturing and knot tying is available that can guide training and monitor performance in these complex surgical skills. In this study we aimed to develop a laparoscopic suturing competency assessment tool (LS-CAT) and assess its inter-observer reliability.

METHODS

We developed a bespoke CAT tool for laparoscopic suturing through a structured, mixed methodology approach, overseen by a steering committee with experience in developing surgical assessment tools. A wide Delphi consultation with over twelve experts in laparoscopic surgery guided the development stages of the tool. Following, subjects with different levels of laparoscopic expertise were included to evaluate this tool, using a simulated laparoscopic suturing task which involved placing of two surgical knots. A research assistant video recorded and anonymised each performance. Two blinded expert surgeons assessed the anonymised videos using the developed LS-CAT. The LS-CAT scores of the two experts were compared to assess the inter-observer reliability. Lastly, we compared the subjects' LS-CAT performance scores at the beginning and end of their learning curve.

RESULTS

This study evaluated a novel LS-CAT performance tool, comprising of four tasks. Thirty-six complete videos were analysed and evaluated with the LS-CAT, of which the scores demonstrated excellent inter-observer reliability. Cohen's Kappa analysis revealed good to excellent levels of agreement for almost all tasks of both instrument handling and tissue handling (0.87; 0.77; 0.75; 0.86; 0.85, all with p < 0.001). Subjects performed significantly better at the end of their learning curve compared to their first attempt for all LS-CAT items (all with p < 0.001).

CONCLUSIONS

We developed the LS-CAT, which is a laparoscopic suturing grading matrix, with excellent inter-rater reliability and to discriminate between experience levels. This LS-CAT has a potential for wider use to objectively assess laparoscopic suturing skills.

Validation of a High-End Virtual Reality Simulator for Training Transurethral Resection of Bladder Tumors

OBJECTIVE

The oncological outcome in patients with bladder cancer (BC) significantly correlates with the quality of transurethral resection of bladder tumors (TUR-BT). Virtual reality (VR) training simulators have been developed to improve surgical skills. We evaluated the advantages and limitations of the novel Uro Trainer (UT) (Karl Storz GmbH, Germany) with respect to training for TUR-BT.

DESIGN

Participants underwent VR training based on 4 different TUR-BT cases accompanied by self-assessment and evaluation questionnaires. Results were compared between experienced endourologists and novices, and furthermore, correlated with self-rated capabilities for content and construct validity. Student's t tests, Pearson's correlation, and chi-squared tests were performed for statistical evaluation.

SETTING

The study was performed at the tertiary care urological department of the Ludwig-Maximilians-University, Munich, Germany.

PARTICIPANTS

A total of 22 urological physicians, including residents and consultants, were included in the study.

RESULTS

There is a need to improve TUR-BT training as 27.3% of the participants had already experienced overtaxing situations during TUR-BT and only a few reported of high satisfaction with the classical "see one-do one" teaching mode. Construct validity was demonstrated, as consultants achieved significantly higher overall scores (p < 0.001) and safety (p = 0.004) and visualization (p = 0.001) subscores. Interestingly, the self-assessed capability to perform a TUR-BT correlated significantly (p = 0.01) with overall UT scores. Significant progress of self-rated abilities was shown for several parameters, including inspection (p = 0.046) and resection (p = 0.026). Although participants indicated improvements in several procedural skills and overall benefit of the VR training with the UT was rated 4.6 on a 5-point scale by consultants, limitations of the UT were demonstrated predominantly for tissue feedback and authenticity of different tissue layers.

CONCLUSIONS

The novel VR simulator showed a high face and construct validity, and therefore has a great potential to complement endourological training.

Video recording in ophthalmic surgery

Video recording in ophthalmic surgery has immense potential to drive quality improvement in patient care, ophthalmology training, and research. Not only do surgical videos permit introspective and critical analyses of surgical technique, they also allow for objective assessment, allow for more informative audits, and are an invaluable medium for surgical education. Unfortunately, medical-grade video recording equipment is often costly. Various novel methods of video recording that utilize commercially available products offer adequate alternatives. Certain ethical and legal issues also need to be considered before the commencement of video recording in the operating room to protect both the patient and surgeon. We review the current applications and methods of video recording in ophthalmic surgery described in the literature, as well as the potential ethical and legal issues surrounding video recording.