Objective Assessment and Standard Setting for Basic Flexible Ureterorenoscopy Skills Among Urology Trainees Using Simulation-Based Methods

Medical Students Cannot Assess Robotic Surgeons Performing Radical Prostatectomy

INTRODUCTION

Medical students have previously been shown to be just as effective for video rating as experts. We want to compare medical students to experienced surgeons as video assessors of simulated robot-assisted radical prostatectomy (RARP) performance.

MATERIALS AND METHODS

Video recordings of three RARP modules on the RobotiX (formerly Simbionix) simulator from a previous study were used. Five novice surgeons, five experienced robotic surgeons, and five experienced robotic surgeons in RARP performed a total of 45 video-recorded procedures. The videos were assessed with the modified Global Evaluative Assessment of Robotic Skills tool as both full-length and an edited edition that only included the first 5 minutes of the procedure.

RESULTS

Fifty medical students and two experienced RARP surgeons (ES) performed a total of 680 video ratings of full-length videos and 5-minute videos (2-9 ratings per video). Medical students and ES showed poor agreement for both full-length videos and 5-minute videos (0.29 and -0.13, respectively). Medical students could not discriminate between the skill level of the surgeons in either full-length videos or 5-minute videos ( P = 0.053-0.36 and P = 0.21-0.82), whereas ES could discriminate between novice surgeons and experienced surgeons (full-length, P < 0.001, and 5 minutes, P = 0.007) and intermediate and experienced surgeons (full-length, P = 0.001, and 5 minutes, P = 0.01) in both full-length videos and 5-minute videos.

CONCLUSION

We found that medical students cannot be used to assess RARP because they showed poor agreement with the ES rating for both full-length videos and 5-minute videos. Medical students could not discriminate between surgical skill levels.

Development of an automated composite ureteroscopic efficiency score (CUES) through simulated ureteroscopic skills assessment

INTRODUCTION

Flexible ureteroscopy (fURS) is the most common procedure for treatment of urolithiasis. We previously utilized kinematic evaluations of simulated fURS to demonstrate certain body movements are associated with efficient ureteroscopic manipulation for complex tasks. Herein, we incorporated computer vision to create an efficiency score using ureteroscope travel distance (DIST), task time (TIME), spectral arc length (SPARC), and percentage of purposeful wall collisions (COLL). The goal is a simulation-based system that can abstract these automated performance metrics (APMs) to differentiate between novice and expert ureteroscopic handling.

METHODS

A ureteroscopic simulation box was used. Body kinematics, task time, and ureteroscopic movements were analyzed using a motion capture system and video camera. Optical flow computer vision was used to track the ureteroscope. DIST, TIME, and SPARC were automatically calculated. Wall collisions were automatically captured and independently judged by two authors; an algorithm was developed to automatically determine the COLL variable. A mixed effects model was used to aggregate these variables and distinguish between surgeons' first and final task attempts. The normalized values of these metrics were added to create a Composite Ureteroscopy Efficiency Score (CUES).

RESULTS

12 urologists completed the simulated tasks. The COLL assessment algorithm determined beneficial wall collisions with an accuracy of 77%. Normalized values of TIME, DIST, SPARC, and COLL were combined to create a composite ureteroscopic efficiency score (CUES). Compared to the first attempt, both the second and third attempts had statistically significant improvements in CUES. The ROC-AUC score reached 0.86, suggesting excellent discrimination between attempts. There was also a statistically significant difference in CUES when comparing resident and attending performance.

CONCLUSIONS

Automated performance metrics can be abstracted using computer vision and artificial intelligence; an aggregate composite score (CUES) may be a promising method for evaluation of ureteroscopic efficiency.

FUNDING

Endoscopic equipment provided by Boston Scientific.

Eye gaze metrics for skill assessment and feedback in kidney stone surgery

PURPOSE

Surgical skill assessment is essential for safe operations. In endoscopic kidney stone surgery, surgeons must perform a highly skill-dependent mental mapping from the pre-operative scan to the intraoperative endoscope image. Poor mental mapping can lead to incomplete exploration of the kidney and high reoperation rates. Yet there are few objective ways to evaluate competency. We propose to use unobtrusive eye-gaze measurements in the task space to evaluate skill and provide feedback.

METHODS

We capture the surgeons' eye gaze on the surgical monitor with the Microsoft Hololens 2. To enable stable and accurate gaze detection, we develop a calibration algorithm to refine the eye tracking of the Hololens. In addition, we use a QR code to locate the eye gaze on the surgical monitor. We then run a user study with three expert and three novice surgeons. Each surgeon is tasked to locate three needles representing kidney stones in three different kidney phantoms.

RESULTS

We find that experts have more focused gaze patterns. They complete the task faster, have smaller total gaze area, and the gaze fewer times outside the area of interest. While fixation to non-fixation ratio did not show significant difference in our findings, tracking the ratio over time shows different patterns between novices and experts.

CONCLUSION

We show that a non-negligible difference holds between novice and expert surgeons' gaze metrics in kidney stone identification in phantoms. Expert surgeons demonstrate more targeted gaze throughout a trial, indicating their higher level of proficiency. To improve the skill acquisition process for novice surgeons, we suggest providing sub-task specific feedback. This approach presents an objective and non-invasive method to assess surgical competence.

Analysis of Flexible Ureteroscopic Motion and Kinematic Efficiency - a Simulation-based Pilot Study

INTRODUCTION

Flexible ureteroscopy (fURS) is the most common surgical procedure for treatment of urolithiasis. Various surgical disciplines and subspecialties have examined surgeon kinematics to improve assessment and generate measures of skill. Despite frequency of utilization, there is no undisputed method for evaluating fURS skills. Our pilot study utilized kinematic evaluations of fURS simulation to determine whether specific surgeon movements, techniques, and strategies correlate with measures of ureteroscopic efficiency.

METHODS

A motion capture system and standard video camera were employed to characterize surgeon movement variables. A ureteroscopic simulation box was used by practicing urologists at various skill levels to perform a series of simple and complex ureteroscopic movement tasks. Two tasks were chosen for this initial pilot analysis. Body kinematics, time to task completion, and ureteroscopic movements were analyzed. Task efficiency was defined as quicker time to task completion and smaller ureteroscope end effector travel distance. A combined performance efficiency score (PES) was calculated using the root sum square of these two measures.

RESULTS

Twelve practicing urologists were enlisted. Average urologist age was 37 years with an average of 10.1 years of training; 50% were female, 50% were residents; and 33% had completed an Endourology fellowship. For the simple task, no kinematic data correlated with PES; for the complex task, participant head and torso movement correlated with PES (r=0.60, p=0.04 for head; r=0.65, p=0.02 for torso), with decreased body movement associated with higher efficiency.

CONCLUSION

Our findings suggest that movement economy measures are associated with efficient ureteroscopic manipulation for complex tasks. Decreased head and torso movement were associated with higher efficiency, suggesting that excess body movement may signal extraneous or improper ureteroscopic movements. Additional assessment of these variables, including analysis in a clinical setting, is warranted as this may serve as a basis for improvement in endoscopic training and evaluation.

Are there predictors of flexible ureteroscopic aptitude among novice trainees? objective assessment using simulation-based trainer

PURPOSE

Non-surgical skills involving hand-eye coordination and bimanual dexterity may have a transferable impact on the acquisition of ureteroscopy skills. In this study, we aim to investigate the predictors of initial flexible ureteroscopy skills among novice trainees.

METHODS

This was a prospective study involving students with no prior ureteroscopy exposure. Non-surgical parameters were assessed with a detailed survey, including demographics, video game, and musical history. Musical ability was objectively evaluated with the mini-Profile of Music Perception Skills test. Ureteroscopic performance was evaluated using a Boston Scientific^© flexible ureteroscope on a bench model. Each participant completed diagnostic ureteroscopy and stone extraction. Outcomes included both speed and quality of performance, based on an Objective Structured Assessment of Technical Skills rubric.

RESULTS

A total of 28 pre-clerkship medical students and 10 urology residents were included. Age and musical background were not associated with ureteroscopic aptitude. Those with video game history tended to perform ureteroscopy tasks faster with a higher OSATS score, although no statistical significance was reached. Male gender was associated with faster task completion with statistically higher OSATS score independent of video game activities (p = 0.011), however, the absolute score difference was small.

CONCLUSIONS

Among novice trainees, musical and video game experience was not predictive of ureteroscopy skills. Male gender was associated with slightly faster and higher ureteroscopy technique scores, however, the differences are small and unlikely to represent clinical significance. Nevertheless, the use of ureteroscopy trainer provides useful insights and should be adopted in training programs as a marker of skills progression.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.