Development and validation of the TOCO–TURBT tool: a summative assessment tool that measures surgical competency in transurethral resection of bladder tumour

Current Standards for Training in Robot-assisted Surgery and Endourology: A Systematic Review

BACKGROUND AND OBJECTIVE

Different training programs have been developed to improve trainee outcomes in urology. However, evidence on the optimal training methodology is sparse. Our aim was to provide a comprehensive description of the training programs available for urological robotic surgery and endourology, assess their validity, and highlight the fundamental elements of future training pathways.

METHODS

We systematically reviewed the literature using PubMed/Medline, Embase, and Web of Science databases. The validity of each training model was assessed. The methodological quality of studies on metrics and curricula was graded using the MERSQI scale. The level of evidence (LoE) and level of recommendation for surgical curricula were awarded using the educational Oxford Centre for Evidence-Based Medicine classification.

KEY FINDINGS AND LIMITATIONS

A total of 75 studies were identified. Many simulators have been developed to aid trainees in mastering skills required for both robotic and endourology procedures, but only four demonstrated predictive validity. For assessment of trainee proficiency, we identified 18 in robotics training and six in endourology training; however, the majority are Likert-type scales. Although proficiency-based progression (PBP) curricula demonstrated superior outcomes to traditional training in preclinical settings, only four of six (67%) in robotics and three of nine (33%) in endourology are PBP-based. Among these, the Fundamentals of Robotic Surgery and the SIMULATE curricula have the highest LoE (level 1b). The lack of a quantitative synthesis is the main limitation of our study.

CONCLUSIONS AND CLINICAL IMPLICATIONS

Training curricula that integrate simulators and PBP methodology have been introduced to standardize trainee outcomes in robotics and endourology. However, evidence regarding their educational impact remains restricted to preclinical studies. Efforts should be made to expand these training programs to different surgical procedures and assess their clinical impact.

PATIENT SUMMARY

Simulation-based training and programs in which progression is based on proficiency represent the new standard of quality for achieving surgical proficiency in urology. Studies have demonstrated the educational impact of these approaches. However, there are still no standardized training pathways for several urology procedures.

Transurethral resection of bladder tumor: A systematic review of simulator-based training courses and curricula

Objective

Transurethral resection of bladder tumor is one of the most common everyday urological procedures. This kind of surgery demands a set of skills that need training and experience. In this review, we aimed to investigate the current literature to find out if simulators, phantoms, and other training models could be used as a tool for teaching urologists.

Methods

A systematic review was performed according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses statement and the recommendations of the European Association of Urology guidelines for conducting systematic reviews. Fifteen out of 932 studies met our inclusion criteria and are presented in the current review.

Results

The UroTrainer (Karl Storz GmbH, Tuttlingen, Germany), a virtual reality training simulator, achieved positive feedback and an excellent face and construct validity by the participants. The inspection of bladder mucosa, blood loss, tumor resection, and procedural time was improved after the training, especially for inexperienced urologists and medical students. The construct validity of UroSim® (VirtaMed, Zurich, Switzerland) was established. SIMBLA simulator (Samed GmbH, Dresden, Germany) was found to be a realistic and useful tool by experts and urologists with intermediate experience. The test objective competency model based on SIMBLA simulator could be used for evaluating urologists. The porcine model of the Asian Urological Surgery Training and Education Group also received positive feedback by the participants that tried it. The Simulation and Technology Enhanced Learning Initiative Project had an extraordinary face and content validity, and 60% of participants would like to use the simulators in the future. The 5-day multimodal training curriculum "Boot Camp" in the United Kingdom achieved an increase of the level of confidence of the participants that lasted months after the project.

Conclusion

Simulators and courses or curricula based on a simulator training could be a valuable learning tool for any surgeon, and there is no doubt that they should be a part of every urologist's technical education.

Development and Application of a Novel and Efficient Skills Assessment Tool: A Pilot Initiative to Measure Vasectomy Competency on a Smartphone

OBJECTIVE

To develop and evaluate a mobile phone-based skills assessment tool that measures procedural competency of urology residents learning to perform a common, non-robotic urology procedure as a means of tracking current skillset and improvement over time.

METHODS

The assessment tool was a Qualtrics survey accessed via a smartphone link that breaks down a vasectomy into 6 critical steps. Level of competency was measured on a scale of '1-novice' to '5-expert.' Nine residents from Post graduate year (PGY)-1 to PGY-5 were evaluated by one instructor after completing a vasectomy (86 single-side cases recorded over a 6-month period). We compared individual trainees to each other, analyzed performance (improvement) over time, and evaluated competency against cohort and program averages.

RESULTS

As an example, a single resident ('Resident 2,' N = 11 cases) was compared to cohort (PGY, M = 7.5/resident) and program (all residents, M = 7.4/resident). Results indicate similar skillfulness across Step 1 (puncturing and isolation of vas and hand positioning; P > 0.1), but marginally lower competency on Step 2 (opening of vasal sheath to expose/isolate vas; vs. cohort: P = 0.076, vs. residents: P = 0.082). Significantly lower competency on Steps 3-6 (all P < 0.04) suggests targeted teaching could improve cautery technique, fascial interposition, hemostasis, and positioning of stumps.

CONCLUSION

Our mobile-based skills assessment is a low cost, novel, and efficient assessment that would support current Accreditation Council for Graduate Medical Education (ACGME) goals to increase competency-based residency training. This tool is easily created and accessed, provides real-time feedback to learners, and can be used for individual and group assessment at a single timepoint or longitudinally.

First Characterization of Resident Clinical Experience at American Urological Training Programs

OBJECTIVE

To provide the first nationwide characterization of the clinical learning environment in American urological training programs.

METHODS

A survey was sent to program directors (PD) at AUA-accredited urological training programs after requesting their email address from each program coordinator (PC). The 21-question survey was designed to ascertain key components of each training environment: demographics, training model, clinic structure, and resident perception.

RESULTS

The PC of 131 AUA-accredited training programs received an email for participation, yielding the PD email for 113 programs. 60/113 (53%) PDs responded to the survey. Residents participated in clinic at the following types of hospitals: Children's 51 (85%), County/Indigent 23 (38%), Private 29 (48%), University 56 (93%), Veterans Administration 38 (63%). Prevalence of clinical training models is presented in table 1. On average, PDs estimated their residents spend 2.6 half days in clinic each week (1-6). 13 (22%) programs reported a "clinic only" rotation, varying from 1-6 months total. PDs reported time constraint and schedule to be the biggest barrier to teaching in clinic and 40% felt residents see clinic as a valuable part of their training while 30% felt residents see clinic as a necessary exercise but with limitations to learning opportunities.

CONCLUSIONS

We present the first characterization of resident participation in the clinical learning environment. Structure is highly variable and directed effort is necessary to move towards improved assessment and monitoring of resident competency in clinic.

Validation of a novel assessment tool identifying proficiency in Transurethral Bladder Tumour Resection: The OSATURBS assessment tool

BACKGROUND

Competence in transurethral bladder tumour resection (TURB) is critical in bladder cancer management and should be ensured before independent practice.

OBJECTIVE

Develop an assessment tool for TURB and explore validity evidence in a clinical context.

DESIGN, SETTING, AND PARTICIPANTS

July 2019-March 2021, a total of 33 volunteer doctors from three hospitals were included. Participants performed two TURB procedures on patients with bladder tumours. A newly developed assessment tool (OSATURBS) was used for direct observation assessment, self-assessment, and blinded video-assessment. Outcome measurements and statistical analysis: Cronbach's alpha and Pearson's r were calculated for across items internal consistency reliability, inter-rater reliability, and test-retest reliability. Correlation between OSATURBS scores and the operative experience was calculated with Pearson's r and a pass/fail score was established. Differences in assessment scores were explored with paired t-test and independent samples t-test.

RESULTS AND LIMITATIONS

The internal consistency reliability across items Cronbach's alpha was 0.94 (n = 260, p < 0.001). Inter-rater reliability = 0.80 (n = 64, p < 0.001). Test-retest correlation was high, r = 0.71 (n = 32, p < 0.001). Relation to TURB experience was high, r = 0.71 (n = 32, p < 0.001). Pass/fail score = 19 points. Direct observation assessments were strongly correlated with video ratings (r = 0.85, p < 0.001) but with a significant social bias with lower scores for inexperienced and higher scores for experienced participants. Participants tended to overestimate their own performances.

CONCLUSIONS

OSATURBS assessment tool for TURB can be used for assessment of surgical proficiency in the clinical setting. Direct observation assessment and self-assessment are biased, and blinded video-assessment of TURB performances is advised.

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

PURPOSE

Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD

In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS

Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS

G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Innovations in Urologic Surgical Training

PURPOSE OF REVIEW

This review aims to summarize innovations in urologic surgical training in the past 5 years.

RECENT FINDINGS

Many assessment tools have been developed to objectively evaluate surgical skills and provide structured feedback to urologic trainees. A variety of simulation modalities (i.e., virtual/augmented reality, dry-lab, animal, and cadaver) have been utilized to facilitate the acquisition of surgical skills outside the high-stakes operating room environment. Three-dimensional printing has been used to create high-fidelity, immersive dry-lab models at a reasonable cost. Non-technical skills such as teamwork and decision-making have gained more attention. Structured surgical video review has been shown to improve surgical skills not only for trainees but also for qualified surgeons. Research and development in urologic surgical training has been active in the past 5 years. Despite these advances, there is still an unfulfilled need for a standardized surgical training program covering both technical and non-technical skills.

Global Assessment of Urological Endoscopic Skills (GAUES): development and validation of a novel assessment tool to evaluate endourological skills

OBJECTIVE

To develop and evaluate an assessment tool for endourological skills during simulation including cystoscopy, ureteroscopy (URS) and transurethral resection (TUR) procedures.

METHODS

We designed a Global Assessment of Urological Endoscopic Skills (GAUES) tool, comprised of nine endourology task-specific and two global-rating skills items. The tool was developed through two rounds of the Delphi process. The GAUES tool was used to assess acquisition of URS and TUR skills of novices (Year 2 core surgical trainees, CT2) and intermediate level trainees (residents at the start of the UK higher surgical training programme in Urology, Speciality Trainee Year 3, ST3) at the Urology Simulation Boot Camp (USBC) between 2016 and 2018. Validity was evaluated by comparing scores between trainees with different levels of urological experience. Inter-rater reliability was also assessed.

RESULTS

We evaluated 130 residents, 52% of trainees were at an intermediate stage of training and 39% were novices. In all, 9% of the anonymous forms were missing demographics. The completion rate of the GAUES tool during the USBC for URS and TUR was 85% and 89%, respectively. Our analysis demonstrated a significant difference in all domains between intermediates and novices at assessment in URS, except for one domain more suited to clinical assessment (P = 0.226). There was excellent intraclass correlation (ICC) overall between the two experts' judgements, ICC = 0.841 (95% confidence interval 0.767-0.893; P < 0.001, n = 88).

CONCLUSIONS

We have developed the novel GAUES tool for cystoscopic, URS and TUR skills. Overall, we demonstrated good face, content and construct validity and excellent reliability, suggesting that the GAUES tool can be useful for endourological skills assessment.

The Learning Curve for Transurethral Resection of Bladder Tumour: How Many is Enough to be Independent, Safe and Effective Surgeon?

OBJECTIVE

Transurethral resection of the bladder tumour (TURBT) is one the most common urological procedures. It is also one the fundamental surgeries performed by residents. The learning curve (LC) for TUR has never been analysed. The aim of the study was to analyse the learning curve of TURBT in a residency setting.

DESIGN, SETTING AND PARTICIPANTS

This retrospective multicentre analysis of prospectively maintained databases enrolled 993 consecutive TURBTs performed by 10 urology residents in 3 academic institutions. Study end-points were as follows: the absence of muscularis propria in a specimen, any intra- or postoperative surgical complication and 3-month recurrence-free survival.

RESULTS

With increasing experience, residents operated more complex cases defined by higher rate of large, multifocal or high-risk tumours. In the same time, surgery time, postoperative catheterization time and hospital stay became shorter. An improvement has been noticed regarding the muscularis propria sampling and 3-month recurrence-free survival, but not regarding the risk of surgical complications. Evident improvement in study end-points was noticed after 101 operations; surgeons achieved the best clinical outcomes after performing 170 procedures, whereas the poorest results for the first 45 operations.

CONCLUSIONS

TURBT has a flat LC with 100 cases being the absolute minimum for a resident in training to achieve acceptable oncological and surgical outcomes.

Non-technical skills: a review of training and evaluation in urology

PURPOSE

With non-technical skills (NTS) deficits being recognised as a major cause for error in surgery, there is an increasing interest in their training and evaluation. A growing number of training courses are emerging and some NTS curricula have also been created. Many different training methods are described in the literature but there is still uncertainty with regards to their optimum combination within a curriculum.

METHODS

A literature review of the electronic database Medline was performed. All articles published before December 2018 were screened by abstract and included if deemed relevant by the author. The included articles' reference lists were also screened for further relevant studies.

RESULTS

Simulation training is accepted as the most effective way to train NTS. Within simulation training, it is shown that the 'igloo' full immersion/distributed simulation environment is appropriate for teaching NTS in urological scenarios where a designated operating room or space is not available. The use of multiple settings, for example wards and clinics as well as the operating room, is advantageous, as is training in an interprofessional team. Classroom teaching also plays a role in NTS training as an adjunct to simulation, with evidence that it improves some parameters of NTS. All levels, including qualified surgeons, benefit from NTS training; however, adaptation to both trainee level and specialty is important. Although less time consuming, training juniors and seniors together mainly benefits juniors, and training NTS at the same time as technical skills detracts from the quality of teaching. Debriefing is an important part of training and should be well structured; there are many debriefing models in existence, allowing for choice of method based on examiner preference and participant demographic. Furthermore, examiners should be well briefed in their task and trained in NTS assessment.

CONCLUSION

To move forward, studies should combine tried and tested learning techniques into a curriculum covering all training levels, which should then be validated and followed up long term to ensure a positive impact on patient safety.

An Opportunity for Cognitive Task Analysis in Neonatal Resuscitation

Approximately 10% of newborn infants require resuscitative intervention at birth. Ideally, this care is provided by a team of expert healthcare professionals who possess exceptional cognitive, psychomotor, and communication skills. Human errors and deviations from resuscitation protocol are common and may be attributable to excessive cognitive demand experienced by the resuscitation team. Cognitive Task Analysis (CTA) is a group of methods used to assess knowledge, judgments, goals, and decision-making of expert healthcare professionals. These methods may be used during neonatal resuscitation to gain an improved understanding of the approaches used by healthcare professionals. CTA methods have been applied in many medical disciplines including neonatology. CTA has been used to identify information previously confined to the intuition of experts. This information has been used to assess, develop, and improve medical technology, clinical decision support tools (DSTs), communication structure, and training methods. Knowledge attained through CTA might be applied similarly to neonatal resuscitation, which may in turn decrease human errors, and improve patient safety.