Assessment of virtual reality robotic simulation performance by urology resident trainees

Validation of a novel, low-fidelity virtual reality simulator and an artificial intelligence assessment approach for peg transfer laparoscopic training

Simulators are widely used in medical education, but objective and automatic assessment is not feasible with low-fidelity simulators, which can be solved with artificial intelligence (AI) and virtual reality (VR) solutions. The effectiveness of a custom-made VR simulator and an AI-based evaluator of a laparoscopic peg transfer exercise was investigated. Sixty medical students were involved in a single-blinded randomised controlled study to compare the VR simulator with the traditional box trainer. A total of 240 peg transfer exercises from the Fundamentals of Laparoscopic Surgery programme were analysed. The experts and AI-based software used the same criteria for evaluation. The algorithm detected pitfalls and measured exercise duration. Skill improvement showed no significant difference between the VR and control groups. The AI-based evaluator exhibited 95% agreement with the manual assessment. The average difference between the exercise durations measured by the two evaluation methods was 2.61 s. The duration of the algorithmic assessment was 59.47 s faster than the manual assessment. The VR simulator was an effective alternative practice compared with the training box simulator. The AI-based evaluation produced similar results compared with the manual assessment, and it could significantly reduce the evaluation time. AI and VR could improve the effectiveness of basic laparoscopic training.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Virtual reality and augmented reality in radiation oncology education – A review and expert commentary

The field of radiation oncology is rapidly advancing through technological and biomedical innovation backed by robust research evidence. However, cancer professionals are notoriously time-poor, meaning there is a need for high quality, accessible and tailored oncologic education programs. While traditional teaching methods including lectures and other in-person delivery formats remain important, digital learning (DL) has provided additional teaching options that can be delivered flexibly and on-demand from anywhere in the world.

While evidence of this digital migration has been evident for some time now, it has not always been met with the same enthusiasm by the teaching community, in part due to questions about its pedagogical effectiveness. Many of these reservations have been driven by a rudimentary utilisation of the medium and inexperience with digital best-practice. With increasing familiarity and understanding of the medium, increasingly sophisticated and pedagogically-driven learning solutions can be produced.

This article will review the application of immersive digital learning tools in radiation oncology education. This includes first and second-generation Virtual Reality (VR) environments and Augmented Reality (AR). It will explore the data behind, and best-practice application of, each of these tools as well as giving practical tips for educators who are looking to implement (or refine) their use of these learning methods. It includes a discussion of how to match the digital learning methods to the content being taught and ends with a horizon scan of where the digital medium may take us in the future. This article is the second in a two-part series, with the companion piece being on Screen-Based Digital Learning Methods in Radiation Oncology.

Overall, the digital space is well-placed to cater to the evolving educational needs of oncology learners. Further uptake over the next decade is likely to be driven by the desire for flexible on demand delivery, high-yield products, engaging delivery methods and programs that are tailored to individual learning needs. Educational programs that embrace these principles will have unique opportunities to thrive in this space.

Postgraduate education in radiation oncology during the COVID-19 pandemic - What did we learn?

Introduction

During the COVID-19 pandemic the ESTRO School who provides international non-profit postgraduate education in Radiation Oncology and related disciplines, including Medical Physics and Radiation Technology, had to close down all live educational activities and turn online, although having only limited experience. The paper describes the experience, discusses the limitations and benefits of online education and suggests directions for the future.

Materials and methods

Data about format and feedback from attendees and faculty members from the course activities held in 2019, 2020 and 2021 were made available from the ESTRO School.

Results

In 2020, all but two out of thirty live courses that happened before the lockdown were canceled. Among the 18 courses scheduled in the second half of the year, seven went online with a short notice. Each course planned their activities quite differently, from compressed courses with consecutive full days online program to courses over several weeks with a few hours online a week. Both numbers of participants and different nationalities were higher than live courses in 2019 for the seven courses happening online, and courses were well evaluated by participants and faculties. Roughly-one-third of participants would prefer online courses in the future.

Discussion

Although online education was well received by the majority, pros and cons exist and especially the personal discussions and networking were missed. Online education and live education are not comparable but can complement each other. Careful balancing these activities in the future is important and strategies for online andragogy are needed.

Non-technical skills in robotic surgery and impact on near-miss events: a multi-center study

BACKGROUND

Robotic surgery requires a set of non-technical skills (NTS), because of the complex environment. We aim to study relationship between NTS and near-miss events in robotic surgery.

METHODS

This is an observational study in five French centers. Three robotic procedures were observed and filmed by one of expert trainers in NTS. They established and scored a non-technical skills in robotic surgery (NTSRS) score, that included eight items, each scored from 1 to 5, to assess the whole surgical teams. The surgical teams also self-assessed their work. The number of near-miss events was recorded and classified as minor, or major but no harm incidents, independently by two surgeons. Correlations were Spearman coefficients.

RESULTS

Of the 26 procedures included, 15 were prostatectomy (58%), 9 nephrectomy (35%), and 2 pyeloplasty (7.7%). Half of procedures (n = 13) were performed by surgeons with extensive RS experience (more than 150 procedures). Per procedure, there was a median (quartiles) of 9 (7; 11) near-miss events. There was 1 (0; 2) major near-miss events, with no harm. The median NTSRS score was 18 (14; 21), out of 40. The number of near-miss events was strongly correlated with the NTSRS score (r = - 0.92, p < 0.001) but was not correlated with the surgeon's experience. The surgeons for fifteen (58%) procedures, and the bed-side surgeons for 11 (42%) procedures, felt that there was no need for an improvement in the quality of their NTS. None of the surgeons gave a negative self-evaluation for any procedure; in three procedures (12%), the bed-side surgeons self-assessed negatively, on ergonomics.

CONCLUSION

Occurrence of near-miss events was reduced in teams managing NTS. Specific NTS surgical team training is essential for robotic surgery as it may have a significant impact on risk management.

Developing a five-step training model for transperineal prostate biopsies in a naïve residents' group: a prospective observational randomised study of two different techniques

PURPOSE

To evaluate a five-step training model for transperineal prostate biopsies (TPPB) and the differences in terms of the detection rate (DR) and the ease of execution when using either the "fan technique" (FT) or the use of a Free Hand technique (FH).

METHODS

A prospective observational randomised study was conducted from September 2015 to November 2017. Six naïve residents, who underwent the same five-steps training model, were randomly subdivided into two different groups of three residents based on the selected TPPB technique: A (FT) and B (FH). Patient characteristics (age, PSA, prostatic volume, DRE, MRI), intraoperative (operative time, number of samples) and postoperative parameters (histologic, pain) were evaluated in the 2 groups. The overall and stratified DR for PSA ranges and prostate volume (PV), operative time and complications were compared.

RESULTS

The overall detection rate was very high in both groups (FT 58.2% vs FH 59.6%) and not statistically different between the two techniques. There were no differences in terms of complication rates and pain. The FH showed a better detection rate in prostates smaller than 40 cc (p = 0.023) and a faster operative time (p = 0.025) compared to FT.

CONCLUSIONS

Within the TPPB, FH is associated with a higher detection rate in patients with prostate < 40 cc compared to an FT when performed by inexperienced trainees. Standardised training organised in consecutive steps seems to contribute to the achievement of overall high detection rates with both methods.

Implementing assessments of robot-assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

OBJECTIVES

To systematically review and synthesise the validity evidence supporting intraoperative and simulation-based assessments of technical skill in urological robot-assisted surgery (RAS), and make evidence-based recommendations for the implementation of these assessments in urological training.

MATERIALS AND METHODS

A literature search of the Medline, PsycINFO and Embase databases was performed. Articles using technical skill and simulation-based assessments in RAS were abstracted. Only studies involving urology trainees or faculty were included in the final analysis.

RESULTS

Multiple tools for the assessment of technical robotic skill have been published, with mixed sources of validity evidence to support their use. These evaluations have been used in both the ex vivo and in vivo settings. Performance evaluations range from global rating scales to psychometrics, and assessments are carried out through automation, expert analysts, and crowdsourcing.

CONCLUSION

There have been rapid expansions in approaches to RAS technical skills assessment, both in simulated and clinical settings. Alternative approaches to assessment in RAS, such as crowdsourcing and psychometrics, remain under investigation. Evidence to support the use of these metrics in high-stakes decisions is likely insufficient at present.

Automatically rating trainee skill at a pediatric laparoscopic suturing task

BACKGROUND

Minimally invasive surgeons must acquire complex technical skills while minimizing patient risk, a challenge that is magnified in pediatric surgery. Trainees need realistic practice with frequent detailed feedback, but human grading is tedious and subjective. We aim to validate a novel motion-tracking system and algorithms that automatically evaluate trainee performance of a pediatric laparoscopic suturing task.

METHODS

Subjects (n = 32) ranging from medical students to fellows performed two trials of intracorporeal suturing in a custom pediatric laparoscopic box trainer after watching a video of ideal performance. The motions of the tools and endoscope were recorded over time using a magnetic sensing system, and both tool grip angles were recorded using handle-mounted flex sensors. An expert rated the 63 trial videos on five domains from the Objective Structured Assessment of Technical Skill (OSATS), yielding summed scores from 5 to 20. Motion data from each trial were processed to calculate 280 features. We used regularized least squares regression to identify the most predictive features from different subsets of the motion data and then built six regression tree models that predict summed OSATS score. Model accuracy was evaluated via leave-one-subject-out cross-validation.

RESULTS

The model that used all sensor data streams performed best, achieving 71% accuracy at predicting summed scores within 2 points, 89% accuracy within 4, and a correlation of 0.85 with human ratings. 59% of the rounded average OSATS score predictions were perfect, and 100% were within 1 point. This model employed 87 features, including none based on completion time, 77 from tool tip motion, 3 from tool tip visibility, and 7 from grip angle.

CONCLUSIONS

Our novel hardware and software automatically rated previously unseen trials with summed OSATS scores that closely match human expert ratings. Such a system facilitates more feedback-intensive surgical training and may yield insights into the fundamental components of surgical skill.

Impact of delay on telesurgical performance: study on the robotic simulator dV-Trainer

PURPOSE

To determine the impact of communication latency on telesurgical performance using the robotic simulator dV-Trainer®.

METHODS

Surgeons were enrolled during three robotic congresses. They were randomly assigned to a delay group (ranging from 100 to 1000 ms). Each group performed three times a set of four exercises on the simulator: the first attempt without delay (Base) and the last two attempts with delay (Warm-up and Test). The impact of different levels of latency was evaluated.

RESULTS

Thirty-seven surgeons were involved. The different latency groups achieved similar baseline performance with a mean task completion time of 207.2 s (p > 0.05). In the Test stage, the task duration increased gradually from 156.4 to 310.7 s as latency increased from 100 to 500 ms. In separate groups, the task duration deteriorated from Base for latency stages at delays ≥300 ms, and the errors increased at 500 ms and above (p < 0.05). The subjects' performance tended to improve from the Warm-up to the Test period. Few subjects completed the tasks with a delay higher than 700 ms.

CONCLUSION

Gradually increasing latency has a growing impact on performances. Measurable deterioration of performance begins at 300 ms. Delays higher than 700 ms are difficult to manage especially in more complex tasks. Surgeons showed the potential to adapt to delay and may be trained to improve their telesurgical performance at lower-latency levels.