1
|
Olsen RG, Svendsen MBS, Tolsgaard MG, Konge L, Røder A, Bjerrum F. Automated performance metrics and surgical gestures: two methods for assessment of technical skills in robotic surgery. J Robot Surg 2024; 18:297. [PMID: 39068261 PMCID: PMC11283394 DOI: 10.1007/s11701-024-02051-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2024] [Accepted: 07/15/2024] [Indexed: 07/30/2024]
Abstract
The objective of this study is to compare automated performance metrics (APM) and surgical gestures for technical skills assessment during simulated robot-assisted radical prostatectomy (RARP). Ten novices and six experienced RARP surgeons performed simulated RARPs on the RobotiX Mentor (Surgical Science, Sweden). Simulator APM were automatically recorded, and surgical videos were manually annotated with five types of surgical gestures. The consequences of the pass/fail levels, which were based on contrasting groups' methods, were compared for APM and surgical gestures. Intra-class correlation coefficient (ICC) analysis and a Bland-Altman plot were used to explore the correlation between APM and surgical gestures. Pass/fail levels for both APM and surgical gesture could fully distinguish between the skill levels of the surgeons with a specificity and sensitivity of 100%. The overall ICC (one-way, random) was 0.70 (95% CI: 0.34-0.88), showing moderate agreement between the methods. The Bland-Altman plot showed a high agreement between the two methods for assessing experienced surgeons but disagreed on the novice surgeons' skill level. APM and surgical gestures could both fully distinguish between novices and experienced surgeons in a simulated setting. Both methods of analyzing technical skills have their advantages and disadvantages and, as of now, those are only to a limited extent available in the clinical setting. The development of assessment methods in a simulated setting enables testing before implementing it in a clinical setting.
Collapse
Affiliation(s)
- Rikke Groth Olsen
- Copenhagen Academy for Medical Education and Simulation (CAMES), Ryesgade 53B, 2100, Copenhagen, Denmark.
- Department of Urology, Copenhagen Prostate Cancer Center, Copenhagen University Hospital-Rigshospitalet, Copenhagen, Denmark.
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark.
| | - Morten Bo Søndergaard Svendsen
- Copenhagen Academy for Medical Education and Simulation (CAMES), Ryesgade 53B, 2100, Copenhagen, Denmark
- Department of Computer Science, University of Copenhagen, Copenhagen, Denmark
| | - Martin G Tolsgaard
- Copenhagen Academy for Medical Education and Simulation (CAMES), Ryesgade 53B, 2100, Copenhagen, Denmark
| | - Lars Konge
- Copenhagen Academy for Medical Education and Simulation (CAMES), Ryesgade 53B, 2100, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Andreas Røder
- Department of Urology, Copenhagen Prostate Cancer Center, Copenhagen University Hospital-Rigshospitalet, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Flemming Bjerrum
- Copenhagen Academy for Medical Education and Simulation (CAMES), Ryesgade 53B, 2100, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
- Gastrounit, Surgical Section, Copenhagen University Hospital-Amager and Hvidovre, Hvidovre, Denmark
| |
Collapse
|
2
|
van Leeuwen FWB, Buckle T, van Oosterom MN, Rietbergen DDD. The Rise of Molecular Image-Guided Robotic Surgery. J Nucl Med 2024:jnumed.124.267783. [PMID: 38991755 DOI: 10.2967/jnumed.124.267783] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Accepted: 06/05/2024] [Indexed: 07/13/2024] Open
Abstract
Following early acceptance by urologists, the use of surgical robotic platforms is rapidly spreading to other surgical fields. This empowerment of surgical perception via robotic advances occurs in parallel to developments in intraoperative molecular imaging. Convergence of these efforts creates a logical incentive to advance the decades-old image-guided robotics paradigm. This yields new radioguided surgery strategies set to optimally exploit the symbiosis between the growing clinical translation of robotics and molecular imaging. These strategies intend to advance surgical precision by increasing dexterity and optimizing surgical decision-making. In this state-of-the-art review, topic-related developments in chemistry (tracer development) and engineering (medical device development) are discussed, and future scientific robotic growth markets for molecular imaging are presented.
Collapse
Affiliation(s)
- Fijs W B van Leeuwen
- Interventional Molecular Imaging Laboratory, Leiden University Medical Center, Leiden, The Netherlands; and
| | - Tessa Buckle
- Interventional Molecular Imaging Laboratory, Leiden University Medical Center, Leiden, The Netherlands; and
| | - Matthias N van Oosterom
- Interventional Molecular Imaging Laboratory, Leiden University Medical Center, Leiden, The Netherlands; and
| | - Daphne D D Rietbergen
- Interventional Molecular Imaging Laboratory, Leiden University Medical Center, Leiden, The Netherlands; and
- Section of Nuclear Medicine, Department of Radiology, Leiden University Medical Center, Leiden, The Netherlands
| |
Collapse
|
3
|
Shafiei SB, Shadpour S, Mohler JL, Rashidi P, Toussi MS, Liu Q, Shafqat A, Gutierrez C. Prediction of Robotic Anastomosis Competency Evaluation (RACE) metrics during vesico-urethral anastomosis using electroencephalography, eye-tracking, and machine learning. Sci Rep 2024; 14:14611. [PMID: 38918593 PMCID: PMC11199555 DOI: 10.1038/s41598-024-65648-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Accepted: 06/21/2024] [Indexed: 06/27/2024] Open
Abstract
Residents learn the vesico-urethral anastomosis (VUA), a key step in robot-assisted radical prostatectomy (RARP), early in their training. VUA assessment and training significantly impact patient outcomes and have high educational value. This study aimed to develop objective prediction models for the Robotic Anastomosis Competency Evaluation (RACE) metrics using electroencephalogram (EEG) and eye-tracking data. Data were recorded from 23 participants performing robot-assisted VUA (henceforth 'anastomosis') on plastic models and animal tissue using the da Vinci surgical robot. EEG and eye-tracking features were extracted, and participants' anastomosis subtask performance was assessed by three raters using the RACE tool and operative videos. Random forest regression (RFR) and gradient boosting regression (GBR) models were developed to predict RACE scores using extracted features, while linear mixed models (LMM) identified associations between features and RACE scores. Overall performance scores significantly differed among inexperienced, competent, and experienced skill levels (P value < 0.0001). For plastic anastomoses, R2 values for predicting unseen test scores were: needle positioning (0.79), needle entry (0.74), needle driving and tissue trauma (0.80), suture placement (0.75), and tissue approximation (0.70). For tissue anastomoses, the values were 0.62, 0.76, 0.65, 0.68, and 0.62, respectively. The models could enhance RARP anastomosis training by offering objective performance feedback to trainees.
Collapse
Affiliation(s)
- Somayeh B Shafiei
- Intelligent Cancer Care Laboratory, Department of Urology, Roswell Park Comprehensive Cancer Center, Elm and Carlton Streets, Buffalo, NY, 14263, USA.
| | - Saeed Shadpour
- Department of Animal Biosciences, University of Guelph, Guelph, ON, N1G 2W1, Canada
| | - James L Mohler
- Department of Urology, Roswell Park Comprehensive Cancer Center, Buffalo, NY, 14263, USA
| | - Parisa Rashidi
- Department of Biomedical Engineering, University of Florida, Gainesville, FL, 32611, USA
| | - Mehdi Seilanian Toussi
- Intelligent Cancer Care Laboratory, Department of Urology, Roswell Park Comprehensive Cancer Center, Elm and Carlton Streets, Buffalo, NY, 14263, USA
| | - Qian Liu
- Department of Biostatistics and Bioinformatics, Roswell Park Comprehensive Cancer Center, Buffalo, NY, USA
| | - Ambreen Shafqat
- Intelligent Cancer Care Laboratory, Department of Urology, Roswell Park Comprehensive Cancer Center, Elm and Carlton Streets, Buffalo, NY, 14263, USA
| | - Camille Gutierrez
- Obstetrics and Gynecology Residency Program, Sisters of Charity Health System, Buffalo, NY, 14214, USA
| |
Collapse
|
4
|
Boal M, Di Girasole CG, Tesfai F, Morrison TEM, Higgs S, Ahmad J, Arezzo A, Francis N. Evaluation status of current and emerging minimally invasive robotic surgical platforms. Surg Endosc 2024; 38:554-585. [PMID: 38123746 PMCID: PMC10830826 DOI: 10.1007/s00464-023-10554-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 10/20/2023] [Indexed: 12/23/2023]
Abstract
BACKGROUND The rapid adoption of robotics within minimally invasive surgical specialties has also seen an explosion of new technology including multi- and single port, natural orifice transluminal endoscopic surgery (NOTES), endoluminal and "on-demand" platforms. This review aims to evaluate the validation status of current and emerging MIS robotic platforms, using the IDEAL Framework. METHODS A scoping review exploring robotic minimally invasive surgical devices, technology and systems in use or being developed was performed, including general surgery, gynaecology, urology and cardiothoracics. Systems operating purely outside the abdomen or thorax and endoluminal or natural orifice platforms were excluded. PubMed, Google Scholar, journal reports and information from the public domain were collected. Each company was approached via email for a virtual interview to discover more about the systems and to quality check data. The IDEAL Framework is an internationally accepted tool to evaluate novel surgical technology, consisting of four stages: idea, development/exploration, assessment, and surveillance. An IDEAL stage, synonymous with validation status in this review, was assigned by reviewing the published literature. RESULTS 21 companies with 23 different robotic platforms were identified for data collection, 13 with national and/or international regulatory approval. Of the 17 multiport systems, 1 is fully evaluated at stage 4, 2 are stage 3, 6 stage 2b, 2 at stage 2a, 2 stage 1, and 4 at the pre-IDEAL stage 0. Of the 6 single-port systems none have been fully evaluated with 1 at stage 3, 3 at stage 1 and 2 at stage 0. CONCLUSIONS The majority of existing robotic platforms are currently at the preclinical to developmental and exploratory stage of evaluation. Using the IDEAL framework will ensure that emerging robotic platforms are fully evaluated with long-term data, to inform the surgical workforce and ensure patient safety.
Collapse
Affiliation(s)
- M Boal
- The Griffin Institute, Northwick Park and St Marks Hospital, London, UK
- Wellcome/EPSRC Centre for Intervention and Surgical Sciences, University College London, London, UK
- Association of Laparoscopic Surgeons of Great Britain and Ireland (ALSGBI) Academy, London, UK
| | | | - F Tesfai
- The Griffin Institute, Northwick Park and St Marks Hospital, London, UK
- Wellcome/EPSRC Centre for Intervention and Surgical Sciences, University College London, London, UK
- Association of Laparoscopic Surgeons of Great Britain and Ireland (ALSGBI) Academy, London, UK
| | - T E M Morrison
- Association of Laparoscopic Surgeons of Great Britain and Ireland (ALSGBI) Academy, London, UK
| | - S Higgs
- Gloucestershire Hospitals NHS Foundation Trust, Gloucester, UK
| | - J Ahmad
- University Hospitals Coventry and Warwickshire, Coventry, UK
| | - A Arezzo
- Department of Surgical Sciences, University of Turin, Turin, Italy
| | - N Francis
- The Griffin Institute, Northwick Park and St Marks Hospital, London, UK.
- Yeovil District Hospital, Somerset NHS Foundation Trust, Yeovil, UK.
| |
Collapse
|
5
|
Olsen RG, Svendsen MBS, Tolsgaard MG, Konge L, Røder A, Bjerrum F. Surgical gestures can be used to assess surgical competence in robot-assisted surgery : A validity investigating study of simulated RARP. J Robot Surg 2024; 18:47. [PMID: 38244130 PMCID: PMC10799775 DOI: 10.1007/s11701-023-01807-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Accepted: 12/23/2023] [Indexed: 01/22/2024]
Abstract
To collect validity evidence for the assessment of surgical competence through the classification of general surgical gestures for a simulated robot-assisted radical prostatectomy (RARP). We used 165 video recordings of novice and experienced RARP surgeons performing three parts of the RARP procedure on the RobotiX Mentor. We annotated the surgical tasks with different surgical gestures: dissection, hemostatic control, application of clips, needle handling, and suturing. The gestures were analyzed using idle time (periods with minimal instrument movements) and active time (whenever a surgical gesture was annotated). The distribution of surgical gestures was described using a one-dimensional heat map, snail tracks. All surgeons had a similar percentage of idle time but novices had longer phases of idle time (mean time: 21 vs. 15 s, p < 0.001). Novices used a higher total number of surgical gestures (number of phases: 45 vs. 35, p < 0.001) and each phase was longer compared with those of the experienced surgeons (mean time: 10 vs. 8 s, p < 0.001). There was a different pattern of gestures between novices and experienced surgeons as seen by a different distribution of the phases. General surgical gestures can be used to assess surgical competence in simulated RARP and can be displayed as a visual tool to show how performance is improving. The established pass/fail level may be used to ensure the competence of the residents before proceeding with supervised real-life surgery. The next step is to investigate if the developed tool can optimize automated feedback during simulator training.
Collapse
Affiliation(s)
- Rikke Groth Olsen
- Copenhagen Academy for Medical Education and Simulation (CAMES), Center for HR & Education, The Capital Region of Denmark, Ryesgade 53B, 2100, Copenhagen, Denmark.
- Department of Urology, Copenhagen Prostate Cancer Center, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark.
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark.
| | - Morten Bo Søndergaard Svendsen
- Copenhagen Academy for Medical Education and Simulation (CAMES), Center for HR & Education, The Capital Region of Denmark, Ryesgade 53B, 2100, Copenhagen, Denmark
- Department of Computer Science, University of Copenhagen, Copenhagen, Denmark
| | - Martin G Tolsgaard
- Copenhagen Academy for Medical Education and Simulation (CAMES), Center for HR & Education, The Capital Region of Denmark, Ryesgade 53B, 2100, Copenhagen, Denmark
| | - Lars Konge
- Copenhagen Academy for Medical Education and Simulation (CAMES), Center for HR & Education, The Capital Region of Denmark, Ryesgade 53B, 2100, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Andreas Røder
- Department of Urology, Copenhagen Prostate Cancer Center, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Flemming Bjerrum
- Copenhagen Academy for Medical Education and Simulation (CAMES), Center for HR & Education, The Capital Region of Denmark, Ryesgade 53B, 2100, Copenhagen, Denmark
- Department of Gastrointestinal and Hepatic Diseases, Copenhagen University Hospital - Herlev and Gentofte, Herlev, Denmark
| |
Collapse
|
6
|
Boal MWE, Anastasiou D, Tesfai F, Ghamrawi W, Mazomenos E, Curtis N, Collins JW, Sridhar A, Kelly J, Stoyanov D, Francis NK. Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review. Br J Surg 2024; 111:znad331. [PMID: 37951600 PMCID: PMC10771126 DOI: 10.1093/bjs/znad331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Revised: 09/18/2023] [Accepted: 09/19/2023] [Indexed: 11/14/2023]
Abstract
BACKGROUND There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula. METHODS A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias. RESULTS Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent. CONCLUSIONS Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.
Collapse
Affiliation(s)
- Matthew W E Boal
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
| | - Dimitrios Anastasiou
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Medical Physics and Biomedical Engineering, UCL, London, UK
| | - Freweini Tesfai
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
| | - Walaa Ghamrawi
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
| | - Evangelos Mazomenos
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Medical Physics and Biomedical Engineering, UCL, London, UK
| | - Nathan Curtis
- Department of General Surgey, Dorset County Hospital NHS Foundation Trust, Dorchester, UK
| | - Justin W Collins
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- University College London Hospitals NHS Foundation Trust, London, UK
| | - Ashwin Sridhar
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- University College London Hospitals NHS Foundation Trust, London, UK
| | - John Kelly
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- University College London Hospitals NHS Foundation Trust, London, UK
| | - Danail Stoyanov
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Computer Science, UCL, London, UK
| | - Nader K Francis
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- Yeovil District Hospital, Somerset Foundation NHS Trust, Yeovil, Somerset, UK
| |
Collapse
|
7
|
Gorard J, Boal M, Swamynathan V, Ghamrawi W, Francis N. The application of objective clinical human reliability analysis (OCHRA) in the assessment of basic robotic surgical skills. Surg Endosc 2024; 38:116-128. [PMID: 37932602 PMCID: PMC10776495 DOI: 10.1007/s00464-023-10510-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Accepted: 10/01/2023] [Indexed: 11/08/2023]
Abstract
BACKGROUND Using a validated, objective, and standardised assessment tool to assess progression and competency is essential for basic robotic surgical training programmes. Objective clinical human reliability analysis (OCHRA) is an error-based assessment tool that provides in-depth analysis of individual technical errors. We conducted a feasibility study to assess the concurrent validity and reliability of OCHRA when applied to basic, generic robotic technical skills assessment. METHODS Selected basic robotic surgical skill tasks, in virtual reality (VR) and dry lab equivalent, were performed by novice robotic surgeons during an intensive 5-day robotic surgical skills course on da Vinci® X and Xi surgical systems. For each task, we described a hierarchical task analysis. Our developed robotic surgical-specific OCHRA methodology was applied to error events in recorded videos with a standardised definition. Statistical analysis to assess concurrent validity with existing tools and inter-rater reliability were performed. RESULTS OCHRA methodology was applied to 272 basic robotic surgical skills tasks performed by 20 novice robotic surgeons. Performance scores improved from the start of the course to the end using all three assessment tools; Global Evaluative Assessment of Robotic Skills (GEARS) [VR: t(19) = - 9.33, p < 0.001] [dry lab: t(19) = - 10.17, p < 0.001], OCHRA [VR: t(19) = 6.33, p < 0.001] [dry lab: t(19) = 10.69, p < 0.001] and automated VR [VR: t(19) = - 8.26, p < 0.001]. Correlation analysis, for OCHRA compared to GEARS and automated VR scores, shows a significant and strong inverse correlation in every VR and dry lab task; OCHRA vs GEARS [VR: mean r = - 0.78, p < 0.001] [dry lab: mean r = - 0.82, p < 0.001] and OCHRA vs automated VR [VR: mean r = - 0.77, p < 0.001]. There is very strong and significant inter-rater reliability between two independent reviewers (r = 0.926, p < 0.001). CONCLUSION OCHRA methodology provides a detailed error analysis tool in basic robotic surgical skills with high reliability and concurrent validity with existing tools. OCHRA requires further evaluation in more advanced robotic surgical procedures.
Collapse
Affiliation(s)
- Jack Gorard
- Division of Surgery & Interventional Science, Royal Free Hospital Campus, University College London, London, UK
| | - Matthew Boal
- Division of Surgery & Interventional Science, Royal Free Hospital Campus, University College London, London, UK
- The Griffin Institute, Northwick Park and St Mark's Hospital, London, UK
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, Charles Bell House, University College London, London, UK
| | - Vishaal Swamynathan
- Division of Surgery & Interventional Science, Royal Free Hospital Campus, University College London, London, UK
| | - Walaa Ghamrawi
- Division of Surgery & Interventional Science, Royal Free Hospital Campus, University College London, London, UK
- The Griffin Institute, Northwick Park and St Mark's Hospital, London, UK
| | - Nader Francis
- Division of Surgery & Interventional Science, Royal Free Hospital Campus, University College London, London, UK.
- The Griffin Institute, Northwick Park and St Mark's Hospital, London, UK.
| |
Collapse
|
8
|
Nakamura W, Sumitomo M, Zennami K, Takenaka M, Ichino M, Takahara K, Teramoto A, Shiroki R. Combination of deep learning and ensemble machine learning using intraoperative video images strongly predicts recovery of urinary continence after robot-assisted radical prostatectomy. Cancer Rep (Hoboken) 2023; 6:e1861. [PMID: 37449339 PMCID: PMC10480482 DOI: 10.1002/cnr2.1861] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Revised: 06/20/2023] [Accepted: 06/25/2023] [Indexed: 07/18/2023] Open
Abstract
BACKGROUND We recently reported the importance of deep learning (DL) of pelvic magnetic resonance imaging in predicting the degree of urinary incontinence (UI) following robot-assisted radical prostatectomy (RARP). However, our results were limited because the prediction accuracy was approximately 70%. AIM To develop a more precise prediction model that can inform patients about UI recovery post-RARP surgery using a DL model based on intraoperative video images. METHODS AND RESULTS The study cohort comprised of 101 patients with localized prostate cancer undergoing RARP. Three snapshots from intraoperative video recordings showing the pelvic cavity (prior to bladder neck incision, immediately following prostate removal, and after vesicourethral anastomosis) were evaluated, including pre- and intraoperative parameters. We evaluated the DL model plus simple or ensemble machine learning (ML), and the area under the receiver operating characteristic curve (AUC) was analyzed through sensitivity and specificity. Of 101, 64 and 37 patients demonstrated "early continence (using 0 or 1 safety pad at 3 months post-RARP)" and "late continence (others)," respectively, at 3 months postoperatively. The combination of DL and simple ML using intraoperative video snapshots with clinicopathological parameters had a notably high performance (AUC, 0.683-0.749) to predict early recovery from UI after surgery. Furthermore, combining DL with ensemble artificial neural network using intraoperative video snapshots had the highest performance (AUC, 0.882; sensitivity, 92.2%; specificity, 78.4%; overall accuracy, 85.3%) to predict early recovery from post-RARP incontinence, with similar results by internal validation. The addition of clinicopathological parameters showed no additive effects for each analysis using DL, EL and simple ML. CONCLUSION Our findings suggest that the DL algorithm with intraoperative video imaging is a reliable method for informing patients about the severity of their recovery from UI after RARP, although it is not clear if our methods are reproducible for predicting long-term UI and pad-free continence.
Collapse
Affiliation(s)
- Wataru Nakamura
- Department of Urology, School of MedicineFujita Health UniversityToyoakeJapan
| | - Makoto Sumitomo
- Department of Urology, School of MedicineFujita Health UniversityToyoakeJapan
- Fujita Cancer CenterFujita Health UniversityToyoakeJapan
| | - Kenji Zennami
- Department of Urology, School of MedicineFujita Health UniversityToyoakeJapan
| | - Masashi Takenaka
- Department of Urology, School of MedicineFujita Health UniversityToyoakeJapan
| | - Manabu Ichino
- Department of Urology, School of MedicineFujita Health UniversityToyoakeJapan
| | - Kiyoshi Takahara
- Department of Urology, School of MedicineFujita Health UniversityToyoakeJapan
| | - Atsushi Teramoto
- Faculty of Radiological Technology, School of Medical SciencesFujita Health UniversityToyoakeJapan
- Faculty of Information EngineeringMeijo UniversityNagoyaJapan
| | - Ryoichi Shiroki
- Department of Urology, School of MedicineFujita Health UniversityToyoakeJapan
| |
Collapse
|
9
|
Ma R, Cen S, Forsyth E, Probst P, Asghar A, Townsend W, Hui A, Desai A, Tzeng M, Cheng E, Ramaswamy A, Wagner C, Hu JC, Hung AJ. Technical surgical skill assessment of neurovascular bundle dissection and urinary continence recovery after robotic-assisted radical prostatectomy. JU OPEN PLUS 2023; 1:e00039. [PMID: 38187460 PMCID: PMC10768840 DOI: 10.1097/ju9.0000000000000035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/09/2024]
Abstract
Purpose To examine the association between the quality of neurovascular bundle dissection and urinary continence recovery after robotic-assisted radical prostatectomy. Materials and Methods Patients who underwent RARPs from 2016 to 2018 in two institutions with ≥1-year postoperative follow-up were included. The primary outcomes were time to urinary continence recovery. Surgical videos were independently assessed by 3 blinded raters using the validated Dissection Assessment for Robotic Technique (DART) tool after standardized training. Cox regression was used to test the association between DART scores and urinary continence recovery while adjusting for relevant patient features. Results 121 RARP performed by 23 surgeons with various experience levels were included. The median follow-up was 24 months (95% CI 20 - 28 months). The median time to continence recovery was 7.3 months (95% CI 4.7 - 9.8 months). After adjusting for patient age, higher scores of certain DART domains, specifically tissue retraction and efficiency, were significantly associated with increased odds of continence recovery (p<0.05). Conclusions Technical skill scores of neurovascular bundle dissection vary among surgeons and correlate with urinary continence recovery. Unveiling the specific robotic dissection skillsets which impact patient outcomes has the potential to focus surgical training.
Collapse
Affiliation(s)
- Runzhuo Ma
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, University of Southern California, Los Angeles, California
| | - Steven Cen
- Department of Radiology, University of Southern California, Los Angeles, California
| | - Edward Forsyth
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, University of Southern California, Los Angeles, California
| | - Patrick Probst
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, University of Southern California, Los Angeles, California
| | - Aeen Asghar
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, University of Southern California, Los Angeles, California
| | - William Townsend
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, University of Southern California, Los Angeles, California
| | - Alvin Hui
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, University of Southern California, Los Angeles, California
| | - Aditya Desai
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, University of Southern California, Los Angeles, California
| | - Michael Tzeng
- Department of Urology, Weill Cornell Medicine, New York, New York
| | - Emily Cheng
- Department of Urology, Weill Cornell Medicine, New York, New York
| | - Ashwin Ramaswamy
- Department of Urology, Weill Cornell Medicine, New York, New York
| | - Christian Wagner
- Department of Urology and Urologic Oncology, St. Antonius-Hospital, Gronau, Germany
| | - Jim C. Hu
- Department of Urology, Weill Cornell Medicine, New York, New York
| | - Andrew J. Hung
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, University of Southern California, Los Angeles, California
| |
Collapse
|
10
|
Liu Z, Bible J, Petersen L, Zhang Z, Roy-Chaudhury P, Singapogu R. Relating process and outcome metrics for meaningful and interpretable cannulation skill assessment: A machine learning paradigm. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 236:107429. [PMID: 37119772 PMCID: PMC10291517 DOI: 10.1016/j.cmpb.2023.107429] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Revised: 02/06/2023] [Accepted: 02/15/2023] [Indexed: 05/21/2023]
Abstract
BACKGROUND AND OBJECTIVES The quality of healthcare delivery depends directly on the skills of clinicians. For patients on hemodialysis, medical errors or injuries caused during cannulation can lead to adverse outcomes, including potential death. To promote objective skill assessment and effective training, we present a machine learning approach, which utilizes a highly-sensorized cannulation simulator and a set of objective process and outcome metrics. METHODS In this study, 52 clinicians were recruited to perform a set of pre-defined cannulation tasks on the simulator. Based on data collected by sensors during their task performance, the feature space was then constructed based on force, motion, and infrared sensor data. Following this, three machine learning models- support vector machine (SVM), support vector regression (SVR), and elastic net (EN)- were constructed to relate the feature space to objective outcome metrics. Our models utilize classification based on the conventional skill classification labels as well as a new method that represents skill on a continuum. RESULTS With less than 5% of trials misplaced by two classes, the SVM model was effective in predicting skill based on the feature space. In addition, the SVR model effectively places both skill and outcome on a fine-grained continuum (versus discrete divisions) that is representative of reality. As importantly, the elastic net model enabled the identification of a set of process metrics that highly impact outcomes of the cannulation task, including smoothness of motion, needle angles, and pinch forces. CONCLUSIONS The proposed cannulation simulator, paired with machine learning assessment, demonstrates definite advantages over current cannulation training practices. The methods presented here can be adopted to drastically increase the effectiveness of skill assessment and training, thereby potentially improving clinical outcomes of hemodialysis treatment.
Collapse
Affiliation(s)
- Zhanhe Liu
- Department of Bioengineering, Clemson University, 301 Rhodes Research Center, Clemson, 29634, SC, USA
| | - Joe Bible
- School of Mathematical and Statistical Sciences, Clemson University, O-110 Martin Hall, Clemson, 29634, SC, USA
| | - Lydia Petersen
- Department of Bioengineering, Clemson University, 301 Rhodes Research Center, Clemson, 29634, SC, USA
| | - Ziyang Zhang
- Department of Bioengineering, Clemson University, 301 Rhodes Research Center, Clemson, 29634, SC, USA
| | - Prabir Roy-Chaudhury
- UNC Kidney Center, University of North Carolina, Chapel Hill, NC, 28144, USA; (Bill Hefner) VA Medical Center, Salisbury, NC, 28144, USA
| | - Ravikiran Singapogu
- Department of Bioengineering, Clemson University, 301 Rhodes Research Center, Clemson, 29634, SC, USA.
| |
Collapse
|
11
|
Nyangoh Timoh K, Huaulme A, Cleary K, Zaheer MA, Lavoué V, Donoho D, Jannin P. A systematic review of annotation for surgical process model analysis in minimally invasive surgery based on video. Surg Endosc 2023:10.1007/s00464-023-10041-w. [PMID: 37157035 DOI: 10.1007/s00464-023-10041-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2022] [Accepted: 03/25/2023] [Indexed: 05/10/2023]
Abstract
BACKGROUND Annotated data are foundational to applications of supervised machine learning. However, there seems to be a lack of common language used in the field of surgical data science. The aim of this study is to review the process of annotation and semantics used in the creation of SPM for minimally invasive surgery videos. METHODS For this systematic review, we reviewed articles indexed in the MEDLINE database from January 2000 until March 2022. We selected articles using surgical video annotations to describe a surgical process model in the field of minimally invasive surgery. We excluded studies focusing on instrument detection or recognition of anatomical areas only. The risk of bias was evaluated with the Newcastle Ottawa Quality assessment tool. Data from the studies were visually presented in table using the SPIDER tool. RESULTS Of the 2806 articles identified, 34 were selected for review. Twenty-two were in the field of digestive surgery, six in ophthalmologic surgery only, one in neurosurgery, three in gynecologic surgery, and two in mixed fields. Thirty-one studies (88.2%) were dedicated to phase, step, or action recognition and mainly relied on a very simple formalization (29, 85.2%). Clinical information in the datasets was lacking for studies using available public datasets. The process of annotation for surgical process model was lacking and poorly described, and description of the surgical procedures was highly variable between studies. CONCLUSION Surgical video annotation lacks a rigorous and reproducible framework. This leads to difficulties in sharing videos between institutions and hospitals because of the different languages used. There is a need to develop and use common ontology to improve libraries of annotated surgical videos.
Collapse
Affiliation(s)
- Krystel Nyangoh Timoh
- Department of Gynecology and Obstetrics and Human Reproduction, CHU Rennes, Rennes, France.
- INSERM, LTSI - UMR 1099, University Rennes 1, Rennes, France.
- Laboratoire d'Anatomie et d'Organogenèse, Faculté de Médecine, Centre Hospitalier Universitaire de Rennes, 2 Avenue du Professeur Léon Bernard, 35043, Rennes Cedex, France.
- Department of Obstetrics and Gynecology, Rennes Hospital, Rennes, France.
| | - Arnaud Huaulme
- INSERM, LTSI - UMR 1099, University Rennes 1, Rennes, France
| | - Kevin Cleary
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, 20010, USA
| | - Myra A Zaheer
- George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| | - Vincent Lavoué
- Department of Gynecology and Obstetrics and Human Reproduction, CHU Rennes, Rennes, France
| | - Dan Donoho
- Division of Neurosurgery, Center for Neuroscience, Children's National Hospital, Washington, DC, 20010, USA
| | - Pierre Jannin
- INSERM, LTSI - UMR 1099, University Rennes 1, Rennes, France
| |
Collapse
|
12
|
Younes MM, Larkins K, To G, Burke G, Heriot A, Warrier S, Mohan H. What are clinically relevant performance metrics in robotic surgery? A systematic review of the literature. J Robot Surg 2022; 17:335-350. [PMID: 36190655 PMCID: PMC10076398 DOI: 10.1007/s11701-022-01457-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2022] [Accepted: 09/17/2022] [Indexed: 10/10/2022]
Abstract
A crucial element of any surgical training program is the ability to provide procedure-specific, objective, and reliable measures of performance. During robotic surgery, objective clinically relevant performance metrics (CRPMs) can provide tailored contextual feedback and correlate with clinical outcomes. This review aims to define CRPMs, assess their validity in robotic surgical training and compare CRPMs to existing measures of robotic performance. A systematic search of Medline and Embase databases was conducted in May 2022 following the PRISMA guidelines. The search terms included Clinically Relevant Performance Metrics (CRPMs) OR Clinically Relevant Outcome Measures (CROMs) AND robotic surgery. The study settings, speciality, operative context, study design, metric details, and validation status were extracted and analysed. The initial search yielded 116 citations, of which 6 were included. Citation searching identified 3 additional studies, resulting in 9 studies included in this review. Metrics were defined as CRPMs, CROMs, proficiency-based performance metrics and reference-procedure metrics which were developed using a modified Delphi methodology. All metrics underwent both contents and construct validation. Two studies found a strong correlation with GEARS but none correlated their metrics with patient outcome data. CRPMs are a validated and objective approach for assessing trainee proficiency. Evaluating CRPMs with other robotic-assessment tools will facilitate a multimodal metric evaluation approach to robotic surgery training. Further studies should assess the correlation with clinical outcomes. This review highlights there is significant scope for the development and validation of CRPMs to establish proficiency-based progression curricula that can be translated from a simulation setting into clinical practice.
Collapse
Affiliation(s)
- Melissa M Younes
- The University of Melbourne, 305 Grattan Street, Parkville, VIC, Australia
| | - Kirsten Larkins
- The University of Melbourne, 305 Grattan Street, Parkville, VIC, Australia. .,Peter MacCallum Cancer Centre, Melbourne, VIC, Australia.
| | - Gloria To
- The University of Melbourne, 305 Grattan Street, Parkville, VIC, Australia
| | - Grace Burke
- International Medical Robotics Academy, North Melbourne, VIC, Australia
| | - Alexander Heriot
- The University of Melbourne, 305 Grattan Street, Parkville, VIC, Australia.,Peter MacCallum Cancer Centre, Melbourne, VIC, Australia.,International Medical Robotics Academy, North Melbourne, VIC, Australia
| | - Satish Warrier
- The University of Melbourne, 305 Grattan Street, Parkville, VIC, Australia.,Peter MacCallum Cancer Centre, Melbourne, VIC, Australia.,International Medical Robotics Academy, North Melbourne, VIC, Australia.,Monash University, Clayton, VIC, Australia
| | - Helen Mohan
- The University of Melbourne, 305 Grattan Street, Parkville, VIC, Australia.,Peter MacCallum Cancer Centre, Melbourne, VIC, Australia.,Austin Health, Heidelberg, VIC, Australia
| |
Collapse
|
13
|
Inouye DA, Ma R, Nguyen JH, Laca J, Kocielnik R, Anandkumar A, Hung AJ. Assessing the efficacy of dissection gestures in robotic surgery. J Robot Surg 2022; 17:597-603. [PMID: 36149590 DOI: 10.1007/s11701-022-01458-x] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2022] [Accepted: 09/17/2022] [Indexed: 10/14/2022]
Abstract
Our group previously defined a dissection gesture classification system that deconstructs robotic tissue dissection into its most elemental yet meaningful movements. The purpose of this study was to expand upon this framework by adding an assessment of gesture efficacy (ineffective, effective, or erroneous) and analyze dissection patterns between groups of surgeons of varying experience. We defined three possible gesture efficacies as ineffective (no meaningful effect on the tissue), effective (intended effect on the tissue), and erroneous (unintended disruption of the tissue). Novices (0 prior robotic cases), intermediates (1-99 cases), and experts (≥ 100 cases) completed a robotic dissection task in a dry-lab training environment. Video recordings were reviewed to classify each gesture and determine its efficacy, then dissection patterns between groups were analyzed. 23 participants completed the task, with 9 novices, 8 intermediates with median caseload 60 (IQR 41-80), and 6 experts with median caseload 525 (IQR 413-900). For gesture selection, we found increasing experience associated with increasing proportion of overall dissection gestures (p = 0.009) and decreasing proportion of retraction gestures (p = 0.009). For gesture efficacy, novices performed the greatest proportion of ineffective gestures (9.8%, p < 0.001), intermediates commit the greatest proportion of erroneous gestures (26.8%, p < 0.001), and the three groups performed similar proportions of overall effective gestures, though experts performed the greatest proportion of effective retraction gestures (85.6%, p < 0.001). Between groups of experience, we found significant differences in gesture selection and gesture efficacy. These relationships may provide insight into further improving surgical training.
Collapse
Affiliation(s)
- Daniel A Inouye
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, University of Southern California Institute of Urology, Los Angeles, CA, USA
| | - Runzhuo Ma
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, University of Southern California Institute of Urology, Los Angeles, CA, USA
| | - Jessica H Nguyen
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, University of Southern California Institute of Urology, Los Angeles, CA, USA
| | - Jasper Laca
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, University of Southern California Institute of Urology, Los Angeles, CA, USA
| | - Rafal Kocielnik
- Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA
| | - Anima Anandkumar
- Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA
| | - Andrew J Hung
- Center for Robotic Simulation & Education, Catherine & Joseph Aresty Department of Urology, University of Southern California Institute of Urology, Los Angeles, CA, USA.
| |
Collapse
|
14
|
Dasgupta P. Re: Artificial Intelligence for Diagnosis and Gleason Grading of Prostate Cancer: The PANDA Challenge. Eur Urol 2022; 82:571. [PMID: 35961871 DOI: 10.1016/j.eururo.2022.07.028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Accepted: 07/26/2022] [Indexed: 11/04/2022]
Affiliation(s)
- Prokar Dasgupta
- King's Health Partners, Guy's Hospital, King's College London, London, UK.
| |
Collapse
|
15
|
Berniker M, Bhattacharyya KD, Brown KC, Jarc A. A Probabilistic Approach To Surgical Tasks and Skill Metrics. IEEE Trans Biomed Eng 2021; 69:2212-2219. [PMID: 34971527 DOI: 10.1109/tbme.2021.3139538] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Identifying and quantifying the activities that compose surgery is essential for effective interventions, computer-aided analyses and the advancement of surgical data science. For example, recent studies have shown that objective metrics (referred to as objective performance indicators, OPIs) computed during key surgical tasks correlate with surgeon skill and clinical outcomes. Unambiguous identification of these surgical tasks can be particularly challenging for both human annotators and algorithms. Each surgical procedure has multiple approaches, each surgeon has their own level of skill, and the initiation and termination of surgical tasks can be subject to interpretation. As such, human annotators and machine learning models face the same basic problem, accurately identifying the boundaries of surgical tasks despite variable and unstructured information. For use in surgeon feedback, OPIs should also be robust to the variability and diversity in this data. To mitigate this difficulty, we propose a probabilistic approach to surgical task identification and calculation of OPIs. Rather than relying on tasks that are identified by hard temporal boundaries, we demonstrate an approach that relies on distributions of start and stop times, for a probabilistic interpretation of when the task was performed. We first use hypothetical data to outline how this approach is superior to other conventional approaches. Then we present similar analyses on surgical data. We find that when surgical tasks are identified by their individual probabilities, the resulting OPIs are less sensitive to noise in the identification of the start and stop times. These results suggest that this probabilistic approach holds promise for the future of surgical data science.
Collapse
|