Improving Augmented Reality Through Deep Learning: Real-time Instrument Delineation in Robotic Renal Surgery

A decade of progress: bringing mixed reality image-guided surgery systems in the operating room

Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.

Towards safer robot-assisted surgery: A markerless augmented reality framework

Robot-assisted surgery is rapidly developing in the medical field, and the integration of augmented reality shows the potential to improve the operation performance of surgeons by providing more visual information. In this paper, we proposed a markerless augmented reality framework to enhance safety by avoiding intra-operative bleeding, which is a high risk caused by collision between surgical instruments and delicate blood vessels (arteries or veins). Advanced stereo reconstruction and segmentation networks are compared to find the best combination to reconstruct the intra-operative blood vessel in 3D space for registration with the pre-operative model, and the minimum distance detection between the instruments and the blood vessel is implemented. A robot-assisted lymphadenectomy is emulated on the da Vinci Research Kit in a dry lab, and ten human subjects perform this operation to explore the usability of the proposed framework. The result shows that the augmented reality framework can help the users to avoid the dangerous collision between the instruments and the delicate blood vessel while not introducing an extra load. It provides a flexible framework that integrates augmented reality into the medical robotic platform to enhance safety during surgery.

Visual extended reality tools in image-guided surgery in urology: a systematic review

PURPOSE

The aim of this systematic review is to assess the clinical implications of employing various Extended Reality (XR) tools for image guidance in urological surgery.

METHODS

In June 2023, a systematic electronic literature search was conducted using the Medline database (via PubMed), Embase (via Ovid), Scopus, and Web of Science. The search strategy was designed based on the PICO (Patients, Intervention, Comparison, Outcome) criteria. Study protocol was registered on PROSPERO (registry number CRD42023449025). We incorporated retrospective and prospective comparative studies, along with single-arm studies, which provided information on the use of XR, Mixed Reality (MR), Augmented Reality (AR), and Virtual Reality (VR) in urological surgical procedures. Studies that were not written in English, non-original investigations, and those involving experimental research on animals or cadavers were excluded from our analysis. The quality assessment of comparative and cohort studies was conducted utilizing the Newcastle-Ottawa scale, whilst for randomized controlled trials (RCTs), the Jadad scale was adopted. The level of evidence for each study was determined based on the guidelines provided by the Oxford Centre for Evidence-Based Medicine.

RESULTS

The initial electronic search yielded 1,803 papers after removing duplicates. Among these, 58 publications underwent a comprehensive review, leading to the inclusion of 40 studies that met the specified criteria for analysis. 11, 20 and 9 studies tested XR on prostate cancer, kidney cancer and miscellaneous, including bladder cancer and lithiasis surgeries, respectively. Focusing on the different technologies 20, 15 and 5 explored the potential of VR, AR and MR. The majority of the included studies (i.e., 22) were prospective non-randomized, whilst 7 and 11 were RCT and retrospective studies respectively. The included studies that revealed how these new tools can be useful both in preoperative and intraoperative setting for a tailored surgical approach.

CONCLUSIONS

AR, VR and MR techniques have emerged as highly effective new tools for image-guided surgery, especially for urologic oncology. Nevertheless, the complete clinical advantages of these innovations are still in the process of evaluation.

Virtual Reality Head-Mounted Display (HMD) and Preoperative Patient-Specific Simulation: Impact on Decision-Making in Pediatric Urology: Preliminary Data

AIM OF THE STUDY

To assess how virtual reality (VR) patient-specific simulations can support decision-making processes and improve care in pediatric urology, ultimately improving patient outcomes.

PATIENTS AND METHODS

Children diagnosed with urological conditions necessitating complex procedures were retrospectively reviewed and enrolled in the study. Patient-specific VR simulations were developed with medical imaging specialists and VR technology experts. Routine CT images were utilized to create a VR environment using advanced software platforms. The accuracy and fidelity of the VR simulations was validated through a multi-step process. This involved comparing the virtual anatomical models to the original medical imaging data and conducting feedback sessions with pediatric urology experts to assess VR simulations' realism and clinical relevance.

RESULTS

A total of six pediatric patients were reviewed. The median age of the participants was 5.5 years (IQR: 3.5-8.5 years), with an equal distribution of males and females across both groups. A minimally invasive laparoscopic approach was performed for adrenal lesions (n = 3), Wilms' tumor (n = 1), bilateral nephroblastomatosis (n = 1), and abdominal trauma in complex vascular and renal malformation (ptotic and hypoplastic kidney) (n = 1). Key benefits included enhanced visualization of the segmental arteries and the deep vascularization of the kidney and adrenal glands in all cases. The high depth perception and precision in the orientation of the arteries and veins to the parenchyma changed the intraoperative decision-making process in five patients. Preoperative VR patient-specific simulation did not offer accuracy in studying the pelvic and calyceal anatomy.

CONCLUSIONS

VR patient-specific simulations represent an empowering tool in pediatric urology. By leveraging the immersive capabilities of VR technology, preoperative planning and intraoperative navigation can greatly impact surgical decision-making. As we continue to advance in medical simulation, VR holds promise in educational programs to include even surgical treatment of more complex urogenital malformations.

Privacy-proof Live Surgery Streaming: Development and Validation of a Low-cost, Real-time Robotic Surgery Anonymization Algorithm

OBJECTIVE

Develop a pioneer surgical anonymization algorithm for reliable and accurate real-time removal of out-of-body images validated across various robotic platforms.

BACKGROUND

The use of surgical video data has become a common practice in enhancing research and training. Video sharing requires complete anonymization, which, in the case of endoscopic surgery, entails the removal of all nonsurgical video frames where the endoscope can record the patient or operating room staff. To date, no openly available algorithmic solution for surgical anonymization offers reliable real-time anonymization for video streaming, which is also robotic-platform and procedure-independent.

METHODS

A data set of 63 surgical videos of 6 procedures performed on four robotic systems was annotated for out-of-body sequences. The resulting 496.828 images were used to develop a deep learning algorithm that automatically detected out-of-body frames. Our solution was subsequently benchmarked against existing anonymization methods. In addition, we offer a postprocessing step to enhance the performance and test a low-cost setup for real-time anonymization during live surgery streaming.

RESULTS

Framewise anonymization yielded a receiver operating characteristic area under the curve score of 99.46% on unseen procedures, increasing to 99.89% after postprocessing. Our Robotic Anonymization Network outperforms previous state-of-the-art algorithms, even on unseen procedural types, despite the fact that alternative solutions are explicitly trained using these procedures.

CONCLUSIONS

Our deep learning model, Robotic Anonymization Network, offers reliable, accurate, and safe real-time anonymization during complex and lengthy surgical procedures regardless of the robotic platform. The model can be used in real time for surgical live streaming and is openly available.

Optimizing drug therapy for older adults: shifting away from problematic polypharmacy

INTRODUCTION

The accelerated discovery and production of pharmaceutical products has resulted in many positive outcomes. However, this progress has also contributed to problematic polypharmacy, one of the rapidly growing threats to public health in this century. Problematic polypharmacy results in adverse patient outcomes and imposes increased strain and financial burden on healthcare systems.

AREAS COVERED

A review was conducted on the current body of evidence concerning factors contributing to and consequences of problematic polypharmacy. Recent trials investigating interventions that target polypharmacy and emerging solutions, including incorporation of artificial intelligence, are also examined in this article.

EXPERT OPINION

To shift away from problematic polypharmacy, a multifaceted interdisciplinary approach is necessary. Any potentially successful strategy must be adapted to suit various healthcare settings and must utilize all available resources, including artificial intelligence.

Artificial Intelligence in Urologic Robotic Oncologic Surgery: A Narrative Review

With the rapid increase in computer processing capacity over the past two decades, machine learning techniques have been applied in many sectors of daily life. Machine learning in therapeutic settings is also gaining popularity. We analysed current studies on machine learning in robotic urologic surgery. We searched PubMed/Medline and Google Scholar up to December 2023. Search terms included "urologic surgery", "artificial intelligence", "machine learning", "neural network", "automation", and "robotic surgery". Automatic preoperative imaging, intraoperative anatomy matching, and bleeding prediction has been a major focus. Early artificial intelligence (AI) therapeutic outcomes are promising. Robot-assisted surgery provides precise telemetry data and a cutting-edge viewing console to analyse and improve AI integration in surgery. Machine learning enhances surgical skill feedback, procedure effectiveness, surgical guidance, and postoperative prediction. Tension-sensors on robotic arms and augmented reality can improve surgery. This provides real-time organ motion monitoring, improving precision and accuracy. As datasets develop and electronic health records are used more and more, these technologies will become more effective and useful. AI in robotic surgery is intended to improve surgical training and experience. Both seek precision to improve surgical care. AI in ''master-slave'' robotic surgery offers the detailed, step-by-step examination of autonomous robotic treatments.

Preoperative three-dimensional modelling and virtual reality planning aids nephron sparing surgery in a child with bilateral Wilms tumour

Bilateral Wilms tumour (BWT) is a surgically challenging condition. Virtual reality (VR) reconstruction aids surgeons to foresee the anatomy ahead of Nephron Sparing Surgery (NSS). Three-dimensional (3D) visualisation improves the anatomical orientation of surgeons performing NSS. We herewith report a case of BWT where VR planning and 3D printing were used to aid NSS. Conventional imaging is often found to be inadequate while assessing the tumour-organ-vascular anatomy. Advances like VR and 3D printing help surgeons plan better for complex surgeries like bilateral NSS. Next-generation extended reality tools will likely aid robotic-assisted precision NSS and improve patient outcomes.

What More Can We Ask of Robotics?

The future of robotics relies heavily on the ongoing synergy between robotic surgery and artificial intelligence. To unlock their full potential, we should address issues such as accessibility, education, data privacy, and ethics.

First-in-human real-time AI-assisted instrument deocclusion during augmented reality robotic surgery

The integration of Augmented Reality (AR) into daily surgical practice is withheld by the correct registration of pre-operative data. This includes intelligent 3D model superposition whilst simultaneously handling real and virtual occlusions caused by the AR overlay. Occlusions can negatively impact surgical safety and as such deteriorate rather than improve surgical care. Robotic surgery is particularly suited to tackle these integration challenges in a stepwise approach as the robotic console allows for different inputs to be displayed in parallel to the surgeon. Nevertheless, real-time de-occlusion requires extensive computational resources which further complicates clinical integration. This work tackles the problem of instrument occlusion and presents, to the authors' best knowledge, the first-in-human on edge deployment of a real-time binary segmentation pipeline during three robot-assisted surgeries: partial nephrectomy, migrated endovascular stent removal, and liver metastasectomy. To this end, a state-of-the-art real-time segmentation and 3D model pipeline was implemented and presented to the surgeon during live surgery. The pipeline allows real-time binary segmentation of 37 non-organic surgical items, which are never occluded during AR. The application features real-time manual 3D model manipulation for correct soft tissue alignment. The proposed pipeline can contribute towards surgical safety, ergonomics, and acceptance of AR in minimally invasive surgery.

Automatic 3D Augmented-Reality Robot-Assisted Partial Nephrectomy Using Machine Learning: Our Pioneer Experience

The aim of "Precision Surgery" is to reduce the impact of surgeries on patients' global health. In this context, over the last years, the use of three-dimensional virtual models (3DVMs) of organs has allowed for intraoperative guidance, showing hidden anatomical targets, thus limiting healthy-tissue dissections and subsequent damage during an operation. In order to provide an automatic 3DVM overlapping in the surgical field, we developed and tested a new software, called "ikidney", based on convolutional neural networks (CNNs). From January 2022 to April 2023, patients affected by organ-confined renal masses amenable to RAPN were enrolled. A bioengineer, a software developer, and a surgeon collaborated to create hyper-accurate 3D models for automatic 3D AR-guided RAPN, using CNNs. For each patient, demographic and clinical data were collected. A total of 13 patients were included in the present study. The average anchoring time was 11 (6-13) s. Unintended 3D-model automatic co-registration temporary failures happened in a static setting in one patient, while this happened in one patient in a dynamic setting. There was one failure; in this single case, an ultrasound drop-in probe was used to detect the neoplasm, and the surgery was performed under ultrasound guidance instead of AR guidance. No major intraoperative nor postoperative complications (i.e., Clavien Dindo > 2) were recorded. The employment of AI has unveiled several new scenarios in clinical practice, thanks to its ability to perform specific tasks autonomously. We employed CNNs for an automatic 3DVM overlapping during RAPN, thus improving the accuracy of the superimposition process.

Clinical applications of artificial intelligence in robotic surgery

Artificial intelligence (AI) is revolutionizing nearly every aspect of modern life. In the medical field, robotic surgery is the sector with some of the most innovative and impactful advancements. In this narrative review, we outline recent contributions of AI to the field of robotic surgery with a particular focus on intraoperative enhancement. AI modeling is allowing surgeons to have advanced intraoperative metrics such as force and tactile measurements, enhanced detection of positive surgical margins, and even allowing for the complete automation of certain steps in surgical procedures. AI is also Query revolutionizing the field of surgical education. AI modeling applied to intraoperative surgical video feeds and instrument kinematics data is allowing for the generation of automated skills assessments. AI also shows promise for the generation and delivery of highly specialized intraoperative surgical feedback for training surgeons. Although the adoption and integration of AI show promise in robotic surgery, it raises important, complex ethical questions. Frameworks for thinking through ethical dilemmas raised by AI are outlined in this review. AI enhancements in robotic surgery is some of the most groundbreaking research happening today, and the studies outlined in this review represent some of the most exciting innovations in recent years.

Computer Vision and Machine-Learning Techniques for Automatic 3D Virtual Images Overlapping During Augmented Reality Guided Robotic Partial Nephrectomy

OBJECTIVES

The research's purpose is to develop a software that automatically integrates and overlay 3D virtual models of kidneys harboring renal masses into the Da Vinci robotic console, assisting surgeon during the intervention.

INTRODUCTION

Precision medicine, especially in the field of minimally-invasive partial nephrectomy, aims to use 3D virtual models as a guidance for augmented reality robotic procedures. However, the co-registration process of the virtual images over the real operative field is performed manually.

METHODS

In this prospective study, two strategies for the automatic overlapping of the model over the real kidney were explored: the computer vision technology, leveraging the super-enhancement of the kidney allowed by the intraoperative injection of Indocyanine green for superimposition and the convolutional neural network technology, based on the processing of live images from the endoscope, after a training of the software on frames from prerecorded videos of the same surgery. The work-team, comprising a bioengineer, a software-developer and a surgeon, collaborated to create hyper-accuracy 3D models for automatic 3D-AR-guided RAPN. For each patient, demographic and clinical data were collected.

RESULTS

Two groups (group A for the first technology with 12 patients and group B for the second technology with 8 patients) were defined. They showed comparable preoperative and post-operative characteristics. Concerning the first technology the average co-registration time was 7 (3-11) seconds while in the case of the second technology 11 (6-13) seconds. No major intraoperative or postoperative complications were recorded. There were no differences in terms of functional outcomes between the groups at every time-point considered.

CONCLUSION

The first technology allowed a successful anchoring of the 3D model to the kidney, despite minimal manual refinements. The second technology improved kidney automatic detection without relying on indocyanine injection, resulting in better organ boundaries identification during tests. Further studies are needed to confirm this preliminary evidence.

Robot-assisted Partial Nephrectomy Using Intra-arterial Renal Hypothermia for Highly Complex Endophytic or Hilar Tumors: Case Series and Description of Surgical Technique

Background

In partial nephrectomy for highly complex tumors with expected long ischemia time, renal hypothermia can be used to minimize ischemic parenchymal damage.

Objective

To describe our case series, surgical technique, and early outcomes for robot-assisted partial nephrectomy (RAPN) using intra-arterial cold perfusion through arteriotomy.

Design setting and participants

A retrospective analysis was conducted of ten patients with renal tumors (PADUA score 9-13) undergoing RAPN between March 2020 and March 2023 with intra-arterial cooling because of expected arterial clamping times longer than 25 min.

Surgical procedure

Multiport transperitoneal RAPN with full renal mobilization and arterial, venous, and ureteral clamping was performed. After arteriotomy and venotomy, 4°C heparinized saline is administered intravascular through a Fogarty catheter to maintain renal hypothermia while performing RAPN.

Measurements

Demographic data, renal function, console and ischemia times, surgical margin status, hospital stay, estimated blood loss, and complications were analyzed.

Results and limitations

The median warm and cold ischemia times were 4 min (interquartile range [IQR] 3-7 min) and 60 min (IQR 33-75 min), respectively. The median rewarming ischemia time was 10.5 min (IQR 6.5-23.75 min). The median pre- and postoperative estimated glomerular filtration rate values at least 1 mo after surgery were 90 ml/min (IQR 78.35-90 ml/min) and 86.9 ml/min (IQR 62.08-90 ml/min), respectively. Limitations include small cohort size and short median follow-up (13 [IQR 9.1-32.4] mo).

Conclusions

We demonstrate the feasibility and first case series for RAPN using intra-arterial renal hypothermia through arteriotomy. This approach broadens the scope for minimal invasive nephron-sparing surgery in highly complex renal masses.

Patient summary

We demonstrate a minimally invasive surgical technique that reduces kidney infarction during complex kidney tumor removal where surrounding healthy kidney tissue is spared. The technique entails arterial cold fluid irrigation, which temporarily decreases renal metabolism and allows more kidneys to be salvaged.

Artificial Intelligence in Surgical Training for Kidney Cancer: A Systematic Review of the Literature

The prevalence of renal cell carcinoma (RCC) is increasing due to advanced imaging techniques. Surgical resection is the standard treatment, involving complex radical and partial nephrectomy procedures that demand extensive training and planning. Furthermore, artificial intelligence (AI) can potentially aid the training process in the field of kidney cancer. This review explores how artificial intelligence (AI) can create a framework for kidney cancer surgery to address training difficulties. Following PRISMA 2020 criteria, an exhaustive search of PubMed and SCOPUS databases was conducted without any filters or restrictions. Inclusion criteria encompassed original English articles focusing on AI's role in kidney cancer surgical training. On the other hand, all non-original articles and articles published in any language other than English were excluded. Two independent reviewers assessed the articles, with a third party settling any disagreement. Study specifics, AI tools, methodologies, endpoints, and outcomes were extracted by the same authors. The Oxford Center for Evidence-Based Medicine's evidence levels were employed to assess the studies. Out of 468 identified records, 14 eligible studies were selected. Potential AI applications in kidney cancer surgical training include analyzing surgical workflow, annotating instruments, identifying tissues, and 3D reconstruction. AI is capable of appraising surgical skills, including the identification of procedural steps and instrument tracking. While AI and augmented reality (AR) enhance training, challenges persist in real-time tracking and registration. The utilization of AI-driven 3D reconstruction proves beneficial for intraoperative guidance and preoperative preparation. Artificial intelligence (AI) shows potential for advancing surgical training by providing unbiased evaluations, personalized feedback, and enhanced learning processes. Yet challenges such as consistent metric measurement, ethical concerns, and data privacy must be addressed. The integration of AI into kidney cancer surgical training offers solutions to training difficulties and a boost to surgical education. However, to fully harness its potential, additional studies are imperative.

The Present and Future of Artificial Intelligence in Urological Cancer

Artificial intelligence has drawn more and more attention for both research and application in the field of medicine. It has considerable potential for urological cancer detection, therapy, and prognosis prediction due to its ability to choose features in data to complete a particular task autonomously. Although the clinical application of AI is still immature and faces drawbacks such as insufficient data and a lack of prospective clinical trials, AI will play an essential role in individualization and the whole management of cancers as research progresses. In this review, we summarize the applications and studies of AI in major urological cancers, including tumor diagnosis, treatment, and prognosis prediction. Moreover, we discuss the current challenges and future applications of AI.