Improving Augmented Reality Through Deep Learning: Real-time Instrument Delineation in Robotic Renal Surgery

Several barriers prevent the integration and adoption of augmented reality (AR) in robotic renal surgery despite the increased availability of virtual three-dimensional (3D) models. Apart from correct model alignment and deformation, not all instruments are clearly visible in AR. Superimposition of a 3D model on top of the surgical stream, including the instruments, can result in a potentially hazardous surgical situation. We demonstrate real-time instrument detection during AR-guided robot-assisted partial nephrectomy and show the generalization of our algorithm to AR-guided robot-assisted kidney transplantation. We developed an algorithm using deep learning networks to detect all nonorganic items. This algorithm learned to extract this information for 65 927 manually labeled instruments on 15 100 frames. Our setup, which runs on a standalone laptop, was deployed in three different hospitals and used by four different surgeons. Instrument detection is a simple and feasible way to enhance the safety of AR-guided surgery. Future investigations should strive to optimize efficient video processing to minimize the 0.5-s delay currently experienced. General AR applications also need further optimization, including detection and tracking of organ deformation, for full clinical implementation.

Multicentric exploration of tool annotation in robotic surgery: lessons learned when starting a surgical artificial intelligence project

BACKGROUND

Artificial intelligence (AI) holds tremendous potential to reduce surgical risks and improve surgical assessment. Machine learning, a subfield of AI, can be used to analyze surgical video and imaging data. Manual annotations provide veracity about the desired target features. Yet, methodological annotation explorations are limited to date. Here, we provide an exploratory analysis of the requirements and methods of instrument annotation in a multi-institutional team from two specialized AI centers and compile our lessons learned.

METHODS

We developed a bottom-up approach for team annotation of robotic instruments in robot-assisted partial nephrectomy (RAPN), which was subsequently validated in robot-assisted minimally invasive esophagectomy (RAMIE). Furthermore, instrument annotation methods were evaluated for their use in Machine Learning algorithms. Overall, we evaluated the efficiency and transferability of the proposed team approach and quantified performance metrics (e.g., time per frame required for each annotation modality) between RAPN and RAMIE.

RESULTS

We found a 0.05 Hz image sampling frequency to be adequate for instrument annotation. The bottom-up approach in annotation training and management resulted in accurate annotations and demonstrated efficiency in annotating large datasets. The proposed annotation methodology was transferrable between both RAPN and RAMIE. The average annotation time for RAPN pixel annotation ranged from 4.49 to 12.6 min per image; for vector annotation, we denote 2.92 min per image. Similar annotation times were found for RAMIE. Lastly, we elaborate on common pitfalls encountered throughout the annotation process.

CONCLUSIONS

We propose a successful bottom-up approach for annotator team composition, applicable to any surgical annotation project. Our results set the foundation to start AI projects for instrument detection, segmentation, and pose estimation. Due to the immense annotation burden resulting from spatial instrumental annotation, further analysis into sampling frequency and annotation detail needs to be conducted.

Peritoneal tuberculosis caused by intravesical instillation with Bacillus Calmette-Guérin (BCG) following nephroureterectomy in a patient with bladder and upper tract urothelial cancer: a case report

BACKGROUND

The standard treatment for high-risk non-muscle-invasive bladder cancer (NMIBC) is trans-urethral resection of the bladder (TURB) followed by instillation of Bacillus Calmette-Guérin (BCG). The occurrence of peritoneal tuberculosis after intravesical BCG instillation is extremely rare and difficult to diagnose.

METHODS

We report the case of a 79-year-old man with urothelial cell carcinoma (UCC) of the kidney and bladder who developed peritoneal tuberculosis after consecutive TURB and nephroureterectomy followed by intravesical BCG instillation. Further investigation revealed an undiagnosed bladder leak.

CONCLUSION

This case serves as a reminder for urologists to be suspicious for urothelium discontinuity when administering BCG shortly after bladder surgery.

Point-of-care 3D printing: a low-cost approach to teaching carotid artery stenting

BACKGROUND

Carotid Artery Stenting (CAS) is increasingly being used in selected patients as a minimal invasive approach to carotid endarterectomy. Despite the long standing tradition of endovascular treatments, visual feedback during stent-deployment is impossible to obtain as deployment is performed under fluoroscopic imaging. Furthermore, the concept of stent-placement is often still unclear to patients. 3D Printing allows to replicate patient-specific anatomies and deploy stents inside them to simulate procedures. As such these models are being used for endovascular training as well as patient education.

PURPOSE

To our knowledge, this study reports the first use of a low-cost patient-specific 3D printed model for teaching CAS deployment under direct visualization, without fluoroscopy.

METHODOLOGY

A CT-angiogram was segmented and converted to STL format using Mimics inPrint™ software. The carotid arteries were bilaterally truncated to fit the whole model on a Formlabs 2 printer without omitting the internal vessel diameter. Next, this model was offset using a 1 mm margin. A ridge was modelled on the original vessel anatomy which was subsequently subtracted from the offset model in order to obtain a deroofed 3D model. All vessels were truncated to facilitate post-processing, flow and guide wire placement.

RESULTS

Carotid artery stents were successfully deployed inside the vessel. The deroofing allows for clear visualization of the bottlenecks and characteristics of CAS deployment and positioning, including stent foreshortening, tapering and recoil. This low-cost 3D model provides visual insights in stent deployment and positioning, and can allow for patient-specific procedure planning.

CONCLUSIONS

The presented approach demonstrates the use of low-cost 3D Printed CAS models in teaching complex stent behavior as observed during deployment. Two main findings are illustrated. On one hand, the feasibility of low-cost in-hospital model production is shown. On the other hand, the teaching of CAS deployment bottlenecks at the carotid level without the need for fluoroscopic guidance, is illustrated. The observed stent characteristics as shown during deployment are difficult to assess in radiologic models. Furthermore, printing patient-specific 3D models preoperatively could possibly assist in accurate patient selection, preoperative planning, case-specific training and patient education.

Conservative treatment of knee osteoarthritis

Osteoarthritis of the knee causes chronic knee pain, loss of function and disability in the ageing population. When no treatment is applied, a guaranteed onset of symptoms and/or structural damage can be observed in the diseased knee. This work reviewed the different published guidelines, proposing combinations of weight reduction, physical therapy and rehabilitation, self-management education programs and pharmacological treatment. Randomized clinical trials, systematic reviews and guidelines were identified using the databases PubMed and Web of Science. Specific journals and reference lists were investigated. Sixty high quality articles were included concerning the conservative treatment of knee osteoarthritis. Weight loss when BMI > 28kg/m 2 ; aerobic, proprioception and strengthening training; NSAIDs (ibuprofen, diclofenac, aceclofenac), IA corticosteroid and IA hyaluronic acid has the highest evidence. To achieve the greatest positive clinical and structural outcome, a combined conservative therapy is recommended.