Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results

Augmented Reality Implementation in Minimally Invasive Surgery for Future Application in Pulmonary Surgery: A Systematic Review

OBJECTIVE

This systematic review investigates of Augmented Reality (AR) systems used in minimally invasive surgery of deformable organs, focusing on initial registration, dynamic tracking, and visualization. The objective is to acquire a comprehensive understanding of the current knowledge, applications, and challenges associated with current AR-techniques, aiming to leverage these insights for developing a dedicated AR pulmonary Video or Robotic Assisted Thoracic Surgery (VATS/RATS) workflow.

METHODS

A systematic search was conducted within Embase, Medline (Ovid) and Web of Science on April 16, 2024, following the Preferred Reporting items for Systematic Reviews and Meta-Analyses (PRISMA). The search focused on intraoperative AR applications and intraoperative navigational purposes for deformable organs. Quality assessment was performed and studies were categorized according to initial registration and dynamic tracking methods.

RESULTS

33 articles were included, of which one involved pulmonary surgery. Studies used both manual and (semi-) automatic registration methods, established through anatomical landmark-based, fiducial-based, or surface-based techniques. Diverse outcome measures were considered, including surgical outcomes and registration accuracy. The majority of studies that reached an registration accuracy below 5 mm applied surface-based registration.

CONCLUSIONS

AR can potentially aid surgeons with real-time navigation and decision making during anatomically complex minimally invasive procedures. Future research for pulmonary applications should focus on exploring surface-based registration methods, considering their non-invasive, marker-less nature, and promising accuracy. Additionally, vascular-labeling-based methods are worth exploring, given the importance and relative stability of broncho-vascular anatomy in pulmonary VATS/RATS. Assessing clinical feasibility of these approaches is crucial, particularly concerning registration accuracy and potential impact on surgical outcomes.

Artificial intelligence in urolithiasis: a systematic review of utilization and effectiveness

PURPOSE

Mirroring global trends, artificial intelligence advances in medicine, notably urolithiasis. It promises accurate diagnoses, effective treatments, and forecasting epidemiological risks and stone passage. This systematic review aims to identify the types of AI models utilised in urolithiasis studies and evaluate their effectiveness.

METHODS

The study was registered with PROSPERO. Pubmed, EMBASE, Google Scholar, and Cochrane Library databases were searched for relevant literature, using keywords such as 'urology,' 'artificial intelligence,' and 'machine learning.' Only original AI studies on urolithiasis were included, excluding reviews, unrelated studies, and non-English articles. PRISMA guidelines followed.

RESULTS

Out of 4851 studies initially identified, 71 were included for comprehensive analysis in the application of AI in urolithiasis. AI showed notable proficiency in stone composition analysis in 12 studies, achieving an average precision of 88.2% (Range 0.65-1). In the domain of stone detection, the average precision remarkably reached 96.9%. AI's accuracy rate in predicting spontaneous ureteral stone passage averaged 87%, while its performance in treatment modalities such as PCNL and SWL achieved average accuracy rates of 82% and 83%, respectively. These AI models were generally superior to traditional diagnostic and treatment methods.

CONCLUSION

The consolidated data underscores AI's increasing significance in urolithiasis management. Across various dimensions-diagnosis, monitoring, and treatment-AI outperformed conventional methodologies. High precision and accuracy rates indicate that AI is not only effective but also poised for integration into routine clinical practice. Further research is warranted to establish AI's long-term utility and to validate its role as a standard tool in urological care.

More than meets the eye: Augmented reality in surgical oncology

BACKGROUND AND OBJECTIVES

In the field of surgical oncology, there has been a desire for innovative techniques to improve tumor visualization, resection, and patient outcomes. Augmented reality (AR) technology superimposes digital content onto the real-world environment, enhancing the user's experience by blending digital and physical elements. A thorough examination of AR technology in surgical oncology has yet to be performed.

METHODS

A scoping review of intraoperative AR in surgical oncology was conducted according to the guidelines and recommendations of The Preferred Reporting Items for Systematic Review and Meta-analyzes Extension for Scoping Reviews (PRISMA-ScR) framework. All original articles examining the use of intraoperative AR during surgical management of cancer were included. Exclusion criteria included virtual reality applications only, preoperative use only, fluorescence, AR not specific to surgical oncology, and study design (reviews, commentaries, abstracts).

RESULTS

A total of 2735 articles were identified of which 83 were included. Most studies (52) were performed on animals or phantom models, while the remaining included patients. A total of 1112 intraoperative AR surgical cases were performed across the studies. The most common anatomic site was brain (20 articles), followed by liver (16), renal (9), and head and neck (8). AR was most often used for intraoperative navigation or anatomic visualization of tumors or critical structures but was also used to identify osteotomy or craniotomy planes.

CONCLUSIONS

AR technology has been applied across the field of surgical oncology to aid in localization and resection of tumors.

Smart Operating Room in Digestive Surgery: A Narrative Review

The introduction of new technologies in current digestive surgical practice is progressively reshaping the operating room, defining the fourth surgical revolution. The implementation of black boxes and control towers aims at streamlining workflow and reducing surgical error by early identification and analysis, while augmented reality and artificial intelligence augment surgeons' perceptual and technical skills by superimposing three-dimensional models to real-time surgical images. Moreover, the operating room architecture is transitioning toward an integrated digital environment to improve efficiency and, ultimately, patients' outcomes. This narrative review describes the most recent evidence regarding the role of these technologies in transforming the current digestive surgical practice, underlining their potential benefits and drawbacks in terms of efficiency and patients' outcomes, as an attempt to foresee the digestive surgical practice of tomorrow.

Potential Applications of New Headsets for Virtual and Augmented Reality in Urology

Virtual and augmented reality (VR/AR) technologies hold great promise in various medical fields. The release of a new generation of headsets for medical enhanced VR/AR (MER) opens new possibilities for applications in medicine, particularly in urology, to improve accessibility to everyone. These innovative headsets offer deep immersion without requiring a controller, which represents a novel approach to VR/AR engagement. The potential of these headsets applies to all aspects of urology, including surgical training, virtual meetings, communication between health care providers, patient counseling, telemedicine, delivering patient advice, and pain control. MER has the potential to improve operative planning and enhance intraoperative navigation and spatial awareness. The surgeon's visualization and overall experience can be significantly enhanced via improved guidance and visualization, ultimately leading to greater precision and safety. This cutting-edge technology has the potential to reshape urology practice, communication methods, and medical procedures, and ultimately to improve patients' experience of their urological condition. PATIENT SUMMARY: This mini review explores how a new generation of headsets for medical enhanced virtual reality could revolutionize urology by improving surgical planning, assistance during procedures, and medical education. Patients can benefit from better pain management and a deeper understanding of their conditions. However, challenges such as costs, accuracy, and ethical concerns must be addressed. This technology holds promise for transforming urological practice and patient care.

The value of Augmented Reality in surgery - A usability study on laparoscopic liver surgery

Augmented Reality (AR) is considered to be a promising technology for the guidance of laparoscopic liver surgery. By overlaying pre-operative 3D information of the liver and internal blood vessels on the laparoscopic view, surgeons can better understand the location of critical structures. In an effort to enable AR, several authors have focused on the development of methods to obtain an accurate alignment between the laparoscopic video image and the pre-operative 3D data of the liver, without assessing the benefit that the resulting overlay can provide during surgery. In this paper, we present a study that aims to assess quantitatively and qualitatively the value of an AR overlay in laparoscopic surgery during a simulated surgical task on a phantom setup. We design a study where participants are asked to physically localise pre-operative tumours in a liver phantom using three image guidance conditions - a baseline condition without any image guidance, a condition where the 3D surfaces of the liver are aligned to the video and displayed on a black background, and a condition where video see-through AR is displayed on the laparoscopic video. Using data collected from a cohort of 24 participants which include 12 surgeons, we observe that compared to the baseline, AR decreases the median localisation error of surgeons on non-peripheral targets from 25.8 mm to 9.2 mm. Using subjective feedback, we also identify that AR introduces usability improvements in the surgical task and increases the perceived confidence of the users. Between the two tested displays, the majority of participants preferred to use the AR overlay instead of navigated view of the 3D surfaces on a separate screen. We conclude that AR has the potential to improve performance and decision making in laparoscopic surgery, and that improvements in overlay alignment accuracy and depth perception should be pursued in the future.

Patient-specific simulations and navigation systems for partial nephrectomy

Partial nephrectomy (PN) is the standard treatment for T1 renal cell carcinoma. PN is affected more by surgical variations and requires greater surgical experience than radical nephrectomy. Patient-specific simulations and navigation systems may help to reduce the surgical experience required for PN. Recent advances in three-dimensional (3D) virtual reality (VR) imaging and 3D printing technology have allowed accurate patient-specific simulations and navigation systems. We reviewed previous studies about patient-specific simulations and navigation systems for PN. Recently, image reconstruction technology has developed, and commercial software that converts two-dimensional images into 3D images has become available. Many urologists are now able to view 3DVR images when preparing for PN. Surgical simulations based on 3DVR images can change surgical plans and improve surgical outcomes, and are useful during patient consultations. Patient-specific simulators that are capable of simulating surgical procedures, the gold-standard form of patient-specific simulations, have also been reported. Besides VR, 3D printing is also useful for understanding patient-specific information. Some studies have reported simulation and navigation systems for PN based on solid 3D models. Patient-specific simulations are a form of preoperative preparation, whereas patient-specific navigation is used intraoperatively. Navigation-assisted PN procedures using 3DVR images have become increasingly common, especially in robotic surgery. Some studies found that these systems produced improvements in surgical outcomes. Once its accuracy has been confirmed, it is hoped that this technology will spread further and become more generalized.

A survey of augmented reality methods to guide minimally invasive partial nephrectomy

INTRODUCTION

Minimally invasive partial nephrectomy (MIPN) has become the standard of care for localized kidney tumors over the past decade. The characteristics of each tumor, in particular its size and relationship with the excretory tract and vessels, allow one to judge its complexity and to attempt predicting the risk of complications. The recent development of virtual 3D model reconstruction and computer vision has opened the way to image-guided surgery and augmented reality (AR).

OBJECTIVE

Our objective was to perform a systematic review to list and describe the different AR techniques proposed to support PN.

MATERIALS AND METHODS

The systematic review of the literature was performed on 12/04/22, using the keywords "nephrectomy" and "augmented reality" on Embase and Medline. Articles were considered if they reported surgical outcomes when using AR with virtual image overlay on real vision, during ex vivo or in vivo MIPN. We classified them according to the registration technique they use.

RESULTS

We found 16 articles describing an AR technique during MIPN procedures that met the eligibility criteria. A moderate to high risk of bias was recorded for all the studies. We classified registration methods into three main families, of which the most promising one seems to be surface-based registration.

CONCLUSION

Despite promising results, there do not exist studies showing an improvement in clinical outcomes using AR. The ideal AR technique is probably yet to be established, as several designs are still being actively explored. More clinical data will be required to establish the potential contribution of this technology to MIPN.

User's image perception improved strategy and application of augmented reality systems in smart medical care: A review

BACKGROUND

Augmented reality (AR) is a new human-computer interaction technology that combines virtual reality, computer vision, and computer networks. With the rapid advancement of the medical field towards intelligence and data visualisation, AR systems are becoming increasingly popular in the medical field because they can provide doctors with clear enough medical images and accurate image navigation in practical applications. However, it has been discovered that different display types of AR systems have different effects on doctors' perception of the image after virtual-real fusion during the actual medical application. If doctors cannot correctly perceive the image, they may be unable to correctly match the virtual information with the real world, which will have a significant impact on their ability to recognise complex structures.

METHODS

This paper uses Citespace, a literature analysis tool, to visualise and analyse the research hotspots when AR systems are used in the medical field.

RESULTS

A visual analysis of the 1163 articles retrieved from the Web of Science Core Collection database reveals that display technology and visualisation technology are the key research directions of AR systems at the moment.

CONCLUSION

This paper categorises AR systems based on their display principles, reviews current image perception optimisation schemes for various types of systems, and analyses and compares different display types of AR systems based on their practical applications in the field of smart medical care so that doctors can select the appropriate display types based on different application scenarios. Finally, the future development direction of AR display technology is anticipated in order for AR technology to be more effectively applied in the field of smart medical care. The advancement of display technology for AR systems is critical for their use in the medical field, and the advantages and disadvantages of various display types should be considered in different application scenarios to select the best AR system.

Effects of a Mixed-Reality Headset on Procedural Outcomes in the Cardiac Catheterization Laboratory

BACKGROUND

Mixed reality head-mounted displays (MR-HMD) are a novel and emerging tool in healthcare. There is a paucity of data on the safety and efficacy of the use of MR-HMD in the cardiac catheterization laboratory (CCL). We sought to analyze and compare fluoroscopy time, procedure time, and complication rates with right heart catheterizations (RHCs) and coronary angiographies (CAs) performed with MR-HMD versus standard LCD medical displays.

METHODS

This is a non-randomized trial that included patients who underwent RHC and CA with MR-HMD between August 2019 and January 2020. Their outcomes were compared to a control group during the same time period. The primary endpoints were procedure time, fluoroscopy time, and dose area product (DAP). The secondary endpoints were contrast volume and intra and postprocedural complications rate.

RESULTS

50 patients were enrolled in the trial, 33 had a RHC done, and 29 had a diagnostic CA performed. They were compared to 232 patients in the control group. The use of MR-HMD was associated with a significantly lower procedure time (20 min (IQR 14-30) vs. 25 min (IQR 18-36), p = 0.038). There were no significant differences in median fluoroscopy time (1.5 min (IQR 0.7-4.9) in the study group vs. 1.3 min (IQR 0.8-3.1), p = 0.84) or median DAP (165.4 mGy·cm² (IQR 13-15,583) in the study group vs. 913 mGy·cm² (IQR 24-6291), p = 0.17). There was no significant increase in intra- or post-procedure complications.

CONCLUSION

MR-HMD use is safe and feasible and may decrease procedure time in the CCL.

A 3D Image Registration Method for Laparoscopic Liver Surgery Navigation

At present, laparoscopic augmented reality (AR) navigation has been applied to minimally invasive abdominal surgery, which can help doctors to see the location of blood vessels and tumors in organs, so as to perform precise surgery operations. Image registration is the process of optimally mapping one or more images to the target image, and it is also the core of laparoscopic AR navigation. The key is how to shorten the registration time and optimize the registration accuracy. We have studied the three-dimensional (3D) image registration technology in laparoscopic liver surgery navigation and proposed a new registration method combining rough registration and fine registration. First, the adaptive fireworks algorithm (AFWA) is applied to rough registration, and then the optimized iterative closest point (ICP) algorithm is applied to fine registration. We proposed a method that is validated by the computed tomography (CT) dataset 3D-IRCADb-01. Experimental results show that our method is superior to other registration methods based on stochastic optimization algorithms in terms of registration time and accuracy.

"Augmented reality" applications in urology: a systematic review

INTRODUCTION

Augmented reality (AR) applied to surgical procedures refers to the superimposition of preoperative or intra-operative images onto the operative field. Augmented reality has been increasingly used in myriad surgical specialties including Urology. The following study reviews advances in the use of AR for improvements in urologic outcomes.

EVIDENCE ACQUISITION

We identified all descriptive, validity, prospective randomized/nonrandomized trials and retrospective comparative/noncomparative studies about the use of AR in Urology up until March 2021. The MEDLINE, Scopus, and Web of Science databases were used for literature search. We conducted the study selection according to the PRISMA (Preferred Reporting Items for Systematic Reviews and meta-analysis statement) guidelines. We limited included studies to only those using AR, excluding all that used virtual reality technology.

EVIDENCE SYNTHESIS

A total of 60 studies were identified and included in the present analysis. Overall, 19 studies were descriptive/validity/phantom studies for specific AR methodologies, 4 studies were case reports, and 37 studies included clinical prospective/retrospective comparative studies.

CONCLUSIONS

Advances in AR have led to increasing registration accuracy as well as increased ability to identify anatomic landmarks and improve outcomes during Urologic procedures such as RARP and robot-assisted partial nephrectomy.

Initial experiences with preoperative three-dimensional image reconstruction technology in laparoscopic pyeloplasty for ureteropelvic junction obstruction

Background

To explore the clinical value of three-dimensional image reconstruction technology (3DIT) on preoperative surgical planning and perioperative outcomes in laparoscopic pyeloplasty (LP).

Methods

Data of 25 patients with ureteropelvic junction obstruction (UPJO) admitted to our hospital from January 2018 to January 2019 was analyzed retrospectively. All patients underwent preoperative enhanced computed tomography (CT) scanning. In the 12 cases in the 3DIT group, preoperative planning involved the use of virtual operation and morphometry based on reconstruction of the CT data into three-dimensional (3D) images. Surgery in the other 13 cases was performed with traditional CT examination. Demographic, surgical outcome, and postoperative parameters were compared between these two groups.

Results

Reconstructed 3D images clearly showed the spatial structural relationships between the UPJO and surrounding blood vessels. In all 25 cases surgery was completed with no conversion to open surgery. Preoperative 3DIT analyses resulted in significant improvements to mean operation time (107.76 vs. 141.58 min, P=0.024), mean time of dissociating ureteropelvic junction (UPJ) (11.26 vs. 19.40 min, P=0.020), and mean estimated blood loss volume (23.84 vs. 49.16 mL, P=0.028). There were no statistically significant differences in perioperative complications, postoperative hospital stays or postoperative drainage time.

Conclusions

3DIT based on enhanced CT scans is of clinical value in the treatment of UPJO, as it can provide accurate anatomical information and reliable guidance for preoperative operation planning, and it facilitates image-guided LP.

Modified Takazawa anatomical classification of renal pelvicalyceal system based on three-dimensional virtual reconstruction models

Background

Previous classification of renal pelvicalyceal anatomical structure may be difficult to intuitively understand and unpractical for endourological surgery. We aim to put forward a modified Takazawa anatomical classification of renal pelvicalyceal system based on three-dimensional (3D) virtual reconstruction models for endourological surgery.

Methods

We retrospectively collected data on 225 patients (320 kidneys) in total between Apr. 2017 and Dec. 2020, spatial anatomical structure of renal pelvis and calyces were modeled and corresponding morphological parameters were measured after 3D virtual reconstruction of computed tomography urography (CTU). The modified Takazawa renal pelvicalyceal anatomical classification was advanced based on the renal pelvicalyceal morphological parameters [bifurcated branches of renal pelvis, cross sectional area of renal pelvis and ureteropelvic junction (UPJ), infundibuloureteral angle (IUA), lower pole infundibular calyceal length (IL)] by 3D virtual reconstruction models, and comparison of renal pelvicalyceal system morphological parameters were performed to evaluate the differences in various classification types of renal pelvis and calyces.

Results

Anatomical structure of renal pelvis and calyces were divided into two main types (Type A and Type B) according to renal pelvic branch patterns. A single pelvis without bifurcated branch was regarded as Type A (62%) and subclassified into three subtypes: Type A1 (22%), Type A2 (27%) and Type A3 (13%), the slimline pelvis was classified as Type A1, the typical pelvis as Type A2 and the broad pelvis as Type A3. A divided pelvis with bifurcated branches was seen as Type B (38%) and subclassified into two subtypes: Type B1 (15%) with the wide and flat lower calyx branch, Type B2 (23%) with the narrow and steep lower calyx branch.

Conclusions

Previous studies have reported that the visualization and classification of renal pelvicalyceal anatomical structure by endocast, autopsy, ultrasonography and excretory urography, the modified Takazawa classification system based on 3D virtual reconstruction models enables to standardized different anatomical morphology of renal pelvicalyceal system and provide intuitive and concise information on anatomy, thus leading to the improvement in treatment modality.

Augmented Reality Navigation for Stereoscopic Laparoscopic Anatomical Hepatectomy of Primary Liver Cancer: Preliminary Experience

Background

Accurate determination of intrahepatic anatomy remains challenging for laparoscopic anatomical hepatectomy (LAH). Laparoscopic augmented reality navigation (LARN) is expected to facilitate LAH of primary liver cancer (PLC) by identifying the exact location of tumors and vessels. The study was to evaluate the safety and effectiveness of our independently developed LARN system in LAH of PLC.

Methods

From May 2018 to July 2020, the study included 85 PLC patients who underwent three-dimensional (3D) LAH. According to whether LARN was performed during the operation, the patients were divided into the intraoperative navigation (IN) group and the non-intraoperative navigation (NIN) group. We compared the preoperative data, perioperative results and postoperative complications between the two groups, and introduced our preliminary experience of this novel technology in LAH.

Results

There were 44 and 41 PLC patients in the IN group and the NIN group, respectively. No significant differences were found in preoperative characteristics and any of the resection-related complications between the two groups (All P > 0.05). Compared with the NIN group, the IN group had significantly less operative bleeding (P = 0.002), lower delta Hb% (P = 0.039), lower blood transfusion rate (P < 0.001), and reduced postoperative hospital stay (P = 0.003). For the IN group, the successful fusion of simulated surgical planning and operative scene helped to determine the extent of resection.

Conclusions

The LARN contributed to the identification of important anatomical structures during LAH of PLC. It reduced vascular injury and accelerated postoperative recovery, showing a potential application prospects in liver surgery.

Accuracy of Touch-Based Registration During Robotic Image-Guided Partial Nephrectomy Before and After Tumor Resection in Validated Phantoms

Aim: Image-guided surgery (IGS) allows for accurate, real-time localization of subsurface critical structures during surgery. No prior IGS systems have described a feasible method of intraoperative reregistration after manipulation of the kidney during robotic partial nephrectomy (PN). We present a method for seamless reregistration during IGS and evaluate accuracy before and after tumor resection in two validated kidney phantoms. Materials and Methods: We performed robotic PN on two validated kidney phantoms-one with an endophytic tumor and one with an exophytic tumor-with our IGS system utilizing the da Vinci Xi robot. Intraoperatively, the kidney phantoms' surfaces were digitized with the da Vinci robotic manipulator via a touch-based method and registered to a three-dimensional segmented model created from cross-sectional CT imaging of the phantoms. Fiducial points were marked with a surgical marking pen and identified after the initial registration using the robotic manipulator. Segmented images were displayed via picture-in-picture in the surgeon console as tumor resection was performed. After resection, reregistration was performed by reidentifying the fiducial points. The accuracy of the initial registration and reregistration was compared. Results: The root mean square (RMS) averages of target registration error (TRE) were 2.53 and 4.88 mm for the endophytic and exophytic phantoms, respectively. IGS enabled resection along preplanned contours. Specifically, the RMS averages of the normal TRE over the entire resection surface were 0.75 and 2.15 mm for the endophytic and exophytic phantoms, respectively. Both tumors were resected with grossly negative margins. Point-based reregistration enabled instantaneous reregistration with minimal impact on RMS TRE compared with the initial registration (from 1.34 to 1.70 mm preresection and from 1.60 to 2.10 mm postresection). Conclusions: We present a novel and accurate registration and reregistration framework for use during IGS for PN with the da Vinci Xi surgical system. The technology is easily integrated into the surgical workflow and does not require additional hardware.

Medical image segmentation and reconstruction of prostate tumor based on 3D AlexNet

BACKGROUND

Prostate cancer is a disease with a high incidence of tumors in men. Due to the long incubation time and insidious condition, early diagnosis is difficult; especially imaging diagnosis is more difficult. In actual clinical practice, the method of manual segmentation by medical experts is mainly used, which is time-consuming and labor-intensive and relies heavily on the experience and ability of medical experts. The rapid, accurate and repeatable segmentation of the prostate area is still a challenging problem. It is important to explore the automated segmentation of prostate images based on the 3D AlexNet network.

METHOD

Taking the medical image of prostate cancer as the entry point, the three-dimensional data is introduced into the deep learning convolutional neural network. This paper proposes a 3D AlexNet method for the automatic segmentation of prostate cancer magnetic resonance images, and the general network ResNet 50, Inception -V4 compares network performance.

RESULTS

Based on the training samples of magnetic resonance images of 500 prostate cancer patients, a set of 3D AlexNet with simple structure and excellent performance was established through adaptive improvement on the basis of classic AlexNet. The accuracy rate was as high as 0.921, the specificity was 0.896, and the sensitivity It is 0.902 and the area under the receiver operating characteristic curve (AUC) is 0.964. The Mean Absolute Distance (MAD) between the segmentation result and the medical expert's gold standard is 0.356 mm, and the Hausdorff distance (HD) is 1.024 mm, the Dice similarity coefficient is 0.9768.

CONCLUSION

The improved 3D AlexNet can automatically complete the structured segmentation of prostate magnetic resonance images. Compared with traditional segmentation methods and depth segmentation methods, the performance of the 3D AlexNet network is superior in terms of training time and parameter amount, or network performance evaluation. Compared with the algorithm, it proves the effectiveness of this method.

Experimental pilot study for augmented reality-enhanced elbow arthroscopy

The purpose of this study was to develop and evaluate a novel elbow arthroscopy system with superimposed bone and nerve visualization using preoperative computed tomography (CT) and magnetic resonance imaging (MRI) data. We obtained bone and nerve segmentation data by CT and MRI, respectively, of the elbow of a healthy human volunteer and cadaveric Japanese monkey. A life size 3-dimensional (3D) model of human organs and frame was constructed using a stereo-lithographic 3D printer. Elbow arthroscopy was performed using the elbow of a cadaveric Japanese monkey. The augmented reality (AR) range of error during rotation of arthroscopy was examined at 20 mm scope-object distances. We successfully performed AR arthroscopy using the life-size 3D elbow model and the elbow of the cadaveric Japanese monkey by making anteromedial and posterior portals. The target registration error was 1.63 ± 0.49 mm (range 1-2.7 mm) with respect to the rotation angle of the lens cylinder from 40° to - 40°. We attained reasonable accuracy and demonstrated the operation of the designed system. Given the multiple applications of AR-enhanced arthroscopic visualization, it has the potential to be a next-generation technology for arthroscopy. This technique will contribute to the reduction of serious complications associated with elbow arthroscopy.

Evaluation of a novel navigation platform for laparoscopic liver surgery with organ deformation compensation using injected fiducials

In laparoscopic liver resection, surgeons conventionally rely on anatomical landmarks detected through a laparoscope, preoperative volumetric images and laparoscopic ultrasound to compensate for the challenges of minimally invasive access. Image guidance using optical tracking and registration procedures is a promising tool, although often undermined by its inaccuracy. This study evaluates a novel surgical navigation solution that can compensate for liver deformations using an accurate and effective registration method. The proposed solution relies on a robotic C-arm to perform registration to preoperative CT/MRI image data and allows for intraoperative updates during resection using fluoroscopic images. Navigation is offered both as a 3D liver model with real-time instrument visualization, as well as an augmented reality overlay on the laparoscope camera view. Testing was conducted through a pre-clinical trial which included four porcine models. Accuracy of the navigation system was measured through two evaluation methods: liver surface fiducials reprojection and a comparison between planned and navigated resection margins. Target Registration Error with the fiducials evaluation shows that the accuracy in the vicinity of the lesion was 3.78±1.89 mm. Resection margin evaluations resulted in an overall median accuracy of 4.44 mm with a maximum error of 9.75 mm over the four subjects. The presented solution is accurate enough to be potentially clinically beneficial for surgical guidance in laparoscopic liver surgery.

[Robotics and intraoperative navigation]

Urology has always been closely linked to technological progress. In the last few decades, we have witnessed increasing implementation of various technologies and innovations in subdisciplines of urology. While conventional laparoscopy is increasingly being replaced by robot-assisted procedures and the introduction of new robotic systems from various manufactures will continue for years, the field of endourolgy is still not dominated by robotic systems. However, new systems (e.g., autonomous, robot-controlled aquablation of the prostate) are becoming increasingly popular and numerous development projects will also probably change clinical care in coming years. In addition, further advancements in the combination of robotics with intraoperative navigation through the integration of imaging and augmented-reality (AR) and virtual reality (VR) technology can be expected. This combination of navigation and robotic technology is already being used successfully in prostate biopsy.

Virtual and augmented reality: potential applications in radiology

The modern-day radiologist must be adept at image interpretation, and the one who most successfully leverages new technologies may provide the highest value to patients, clinicians, and trainees. Applications of virtual reality (VR) and augmented reality (AR) have the potential to revolutionize how imaging information is applied in clinical practice and how radiologists practice. This review provides an overview of VR and AR, highlights current applications, future developments, and limitations hindering adoption.

A Review of Training and Guidance Systems in Medical Surgery

In this paper, a map of the state of the art of recent medical simulators that provide evaluation and guidance for surgical procedures is performed. The systems are reviewed and compared from the viewpoint of the used technology, force feedback, learning evaluation, didactic and visual aid, guidance, data collection and storage, and type of solution (commercial or non-commercial). The works’ assessment was made to identify if—(1) current applications can provide assistance and track performance in training, and (2) virtual environments are more suitable for practicing than physical applications. Automatic analysis of the papers was performed to minimize subjective bias. It was found that some works limit themselves to recording the session data to evaluate them internally, while others assess it and provide immediate user feedback. However, it was found that few works are currently implementing guidance, aid during sessions, and assessment. Current trends suggest that the evaluation process’s automation could reduce the workload of experts and let them focus on improving the curriculum covered in medical education. Lastly, this paper also draws several conclusions, observations per area, and suggestions for future work.

A case study: impact of target surface mesh size and mesh quality on volume-to-surface registration performance in hepatic soft tissue navigation

Purpose

Soft tissue deformation severely impacts the registration of pre- and intra-operative image data during computer-assisted navigation in laparoscopic liver surgery. However, quantifying the impact of target surface size, surface orientation, and mesh quality on non-rigid registration performance remains an open research question. This paper aims to uncover how these affect volume-to-surface registration performance.

Methods

To find such evidence, we design three experiments that are evaluated using a three-step pipeline: (1) volume-to-surface registration using the physics-based shape matching method or PBSM, (2) voxelization of the deformed surface to a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1024^3$$\end{document}10243 voxel grid, and (3) computation of similarity (e.g., mutual information), distance (i.e., Hausdorff distance), and classical metrics (i.e., mean squared error or MSE).

Results

Using the Hausdorff distance, we report a statistical significance for the different partial surfaces. We found that removing non-manifold geometry and noise improved registration performance, and a target surface size of only 16.5% was necessary.

Conclusion

By investigating three different factors and improving registration results, we defined a generalizable evaluation pipeline and automatic post-processing strategies that were deemed helpful. All source code, reference data, models, and evaluation results are openly available for download: https://github.com/ghattab/EvalPBSM/.

A new era: artificial intelligence and machine learning in prostate cancer

Artificial intelligence (AI) - the ability of a machine to perform cognitive tasks to achieve a particular goal based on provided data - is revolutionizing and reshaping our health-care systems. The current availability of ever-increasing computational power, highly developed pattern recognition algorithms and advanced image processing software working at very high speeds has led to the emergence of computer-based systems that are trained to perform complex tasks in bioinformatics, medical imaging and medical robotics. Accessibility to 'big data' enables the 'cognitive' computer to scan billions of bits of unstructured information, extract the relevant information and recognize complex patterns with increasing confidence. Computer-based decision-support systems based on machine learning (ML) have the potential to revolutionize medicine by performing complex tasks that are currently assigned to specialists to improve diagnostic accuracy, increase efficiency of throughputs, improve clinical workflow, decrease human resource costs and improve treatment choices. These characteristics could be especially helpful in the management of prostate cancer, with growing applications in diagnostic imaging, surgical interventions, skills training and assessment, digital pathology and genomics. Medicine must adapt to this changing world, and urologists, oncologists, radiologists and pathologists, as high-volume users of imaging and pathology, need to understand this burgeoning science and acknowledge that the development of highly accurate AI-based decision-support applications of ML will require collaboration between data scientists, computer researchers and engineers.

Virtual simulation, preoperative planning and intraoperative navigation during laparoscopic partial nephrectomy

Introduction

The use of computer navigation systems is a new and actively explored method used for surgical procedures concerning the abdominal and retroperitoneal organs. In this paper, we propose an original hardware – software complex, which forms a virtual body model, based on preoperative computer tomography data, transmitted to the operating screen monitor using a surgical navigation system, involving a mechanical digitizer.

Material and methods

During a laparoscopic procedure, a three-dimensional (3D) model of a kidney with a tumor was used to obtain additional information on the primary or secondary monitor or for combining the virtual model and video images on the main or additional monitor in the operating room. This method was used for laparoscopic partial nephrectomy, where twelve patients were operated with an average age of 45.4 (38–54) years, with clear cell renal cell carcinoma size 27.08 (15–40) mm.

Results

All patients successfully underwent laparoscopic partial nephrectomy with intraoperative navigation. The mean operative time was 97.2 (80–155) minutes, warm ischemia time – 18.0 (12–25) minutes. Selective clamping of segmental renal arteries was performed in 7 (58.3%) cases, in the remaining 5 (41.6%) cases the renal artery was clamped. There were no serious complications. The average duration of hospital stay was 7.0 (5–10) days.

Conclusions

Preliminary results of our clinical study have shown the success of 3D modeling for qualitative visualization of kidney tumors in the course of surgical intervention, both for the surgeon and for the patient to understand the nature of the pathological process.

3D mixed reality holograms for preoperative surgical planning of nephron-sparing surgery: evaluation of surgeons' perception

BACKGROUND

3D reconstructions are gaining a wide diffusion in nephron-sparing surgery (NSS) planning. They have usually been studied on common 2D flat supports, with limitations regarding real depth comprehension and interaction. Nowadays, it is possible to visualize kidney 3D reconstructions as holograms in a "mixed reality" (MR) setting. The aim of this study was to test the face and content validity of this technology, and to assess the role of 3D holograms in aiding preoperative planning for highly complex renal tumors amenable by NSS.

METHODS

We evaluated surgeons' perception of mixed reality for partial nephrectomy during a urological international meeting organized at our Institution in January 2019. Thanks to the images of preoperative CT, hyper-accuracy 3D (HA3D^TM) reconstructions were performed. Then, a virtual environment was created, and it interacted with the models in mixed reality setting by using HoloLens. We submitted to all the attendees a questionnaire, expressed by the Likert scale (1-10), about their opinion over the use and application of the MR. Moreover, the attendees had the chance to perform a first-hand MR experience; then, they were asked to choose their clamping and resection approach.

RESULTS

Overall 172 questionnaires were collected. The scores obtained regarding both surgical planning (scored 8/10) and anatomical accuracy (9/10) were very positive. High satisfaction toward the potential role of this technology in surgical planning and understanding of surgical complexity (both scored 9/10) were expressed. After a first-hand experience with HoloLens and MR, 64.4% and 44.4% of the surgeons changed their clamping and resection approach, respectively - compared to CT image visualization only - choosing a more selective one.

CONCLUSIONS

Our study suggests that surgeons perceive holograms and MR as a useful and interesting tool for the preoperative setting before partial nephrectomy, in the direction of an ever more precise surgery.

How well do software assistants for minimally invasive partial nephrectomy meet surgeon information needs? A cognitive task analysis and literature review study

INTRODUCTION

Intraoperative software assistance is gaining increasing importance in laparoscopic and robot-assisted surgery. Within the user-centred development process of such systems, the first question to be asked is: What information does the surgeon need and when does he or she need it? In this article, we present an approach to investigate these surgeon information needs for minimally invasive partial nephrectomy and compare these needs to the relevant surgical computer assistance literature.

MATERIALS AND METHODS

First, we conducted a literature-based hierarchical task analysis of the surgical procedure. This task analysis was taken as a basis for a qualitative in-depth interview study with nine experienced surgical urologists. The study employed a cognitive task analysis method to elicit surgeons' information needs during minimally invasive partial nephrectomy. Finally, a systematic literature search was conducted to review proposed software assistance solutions for minimally invasive partial nephrectomy. The review focused on what information the solutions present to the surgeon and what phase of the surgery they aim to support.

RESULTS

The task analysis yielded a workflow description for minimally invasive partial nephrectomy. During the subsequent interview study, we identified three challenging phases of the procedure, which may particularly benefit from software assistance. These phases are I. Hilar and vascular management, II. Tumour excision, and III. Repair of the renal defects. Between these phases, 25 individual challenges were found which define the surgeon information needs. The literature review identified 34 relevant publications, all of which aim to support the surgeon in hilar and vascular management (phase I) or tumour excision (phase II).

CONCLUSION

The work presented in this article identified unmet surgeon information needs in minimally invasive partial nephrectomy. Namely, our results suggest that future solutions should address the repair of renal defects (phase III) or put more focus on the renal collecting system as a critical anatomical structure.

[Application of U-shaped convolutional neural network in auto segmentation and reconstruction of 3D prostate model in laparoscopic prostatectomy navigation]

OBJECTIVE

To investigate the efficacy of intraoperative cognitive navigation on laparoscopic radical prostatectomy using 3D prostatic models created by U-shaped convolutional neural network (U-net) and reconstructed through Medical Image Interaction Tool Kit (MITK) platform.

METHODS

A total of 5 000 pieces of prostate cancer magnetic resonance (MR) imaging discovery sets with manual annotations were used to train a modified U-net, and a set of clinically demand-oriented, stable and efficient full convolutional neural network algorithm was constructed. The MR images were cropped and segmented automatically by using modified U-net, and the segmentation data were automatically reconstructed using MITK platform according to our own protocols. The modeling data were output as STL format, and the prostate models were simultaneously displayed on an android tablet during the operation to help achieving cognitive navigation.

RESULTS

Based on original U-net architecture, we established a modified U-net from a 201-case MR imaging training set. The network performance was tested and compared with human segmentations and other segmentation networks by using one certain testing data set. Auto segmentation of multi-structures (such as prostate, prostate tumors, seminal vesicles, rectus, neurovascular bundles and dorsal venous complex) were successfully achieved. Secondary automatic 3D reconstruction had been carried out through MITK platform. During the surgery, 3D models of prostatic area were simultaneously displayed on an android tablet, and the cognitive navigation was successfully achieved. Intra-operation organ visualization demonstrated the structural relationships among the key structures in great detail and the degree of tumor invasion was visualized directly.

CONCLUSION

The modified U-net was able to achieve automatic segmentations of important structures of prostate area. Secondary 3D model reconstruction and demonstration could provide intraoperative visualization of vital structures of prostate area, which could help achieve cognitive fusion navigation for surgeons. The application of these techniques could finally reduce positive surgical margin rates, and may improve the efficacy and oncological outcomes of laparoscopic prostatectomy.

Assessment of surgical skills by using surgical navigation in robot-assisted partial nephrectomy

PURPOSE

To assess surgical skills in robot-assisted partial nephrectomy (RAPN) with and without surgical navigation (SN).

METHODS

We employed an SN system that synchronizes the real-time endoscopic image with a virtual reality three-dimensional (3D) model for RAPN and evaluated the skills of two expert surgeons with regard to the identification and dissection of the renal artery (non-SN group, n = 21 [first surgeon n = 9, second surgeon n = 12]; SN group, n = 32 [first surgeon n = 11, second surgeon n = 21]). We converted all movements of the robotic forceps during RAPN into a dedicated vocabulary. Using RAPN videos, we classified all movements of the robotic forceps into direct action (defined as movements of the robotic forceps that directly affect tissues) and connected motion (defined as movements that link actions). In addition, we analyzed the frequency, duration, and occupancy rate of the connected motion.

RESULTS

In the SN group, the R.E.N.A.L nephrometry score was lower (7 vs. 6, P = 0.019) and the time to identify and dissect the renal artery (16 vs. 9 min, P = 0.008) was significantly shorter. The connected motions of inefficient "insert," "pull," and "rotate" motions were significantly improved by SN. SN significantly improved the frequency, duration, and occupancy rate of connected motions of the right hand of the first surgeon and of both hands of the second surgeon. The improvements in connected motions were positively associated with SN for both surgeons.

CONCLUSION

This is the first study to investigate SN for nephron-sparing surgery. SN with 3D models might help improve the connected motions of expert surgeons to ensure efficient RAPN.

Augmented reality glasses improve adherence to evidence-based intubation practice

Background: The risk of failing or delaying endotracheal intubation in critically ill patients has commonly been associated with inadequate procedure preparation. Clinicians and trainees in simulation courses for tracheal intubation are encouraged to recall the steps of how to intubate in order to mitigate the risk of a failed intubation. The purpose of this study was to assess the effectiveness of using optical head mounted display augmented reality (AR) glasses as an assistance tool to perform intubation simulation procedure. Methods: A total of 32 subjects with a mean age of 30±7.8, AR (n₁=15) vs non-augmented reality(non-AR) (n₂=17). The majority were males (n=22, 68.7%). Subjects were randomly assigned into two groups: the AR group and the non-AR group. Both groups reviewed a video on how to intubate following the New England Journal of Medicine (NEJM) intubation guidelines. The AR group had to intubate using the AR glasses head mount display compared to the non-AR where they performed regular intubation. Results: The AR group took longer median (min, max) time (seconds) to ventilate than the non-AR group (280 (130,740) vs 205 (100,390); η ² =1.0, p=0.005, respectively). Similarly, there was a higher percent adherence of NEJM intubation checklist (100% in the AR group vs 82.4% in the non-AR group; η²=1.8, p<0.001). Conclusion: The AR glasses showed promise in assisting different health care professionals on endotracheal intubation simulation. Participants in the AR group took a longer time to ventilate but scored 100% in the developed checklist that followed the NEJM protocol. This finding shows that the AR technology can be used in a simulation setting and requires further study before clinical use.

A markerless automatic deformable registration framework for augmented reality navigation of laparoscopy partial nephrectomy

Purpose Video see-through augmented reality (VST-AR) navigation for laparoscopic partial nephrectomy (LPN) can enhance intraoperative perception of surgeons by visualizing surgical targets and critical structures of the kidney tissue. Image registration is the main challenge in the procedure. Existing registration methods in laparoscopic navigation systems suffer from limitations such as manual alignment, invasive external marker fixation, relying on external tracking devices with bulky tracking sensors and lack of deformation compensation. To address these issues, we present a markerless automatic deformable registration framework for LPN VST-AR navigation.

METHOD

Dense stereo matching and 3D reconstruction, automatic segmentation and surface stitching are combined to obtain a larger dense intraoperative point cloud of the renal surface. A coarse-to-fine deformable registration is performed to achieve a precise automatic registration between the intraoperative point cloud and the preoperative model using the iterative closest point algorithm followed by the coherent point drift algorithm. Kidney phantom experiments and in vivo experiments were performed to evaluate the accuracy and effectiveness of our approach.

RESULTS

The average segmentation accuracy rate of the automatic segmentation was 94.9%. The mean target registration error of the phantom experiments was found to be 1.28 ± 0.68 mm (root mean square error). In vivo experiments showed that tumor location was identified successfully by superimposing the tumor model on the laparoscopic view.

CONCLUSION

Experimental results have demonstrated that the proposed framework could accurately overlay comprehensive preoperative models on deformable soft organs automatically in a manner of VST-AR without using extra intraoperative imaging modalities and external tracking devices, as well as its potential clinical use.

Implementing Virtual and Augmented Reality Tools for Radiology Education and Training, Communication, and Clinical Care

Advances in virtual immersive and augmented reality technology, commercially available for the entertainment and gaming industry, hold potential for education and clinical use in medicine and the field of medical imaging. Radiology departments have begun exploring the use of these technologies to help with radiology education and clinical care. The purpose of this review article is to summarize how three institutions have explored using virtual and augmented reality for radiology.

Systematic review of augmented reality in urological interventions: the evidences of an impact on surgical outcomes are yet to come

PURPOSE

To perform a systematic literature review on the clinical impact of augmented reality (AR) for urological interventions.

METHODS

As of June 21, 2018, systematic literature review was performed via Medline, Embase and Cochrane databases in accordance with the PRISMA guidelines and registered at PROSPERO (CRD42018102194). Only full text articles in English were included, without time restrictions. Articles were considered if they reported on the use of AR during urological intervention and the impact on the surgical outcomes. The risk of bias and the quality of each study included were independently assessed using the standard Cochrane Collaboration risk of bias tool and the Risk Of Bias In Non-randomised Studies-of Interventions Tool (ROBINS-I).

RESULTS

131 articles were identified. 102 remained after duplicate removal and were critically reviewed for evidence synthesis. 20 studies reporting on the outcomes of the use of AR during urological interventions in a clinical setting were considered. Given the mostly non-comparative design of the studies identified, the evidence synthesis was performed in a descriptive and narrative manner. Only one comparative study was found, with the remaining 19 items being single-arm observational studies. Based on the existing evidence, we are unable to state that AR improves the outcomes of urological interventions. The major limitation of AR-assisted surgery is inaccuracy in registration, translating into a poor navigation precision.

CONCLUSIONS

To date, there is limited evidence showing superior therapeutic benefits of AR-guided surgery when compared with the conventional surgical approach to the respective disease.

A practical marker-less image registration method for augmented reality oral and maxillofacial surgery

BACKGROUND

Image registration lies in the core of augmented reality (AR), which aligns the virtual scene with the reality. In AR surgical navigation, the performance of image registration is vital to the surgical outcome.

METHODS

This paper presents a practical marker-less image registration method for AR-guided oral and maxillofacial surgery where a virtual scene is generated and mixed with reality to guide surgical operation or provide surgical outcome visualization in the manner of video see-through overlay. An intraoral 3D scanner is employed to acquire the patient's teeth shape model intraoperatively. The shape model is then registered with a custom-made stereo camera system using a novel 3D stereo matching algorithm and with the patient's CT-derived 3D model using an iterative closest point scheme, respectively. By leveraging the intraoral 3D scanner, the CT space and the stereo camera space are associated so that surrounding anatomical models and virtual implants could be overlaid on the camera's view to achieve AR surgical navigation.

RESULTS

Jaw phantom experiments were performed to evaluate the target registration error of the overlay, which yielded an average error of less than 0.50 mm with the time cost less than 0.5 s. Volunteer trial was also conducted to show the clinical feasibility.

CONCLUSIONS

The proposed registration method does not rely on any external fiducial markers attached to the patient. It performs automatically so as to maintain a correct AR scene, overcoming the misalignment difficulty caused by patient's movement. Therefore, it is noninvasive and practical in oral and maxillofacial surgery.

3D reconstruction of CT scans aid in preoperative planning for sarcomatoid renal cancer: A case report and mini-review

Contrast-enhanced multi-slice computed tomography (MSCT) is commonly used in the diagnosis of complex malignant tumours. This technology provides comprehensive and accurate information about tumour size and shape in relation to solid tumours and the affected adjacent organs and tissues. This case report demonstrates the benefit of using MSCT 3D imaging for preoperative planning in a patient with late-stage (T4) sarcomatoid renal cell carcinoma, a rare renal malignant tumour. The surgical margin on the liver was negative, and no metastases to veins, lungs or other organs were detected by abdominal and chest contrast-enhanced CT. Although sarcomatoid histology is considered to be a poor prognostic factor, the patient is alive and well 17 months after surgery. The MSCT imaging modality enables 3D rendering of an area of interest, which assists surgical decision-making in cases of advanced renal tumours. In this case, as a result of MSCT 3D reconstruction, the patient received justified surgical treatment without compromising oncological principles.

Global rigid registration of CT to video in laparoscopic liver surgery

PURPOSE

Image-guidance systems have the potential to aid in laparoscopic interventions by providing sub-surface structure information and tumour localisation. The registration of a preoperative 3D image with the intraoperative laparoscopic video feed is an important component of image guidance, which should be fast, robust and cause minimal disruption to the surgical procedure. Most methods for rigid and non-rigid registration require a good initial alignment. However, in most research systems for abdominal surgery, the user has to manually rotate and translate the models, which is usually difficult to perform quickly and intuitively.

METHODS

We propose a fast, global method for the initial rigid alignment between a 3D mesh derived from a preoperative CT of the liver and a surface reconstruction of the intraoperative scene. We formulate the shape matching problem as a quadratic assignment problem which minimises the dissimilarity between feature descriptors while enforcing geometrical consistency between all the feature points. We incorporate a novel constraint based on the liver contours which deals specifically with the challenges introduced by laparoscopic data.

RESULTS

We validate our proposed method on synthetic data, on a liver phantom and on retrospective clinical data acquired during a laparoscopic liver resection. We show robustness over reduced partial size and increasing levels of deformation. Our results on the phantom and on the real data show good initial alignment, which can successfully converge to the correct position using fine alignment techniques. Furthermore, since we can pre-process the CT scan before surgery, the proposed method runs faster than current algorithms.

CONCLUSION

The proposed shape matching method can provide a fast, global initial registration, which can be further refined by fine alignment methods. This approach will lead to a more usable and intuitive image-guidance system for laparoscopic liver surgery.

A Systematic Review of 10 Years of Augmented Reality Usability Studies: 2005 to 2014

Augmented Reality (AR) interfaces have been studied extensively over the last few decades, with a growing number of user-based experiments. In this paper, we systematically review 10 years of the most influential AR user studies, from 2005 to 2014. A total of 291 papers with 369 individual user studies have been reviewed and classified based on their application areas. The primary contribution of the review is to present the broad landscape of user-based AR research, and to provide a high-level view of how that landscape has changed. We summarize the high-level contributions from each category of papers, and present examples of the most influential user studies. We also identify areas where there have been few user studies, and opportunities for future research. Among other things, we find that there is a growing trend toward handheld AR user studies, and that most studies are conducted in laboratory settings and do not involve pilot testing. This research will be useful for AR researchers who want to follow best practices in designing their own AR user studies.

Intra-operative ultrasound-based augmented reality guidance for laparoscopic surgery

In laparoscopic surgery, the surgeon must operate with a limited field of view and reduced depth perception. This makes spatial understanding of critical structures difficult, such as an endophytic tumour in a partial nephrectomy. Such tumours yield a high complication rate of 47%, and excising them increases the risk of cutting into the kidney's collecting system. To overcome these challenges, an augmented reality guidance system is proposed. Using intra-operative ultrasound, a single navigation aid, and surgical instrument tracking, four augmentations of guidance information are provided during tumour excision. Qualitative and quantitative system benefits are measured in simulated robot-assisted partial nephrectomies. Robot-to-camera calibration achieved a total registration error of 1.0 ± 0.4 mm while the total system error is 2.5 ± 0.5 mm. The system significantly reduced healthy tissue excised from an average (±standard deviation) of 30.6 ± 5.5 to 17.5 ± 2.4 cm³ (p < 0.05) and reduced the depth from the tumor underside to cut from an average (±standard deviation) of 10.2 ± 4.1 to 3.3 ± 2.3 mm (p < 0.05). Further evaluation is required in vivo, but the system has promising potential to reduce the amount of healthy parenchymal tissue excised.

Virtual and Augmented Reality Systems for Renal Interventions: A Systematic Review

PURPOSE

Many virtual and augmented reality systems have been proposed to support renal interventions. This paper reviews such systems employed in the treatment of renal cell carcinoma and renal stones.

METHODS

A systematic literature search was performed. Inclusion criteria were virtual and augmented reality systems for radical or partial nephrectomy and renal stone treatment, excluding systems solely developed or evaluated for training purposes.

RESULTS

In total, 52 research papers were identified and analyzed. Most of the identified literature (87%) deals with systems for renal cell carcinoma treatment. About 44% of the systems have already been employed in clinical practice, but only 20% in studies with ten or more patients. Main challenges remaining for future research include the consideration of organ movement and deformation, human factor issues, and the conduction of large clinical studies.

CONCLUSION

Augmented and virtual reality systems have the potential to improve safety and outcomes of renal interventions. In the last ten years, many technical advances have led to more sophisticated systems, which are already applied in clinical practice. Further research is required to cope with current limitations of virtual and augmented reality assistance in clinical environments.