Navigation in Endoscopic Soft Tissue Surgery: Perspectives and Limitations

OneSLAM to map them all: a generalized approach to SLAM for monocular endoscopic imaging based on tracking any point

PURPOSE

Monocular SLAM algorithms are the key enabling technology for image-based surgical navigation systems for endoscopic procedures. Due to the visual feature scarcity and unique lighting conditions encountered in endoscopy, classical SLAM approaches perform inconsistently. Many of the recent approaches to endoscopic SLAM rely on deep learning models. They show promising results when optimized on singular domains such as arthroscopy, sinus endoscopy, colonoscopy or laparoscopy, but are limited by an inability to generalize to different domains without retraining.

METHODS

To address this generality issue, we propose OneSLAM a monocular SLAM algorithm for surgical endoscopy that works out of the box for several endoscopic domains, including sinus endoscopy, colonoscopy, arthroscopy and laparoscopy. Our pipeline builds upon robust tracking any point (TAP) foundation models to reliably track sparse correspondences across multiple frames and runs local bundle adjustment to jointly optimize camera poses and a sparse 3D reconstruction of the anatomy.

RESULTS

We compare the performance of our method against three strong baselines previously proposed for monocular SLAM in endoscopy and general scenes. OneSLAM presents better or comparable performance over existing approaches targeted to that specific data in all four tested domains, generalizing across domains without the need for retraining.

CONCLUSION

OneSLAM benefits from the convincing performance of TAP foundation models but generalizes to endoscopic sequences of different anatomies all while demonstrating better or comparable performance over domain-specific SLAM approaches. Future research on global loop closure will investigate how to reliably detect loops in endoscopic scenes to reduce accumulated drift and enhance long-term navigation capabilities.

Intraoperative gaze guidance with mixed reality

Efficient communication and collaboration are essential in the operating room for successful and safe surgery. While many technologies are improving various aspects of surgery, communication between attending surgeons, residents, and surgical teams is still limited to verbal interactions that are prone to misunderstandings. Novel modes of communication can increase speed and accuracy, and transform operating rooms. A mixed reality (MR) based gaze sharing application on Microsoft HoloLens 2 headset that can help expert surgeons indicate specific regions, communicate with decreased verbal effort, and guide novices throughout an operation is presented. The utility of the application is tested with a user study of endoscopic kidney stone localization completed by urology experts and novice surgeons. Improvement is observed in the NASA task load index surveys (up to 25.23%), in the success rate of the task (6.98% increase in localized stone percentage), and in gaze analyses (up to 31.99%). The proposed application shows promise in both operating room applications and surgical training tasks.

Are 3D Image Guidance Systems Ready for Use? A Comparative Analysis of 3D Image Guidance Implementations in Minimally Invasive Partial Nephrectomy

Introduction: Three-dimensional image-guided surgical (3D-IGS) systems for minimally invasive partial nephrectomy (MIPN) can potentially improve the efficiency and accuracy of intraoperative anatomical localization and tumor resection. This review seeks to analyze the current state of research regarding 3D-IGS, including the evaluation of clinical outcomes, system functionality, and qualitative insights regarding 3D-IGS's impact on surgical procedures. Methods: We have systematically reviewed the clinical literature pertaining to 3D-IGS deployed for MIPN. For inclusion, studies must produce a patient-specific 3D anatomical model from two-dimensional imaging. Data extracted from the studies include clinical results, registration (alignment of the 3D model to the surgical scene) method used, limitations, and data types reported. A subset of studies was qualitatively analyzed through an inductive coding approach to identify major themes and subthemes across the studies. Results: Twenty-five studies were included in the review. Eight (32%) studies reported clinical results that point to 3D-IGS improving multiple surgical outcomes. Manual registration was the most utilized (48%). Soft tissue deformation was the most cited limitation among the included studies. Many studies reported qualitative statements regarding surgeon accuracy improvement, but quantitative surgeon accuracy data were not reported. During the qualitative analysis, six major themes emerged across the nine applicable studies. They are as follows: 3D-IGS is necessary, 3D-IGS improved surgical outcomes, researcher/surgeon confidence in 3D-IGS system, enhanced surgeon ability/accuracy, anatomical explanation for qualitative assessment, and claims without data or reference to support. Conclusions: Currently, clinical outcomes are the main source of quantitative data available to point to 3D-IGS's efficacy. However, the literature qualitatively suggests the benefit of accurate 3D-IGS for robotic partial nephrectomy.

Application and evaluation of surgical tool and tool tip recognition based on Convolutional Neural Network in multiple endoscopic surgical scenarios

BACKGROUND

In recent years, computer-assisted intervention and robot-assisted surgery are receiving increasing attention. The need for real-time identification and tracking of surgical tools and tool tips is constantly demanding. A series of researches focusing on surgical tool tracking and identification have been performed. However, the size of dataset, the sensitivity/precision, and the response time of these studies were limited. In this work, we developed and utilized an automated method based on Convolutional Neural Network (CNN) and You Only Look Once (YOLO) v3 algorithm to locate and identify surgical tools and tool tips covering five different surgical scenarios.

MATERIALS AND METHODS

An algorithm of object detection was applied to identify and locate the surgical tools and tool tips. DarkNet-19 was used as Backbone Network and YOLOv3 was modified and applied for the detection. We included a series of 181 endoscopy videos covering 5 different surgical scenarios: pancreatic surgery, thyroid surgery, colon surgery, gastric surgery, and external scenes. A total amount of 25,333 images containing 94,463 targets were collected. Training and test sets were divided in a proportion of 2.5:1. The data sets were openly stored at the Kaggle database.

RESULTS

Under an Intersection over Union threshold of 0.5, the overall sensitivity and precision rate of the model were 93.02% and 89.61% for tool recognition and 87.05% and 83.57% for tool tip recognition, respectively. The model demonstrated the highest tool and tool tip recognition sensitivity and precision rate under external scenes. Among the four different internal surgical scenes, the network had better performances in pancreatic and colon surgeries and poorer performances in gastric and thyroid surgeries.

CONCLUSION

We developed a surgical tool and tool tip recognition model based on CNN and YOLOv3. Validation of our model demonstrated satisfactory precision, accuracy, and robustness across different surgical scenes.

A particle filter approach to dynamic kidney pose estimation in robotic surgical exposure

PURPOSE

Traditional soft tissue registration methods require direct intraoperative visualization of a significant portion of the target anatomy in order to produce acceptable surface alignment. Image guidance is therefore generally not available during the robotic exposure of structures like the kidneys which are not immediately visualized upon entry into the abdomen. This paper proposes guiding surgical exposure with an iterative state estimator that assimilates small visual cues into an a priori anatomical model as exposure progresses, thereby evolving pose estimates for the occluded structures of interest.

METHODS

Intraoperative surface observations of a right kidney are simulated using endoscope tracking and preoperative tomography from a representative robotic partial nephrectomy case. Clinically relevant random perturbations of the true kidney pose are corrected using this sequence of observations in a particle filter framework to estimate an optimal similarity transform for fitting a patient-specific kidney model at each step. The temporal response of registration error is compared against that of serial rigid coherent point drift (CPD) in both static and simulated dynamic surgical fields, and for varying levels of observation persistence.

RESULTS

In the static case, both particle filtering and persistent CPD achieved sub-5 mm accuracy, with CPD processing observations 75% faster. Particle filtering outperformed CPD in the dynamic case under equivalent computation times due to the former requiring only minimal persistence.

CONCLUSION

This proof-of-concept simulation study suggests that Bayesian state estimation may provide a viable pathway to image guidance for surgical exposure in the abdomen, especially in the presence of dynamic intraoperative tissue displacement and deformation.

Local Style Preservation in Improved GAN-Driven Synthetic Image Generation for Endoscopic Tool Segmentation

Accurate semantic image segmentation from medical imaging can enable intelligent vision-based assistance in robot-assisted minimally invasive surgery. The human body and surgical procedures are highly dynamic. While machine-vision presents a promising approach, sufficiently large training image sets for robust performance are either costly or unavailable. This work examines three novel generative adversarial network (GAN) methods of providing usable synthetic tool images using only surgical background images and a few real tool images. The best of these three novel approaches generates realistic tool textures while preserving local background content by incorporating both a style preservation and a content loss component into the proposed multi-level loss function. The approach is quantitatively evaluated, and results suggest that the synthetically generated training tool images enhance UNet tool segmentation performance. More specifically, with a random set of 100 cadaver and live endoscopic images from the University of Washington Sinus Dataset, the UNet trained with synthetically generated images using the presented method resulted in 35.7% and 30.6% improvement over using purely real images in mean Dice coefficient and Intersection over Union scores, respectively. This study is promising towards the use of more widely available and routine screening endoscopy to preoperatively generate synthetic training tool images for intraoperative UNet tool segmentation.

Surgical navigation system for laparoscopic lateral pelvic lymph node dissection in rectal cancer surgery using laparoscopic-vision-tracked ultrasonic imaging

BACKGROUND

Laparoscopic lateral pelvic lymph node dissection (LPLND) in rectal cancer surgery requires considerable skill because the pelvic arteries, which need to be located to guide the dissection, are covered by other tissues and cannot be observed on laparoscopic views. Therefore, surgeons need to localize the pelvic arteries accurately before dissection, to prevent injury to these arteries.

METHODS

This report proposes a surgical navigation system to facilitate artery localization in laparoscopic LPLND by combining ultrasonic imaging and laparoscopy. Specifically, free-hand laparoscopic ultrasound (LUS) is employed to capture the arteries intraoperatively in this approach, and a laparoscopic vision-based tracking system is utilized to track the LUS probe. To extract the artery contours from the two-dimensional ultrasound image sequences efficiently, an artery extraction framework based on local phase-based snakes was developed. After reconstructing the three-dimensional intraoperative artery model from ultrasound images, a high-resolution artery model segmented from preoperative computed tomography (CT) images was rigidly registered to the intraoperative artery model and overlaid onto the laparoscopic view to guide laparoscopic LPLND.

RESULTS

Experiments were conducted to evaluate the performance of the vision-based tracking system, and the average reconstruction error of the proposed tracking system was found to be 2.4 mm. Then, the proposed navigation system was quantitatively evaluated on an artery phantom. The reconstruction time and average navigation error were 8 min and 2.3 mm, respectively. A navigation system was also successfully constructed to localize the pelvic arteries in laparoscopic and open surgeries of a swine. This demonstrated the feasibility of the proposed system in vivo. The construction times in the laparoscopic and open surgeries were 14 and 12 min, respectively.

CONCLUSIONS

The experimental results showed that the proposed navigation system can guide laparoscopic LPLND and requires a significantly shorter setting time than the state-of-the-art navigation systems do.

Virtual and augmented reality: potential applications in radiology

The modern-day radiologist must be adept at image interpretation, and the one who most successfully leverages new technologies may provide the highest value to patients, clinicians, and trainees. Applications of virtual reality (VR) and augmented reality (AR) have the potential to revolutionize how imaging information is applied in clinical practice and how radiologists practice. This review provides an overview of VR and AR, highlights current applications, future developments, and limitations hindering adoption.

A Holistically-Nested U-Net: Surgical Instrument Segmentation Based on Convolutional Neural Network

Surgical instrument segmentation is an essential task in the domain of computer-assisted surgical system. It is critical to increase the context-awareness of surgeons during the operation. We propose a new model based on the U-Net architecture for surgical instrument segmentation, which aggregates multi-scale feature maps and has cascaded dilated convolution layers. The model adopts dense upsampling convolution instead of deconvolution for upsampling. We set the side loss function on each side-output layer. The loss function includes an output loss function and all side loss functions to supervise the training of each layer. To validate our model, we compare our proposed model with advanced architecture U-Net in the dataset consisting of laparoscopy images from multiple surgical operations. Experiment results demonstrate that our model achieves good performance.

GPS Laparoscopic Ultrasound: Embedding an Electromagnetic Sensor in a Laparoscopic Ultrasound Transducer

Tracking the location and orientation of a laparoscopic ultrasound (LUS) transducer is a prerequisite in many surgical visualization and navigation applications. Electromagnetic (EM) tracking is a preferred method to track an LUS transducer with an articulating imaging tip. The conventional approach to integrating EM tracking with LUS is to attach an EM sensor on the outer surface of the imaging tip (external setup), which is not ideal for routine clinical use. In this work, we embedded an EM sensor inside a standard LUS transducer. We found that ultrasound image quality and the four-way articulation function of the transducer were not affected by this sensor integration. Furthermore, we found that the tracking accuracy of our integrated transducer was comparable to that of the external setup. An animal study conducted using the developed transducer suggests that an internally embedded EM sensor is a clinically more viable approach, and may be the future of tracking an articulating LUS transducer.

Uncertainty-aware asynchronous scattered motion interpolation using Gaussian process regression

We address the problem of interpolating randomly non-uniformly spatiotemporally scattered uncertain motion measurements, which arises in the context of soft tissue motion estimation. Soft tissue motion estimation is of great interest in the field of image-guided soft-tissue intervention and surgery navigation, because it enables the registration of pre-interventional/pre-operative navigation information on deformable soft-tissue organs. To formally define the measurements as spatiotemporally scattered motion signal samples, we propose a novel motion field representation. To perform the interpolation of the motion measurements in an uncertainty-aware optimal unbiased fashion, we devise a novel Gaussian process (GP) regression model with a non-constant-mean prior and an anisotropic covariance function and show through an extensive evaluation that it outperforms the state-of-the-art GP models that have been deployed previously for similar tasks. The employment of GP regression enables the quantification of uncertainty in the interpolation result, which would allow the amount of uncertainty present in the registered navigation information governing the decisions of the surgeon or intervention specialist to be conveyed.

Mobile, real-time, and point-of-care augmented reality is robust, accurate, and feasible: a prospective pilot study

BACKGROUND

Augmented reality (AR) systems are currently being explored by a broad spectrum of industries, mainly for improving point-of-care access to data and images. Especially in surgery and especially for timely decisions in emergency cases, a fast and comprehensive access to images at the patient bedside is mandatory. Currently, imaging data are accessed at a distance from the patient both in time and space, i.e., at a specific workstation. Mobile technology and 3-dimensional (3D) visualization of radiological imaging data promise to overcome these restrictions by making bedside AR feasible.

METHODS

In this project, AR was realized in a surgical setting by fusing a 3D-representation of structures of interest with live camera images on a tablet computer using marker-based registration. The intent of this study was to focus on a thorough evaluation of AR. Feasibility, robustness, and accuracy were thus evaluated consecutively in a phantom model and a porcine model. Additionally feasibility was evaluated in one male volunteer.

RESULTS

In the phantom model (n = 10), AR visualization was feasible in 84% of the visualization space with high accuracy (mean reprojection error ± standard deviation (SD): 2.8 ± 2.7 mm; 95th percentile = 6.7 mm). In a porcine model (n = 5), AR visualization was feasible in 79% with high accuracy (mean reprojection error ± SD: 3.5 ± 3.0 mm; 95th percentile = 9.5 mm). Furthermore, AR was successfully used and proved feasible within a male volunteer.

CONCLUSIONS

Mobile, real-time, and point-of-care AR for clinical purposes proved feasible, robust, and accurate in the phantom, animal, and single-trial human model shown in this study. Consequently, AR following similar implementation proved robust and accurate enough to be evaluated in clinical trials assessing accuracy, robustness in clinical reality, as well as integration into the clinical workflow. If these further studies prove successful, AR might revolutionize data access at patient bedside.

Navigation of Fluorescence Cameras during Soft Tissue Surgery-Is it Possible to Use a Single Navigation Setup for Various Open and Laparoscopic Urological Surgery Applications?

PURPOSE

Real-time visualization fluorescence imaging can guide surgeons during tissue resection. Unfortunately tissue induced signal attenuation limits the value of this technique to superficial applications. By positioning the fluorescence camera via a dedicated navigation setup we reasoned that the technology could be made compatible with deeper lesions, increasing its impact on clinical care. Such an impact would benefit from the ability to implement the navigation technology in different surgical settings. For that reason we evaluated whether a single fluorescence camera could be navigated toward targeted lesions during open and laparoscopic surgery.

MATERIALS AND METHODS

A fluorescence camera with scopes available for open and laparoscopic procedures was integrated with a navigation platform. Lymph nodes identified on SPECT/CT (single photon emission computerized tomography/computerized tomography) or free-hand single photon emission computerized tomography acted as navigation targets and were displayed as augmented overlays in the fluorescence camera video feed. The accuracy of this setup was evaluated in a phantom study of 4 scans per single photon emission computerized tomography imaging modality. This was followed by 4 first in human translations into sentinel lymph node biopsy procedures for penile (open surgery) and prostate (laparoscopic surgery) cancer.

RESULTS

Overall the phantom studies revealed a tool-target distance accuracy of 2.1 mm for SPECT/CT and 3.2 mm for freehand single photon emission computerized tomography, and an augmented reality registration accuracy of 1.1 and 2.2 mm, respectively. Subsequently open and laparoscopic navigation efforts were accurate enough to localize the fluorescence signals of the targeted tissues in vivo.

CONCLUSIONS

The phantom and human studies performed suggested that the single navigation setup is applicable in various open and laparoscopic urological surgery applications. Further evaluation in larger patient groups with a greater variety of malignancies is recommended to strengthen these results.

Registration and fusion quantification of augmented reality based nasal endoscopic surgery

This paper quantifies the registration and fusion display errors of augmented reality-based nasal endoscopic surgery (ARNES). We comparatively investigated the spatial calibration process for front-end endoscopy and redefined the accuracy level of a calibrated endoscope by using a calibration tool with improved structural reliability. We also studied how registration accuracy was combined with the number and distribution of the deployed fiducial points (FPs) for positioning and the measured registration time. A physically integrated ARNES prototype was customarily configured for performance evaluation in skull base tumor resection surgery with an innovative approach of dynamic endoscopic vision expansion. As advised by surgical experts in otolaryngology, we proposed a hierarchical rendering scheme to properly adapt the fused images with the required visual sensation. By constraining the rendered sight in a known depth and radius, the visual focus of the surgeon can be induced only on the anticipated critical anatomies and vessel structures to avoid misguidance. Furthermore, error analysis was conducted to examine the feasibility of hybrid optical tracking based on point cloud, which was proposed in our previous work as an in-surgery registration solution. Measured results indicated that the error of target registration for ARNES can be reduced to 0.77 ± 0.07 mm. For initial registration, our results suggest that a trade-off for a new minimal time of registration can be reached when the distribution of five FPs is considered. For in-surgery registration, our findings reveal that the intrinsic registration error is a major cause of performance loss. Rigid model and cadaver experiments confirmed that the scenic integration and display fluency of ARNES are smooth, as demonstrated by three clinical trials that surpassed practicality.

Biocompatible Near-Infrared Three-Dimensional Tracking System

A fundamental challenge in soft-tissue surgery is that target tissue moves and deforms, becomes occluded by blood or other tissue, and is difficult to differentiate from surrounding tissue. We developed small biocompatible near-infrared fluorescent (NIRF) markers with a novel fused plenoptic and NIR camera tracking system, enabling three-dimensional tracking of tools and target tissue while overcoming blood and tissue occlusion in the uncontrolled, rapidly changing surgical environment. In this work, we present the tracking system and marker design and compare tracking accuracies to standard optical tracking methods using robotic experiments. At speeds of 1 mm/s, we observe tracking accuracies of 1.61 mm, degrading only to 1.71 mm when the markers are covered in blood and tissue.

Automatic error correction using adaptive weighting for vessel-based deformable image registration

In this paper, we extend our previous work on deformable image registration to inhomogenous tissues. Inhomogenous tissues include the tissues with embedded tumors, which is common in clinical applications. It is a very challenging task since the registration method that works for homogenous tissues may not work well with inhomogenous tissues. The maximum error normally occurs in the regions with tumors and often exceeds the acceptable error threshold. In this paper, we propose a new error correction method with adaptive weighting to reduce the maximum registration error. Our previous fast deformable registration method is used in the inner loop. We have also proposed a new evaluation metric average error of deformation field (AEDF) to evaluate the registration accuracy in regions between vessels and bifurcation points. We have validated the proposed method using liver MR images from human subjects. AEDF results show that the proposed method can greatly reduce the maximum registration errors when compared with the previous method with no adaptive weighting. The proposed method has the potential to be used in clinical applications to reduce registration errors in regions with tumors.

A pilot study of SPECT/CT-based mixed-reality navigation towards the sentinel node in patients with melanoma or Merkel cell carcinoma of a lower extremity

OBJECTIVE

To explore the feasibility of an intraoperative navigation technology based on preoperatively acquired single photon emission computed tomography combined with computed tomography (SPECT/CT) images during sentinel node (SN) biopsy in patients with melanoma or Merkel cell carcinoma.

MATERIALS AND METHODS

Patients with a melanoma (n=4) or Merkel cell carcinoma (n=1) of a lower extremity scheduled for wide re-excision of the primary lesion site and SN biopsy were studied. Following a Tc-nanocolloid injection and lymphoscintigraphy, SPECT/CT images were acquired with a reference target (ReTp) fixed on the leg or the iliac spine. Intraoperatively, a sterile ReTp was placed at the same site to enable SPECT/CT-based mixed-reality navigation of a gamma ray detection probe also containing a reference target (ReTgp).The accuracy of the navigation procedure was determined in the coronal plane (x, y-axis) by measuring the discrepancy between standard gamma probe-based SN localization and mixed-reality-based navigation to the SN. To determine the depth accuracy (z-axis), the depth estimation provided by the navigation system was compared to the skin surface-to-node distance measured in the computed tomography component of the SPECT/CT images.

RESULTS

In four of five patients, it was possible to navigate towards the preoperatively defined SN. The average navigational error was 8.0 mm in the sagittal direction and 8.5 mm in the coronal direction. Intraoperative sterile ReTp positioning and tissue movement during surgery exerted a distinct influence on the accuracy of navigation.

CONCLUSION

Intraoperative navigation during melanoma or Merkel cell carcinoma surgery is feasible and can provide the surgeon with an interactive 3D roadmap towards the SN or SNs in the groin. However, further technical optimization of the modality is required before this technology can become routine practice.

Phantom Study Investigating the Accuracy of Manual and Automatic Image Fusion with the GE Logiq E9: Implications for use in Percutaneous Liver Interventions

Purpose

To determine the accuracy of automatic and manual co-registration methods for image fusion of three-dimensional computed tomography (CT) with real-time ultrasonography (US) for image-guided liver interventions.

Materials and Methods

CT images of a skills phantom with liver lesions were acquired and co-registered to US using GE Logiq E9 navigation software. Manual co-registration was compared to automatic and semiautomatic co-registration using an active tracker. Also, manual point registration was compared to plane registration with and without an additional translation point. Finally, comparison was made between manual and automatic selection of reference points. In each experiment, accuracy of the co-registration method was determined by measurement of the residual displacement in phantom lesions by two independent observers.

Results

Mean displacements for a superficial and deep liver lesion were comparable after manual and semiautomatic co-registration: 2.4 and 2.0 mm versus 2.0 and 2.5 mm, respectively. Both methods were significantly better than automatic co-registration: 5.9 and 5.2 mm residual displacement (p < 0.001; p < 0.01). The accuracy of manual point registration was higher than that of plane registration, the latter being heavily dependent on accurate matching of axial CT and US images by the operator. Automatic reference point selection resulted in significantly lower registration accuracy compared to manual point selection despite lower root-mean-square deviation (RMSD) values.

Conclusion

The accuracy of manual and semiautomatic co-registration is better than that of automatic co-registration. For manual co-registration using a plane, choosing the correct plane orientation is an essential first step in the registration process. Automatic reference point selection based on RMSD values is error-prone.

Vision-based and marker-less surgical tool detection and tracking: a review of the literature

In recent years, tremendous progress has been made in surgical practice for example with Minimally Invasive Surgery (MIS). To overcome challenges coming from deported eye-to-hand manipulation, robotic and computer-assisted systems have been developed. Having real-time knowledge of the pose of surgical tools with respect to the surgical camera and underlying anatomy is a key ingredient for such systems. In this paper, we present a review of the literature dealing with vision-based and marker-less surgical tool detection. This paper includes three primary contributions: (1) identification and analysis of data-sets used for developing and testing detection algorithms, (2) in-depth comparison of surgical tool detection methods from the feature extraction process to the model learning strategy and highlight existing shortcomings, and (3) analysis of validation techniques employed to obtain detection performance results and establish comparison between surgical tool detectors. The papers included in the review were selected through PubMed and Google Scholar searches using the keywords: "surgical tool detection", "surgical tool tracking", "surgical instrument detection" and "surgical instrument tracking" limiting results to the year range 2000 2015. Our study shows that despite significant progress over the years, the lack of established surgical tool data-sets, and reference format for performance assessment and method ranking is preventing faster improvement.

Navigation of a robot-integrated fluorescence laparoscope in preoperative SPECT/CT and intraoperative freehand SPECT imaging data: a phantom study

Robot-assisted laparoscopic surgery is becoming an established technique for prostatectomy and is increasingly being explored for other types of cancer. Linking intraoperative imaging techniques, such as fluorescence guidance, with the three-dimensional insights provided by preoperative imaging remains a challenge. Navigation technologies may provide a solution, especially when directly linked to both the robotic setup and the fluorescence laparoscope. We evaluated the feasibility of such a setup. Preoperative single-photon emission computed tomography/X-ray computed tomography (SPECT/CT) or intraoperative freehand SPECT (fhSPECT) scans were used to navigate an optically tracked robot-integrated fluorescence laparoscope via an augmented reality overlay in the laparoscopic video feed. The navigation accuracy was evaluated in soft tissue phantoms, followed by studies in a human-like torso phantom. Navigation accuracies found for SPECT/CT-based navigation were 2.25 mm (coronal) and 2.08 mm (sagittal). For fhSPECT-based navigation, these were 1.92 mm (coronal) and 2.83 mm (sagittal). All errors remained below the <1-cm detection limit for fluorescence imaging, allowing refinement of the navigation process using fluorescence findings. The phantom experiments performed suggest that SPECT-based navigation of the robot-integrated fluorescence laparoscope is feasible and may aid fluorescence-guided surgery procedures.

[Navigation in urological surgery: Possibilities and limits of current techniques]

Surgical navigation describes the concept of real-time processing and presentation of preoperative and intraoperative data from different sources to intraoperatively provide surgeons with additional cognitive support. Imaging methods such as 3D ultrasound, magnetic resonance imaging (MRI) and computed tomography (CT) and data from optical, electromagnetic or mechanical tracking methods are used. The resulting information of the navigation system will be presented by the means of visual methods. Mostly virtual reality or augmented reality visualization is used. There are different guidance systems for various disciplines introduced. Mostly it operates on rigid structures (bone, brain). For soft tissue navigation motion compensation and deformation detection are necessary. Therefore, marker-based tracking methods are used in several urological application examples; however, the systems are often still under development and have not yet arrived in the clinical routine.

Simultaneous Multi-Structure Segmentation and 3D Nonrigid Pose Estimation in Image-Guided Robotic Surgery

In image-guided robotic surgery, segmenting the endoscopic video stream into meaningful parts provides important contextual information that surgeons can exploit to enhance their perception of the surgical scene. This information provides surgeons with real-time decision-making guidance before initiating critical tasks such as tissue cutting. Segmenting endoscopic video is a challenging problem due to a variety of complications including significant noise attributed to bleeding and smoke from cutting, poor appearance contrast between different tissue types, occluding surgical tools, and limited visibility of the objects' geometries on the projected camera views. In this paper, we propose a multi-modal approach to segmentation where preoperative 3D computed tomography scans and intraoperative stereo-endoscopic video data are jointly analyzed. The idea is to segment multiple poorly visible structures in the stereo/multichannel endoscopic videos by fusing reliable prior knowledge captured from the preoperative 3D scans. More specifically, we estimate and track the pose of the preoperative models in 3D and consider the models' non-rigid deformations to match with corresponding visual cues in multi-channel endoscopic video and segment the objects of interest. Further, contrary to most augmented reality frameworks in endoscopic surgery that assume known camera parameters, an assumption that is often violated during surgery due to non-optimal camera calibration and changes in camera focus/zoom, our method embeds these parameters into the optimization hence correcting the calibration parameters within the segmentation process. We evaluate our technique on synthetic data, ex vivo lamb kidney datasets, and in vivo clinical partial nephrectomy surgery with results demonstrating high accuracy and robustness.

Application of a computer-aided navigation technique in surgery for recurrent malignant infratemporal fossa tumors

BACKGROUND

Because of the complexity of the local anatomy, tumors in the infratemporal fossa present a great challenge to oral and maxillofacial surgeons. Recurrent malignant tumors in this area are particularly difficult and precarious to resect because scars from previous operations may dislocate some important structures.

METHODS

From August 2010 to December 2013, all recurrent cases of malignant infratemporal fossa tumors at Peking University Stomatological Hospital were enrolled in this study. The patients were divided into 2 groups: the navigation group and the nonnavigation group, with different managements. The following factors were evaluated: operation time, bleeding volume, tumor size, surgical approach and complications, follow-up survey, and outcomes.In addition, survival analyses were performed for all patients.

RESULTS

In total, 42 patients were investigated. The mean operation time for the navigation group was not significantly longer than that of the nonnavigation group (283.64 versus 252.10 min, respectively; P = 0.393); the group's mean intraoperative bleeding volumes were similar (536.36 versus 503.87 mL, respectively; P = 0.814). The surgical approach was determined and categorized as an inferior approach (transmandibular approach, with or without splitting of the mandible), anterior approach (transmaxillary approach), lateral approach (subtemporal-preauricular approach), or combined approach. The inferior approach was most frequently used in both groups (ie, 63.6% for the navigation group and 80.6% for the nonnavigation group). The tumors were completely resected in 4 patients from the navigation group and 24 patients from the nonnavigation group. Regarding complications in the navigation and nonnavigation groups, the incidence was not significantly different (27.2% versus 41.9%, respectively; P = 0.485). The 3-year survival for patients in the navigation group was 71.6% compared with 52.9% in the nonnavigation group, with no significant difference. In the survival analysis, no significant factor was determined.

CONCLUSIONS

A computer-aided navigation technique has been successfully introduced to resect infratemporal fossa tumors and was successfully applied to the resection of recurrent malignant tumors. This new technique alone does not determine the outcome of patients with recurrent malignant infratemporal fossa tumors. Although some improvements are necessary, the visible navigation during surgery could increase the accuracy and safety of the operations and enhance surgeon confidence.

Toward real-time remote processing of laparoscopic video

Laparoscopic surgery is a minimally invasive surgical technique where surgeons insert a small video camera into the patient's body to visualize internal organs and use small tools to perform surgical procedures. However, the benefit of small incisions has a drawback of limited visualization of subsurface tissues, which can lead to navigational challenges in the delivering of therapy. Image-guided surgery uses the images to map subsurface structures and can reduce the limitations of laparoscopic surgery. One particular laparoscopic camera system of interest is the vision system of the daVinci-Si robotic surgical system (Intuitive Surgical, Sunnyvale, California). The video streams generate approximately 360 MB of data per second, demonstrating a trend toward increased data sizes in medicine, primarily due to higher-resolution video cameras and imaging equipment. Processing this data on a bedside PC has become challenging and a high-performance computing (HPC) environment may not always be available at the point of care. To process this data on remote HPC clusters at the typical 30 frames per second (fps) rate, it is required that each 11.9 MB video frame be processed by a server and returned within 1/30th of a second. The ability to acquire, process, and visualize data in real time is essential for the performance of complex tasks as well as minimizing risk to the patient. As a result, utilizing high-speed networks to access computing clusters will lead to real-time medical image processing and improve surgical experiences by providing real-time augmented laparoscopic data. We have performed image processing algorithms on a high-definition head phantom video (1920 × 1080 pixels) and transferred the video using a message passing interface. The total transfer time is around 53 ms or 19 fps. We will optimize and parallelize these algorithms to reduce the total time to 30 ms.

Sentinel node approach in prostate cancer

In general terms, one of the main objectives of sentinel lymph node (SLN) biopsy is to identify the 20-25% of patients with occult regional metastatic involvement. This technique reduces the associated morbidity from lymphadenectomy, as well as increasing the identification rate of occult lymphatic metastases by offering the pathologist those lymph nodes with the highest probability of containing metastatic cells. Pre-surgical lymphoscintigraphy is considered a "road map" to guide the surgeon towards the sentinel nodes and to ascertain unpredictable lymphatic drainages. In prostate cancer this aspect is essential due to the multidirectional character of the lymphatic drainage in the pelvis. In this context the inclusion of SPECT/CT should be mandatory in order to improve the SLN detection rate, to clarify the location when SLNs are difficult to interpret on planar images, to achieve a better definition of them in locations close to injection site, and to provide anatomical landmarks to be recognized during operation to locate SLNs. Conventional and laparoscopic hand-held gamma probes allow the SLN technique to be applied in any kind of surgery. The introduction and combination of new tracers and devices refines this technique, and the use of intraoperative images. These aspects become of vital importance due to the recent incorporation of robot-assisted procedures for SLN biopsy. In spite of these advances various aspects of SLN biopsy in prostate cancer patients still need to be discussed, and therefore their clinical application is not widely used.

Navigation surgery using an augmented reality for pancreatectomy

AIM

The aim of this study was to evaluate the utility of navigation surgery using augmented reality technology (AR-based NS) for pancreatectomy.

METHODS

The 3D reconstructed images from CT were created by segmentation. The initial registration was performed by using the optical location sensor. The reconstructed images were superimposed onto the real organs in the monitor display. Of the 19 patients who had undergone hepatobiliary and pancreatic surgery using AR-based NS, the accuracy, visualization ability, and utility of our system were assessed in five cases with pancreatectomy.

RESULTS

The position of each organ in the surface-rendering image corresponded almost to that of the actual organ. Reference to the display image allowed for safe dissection while preserving the adjacent vessels or organs. The locations of the lesions and resection line on the targeted organ were overlaid on the operating field. The initial mean registration error was improved to approximately 5 mm by our refinements. However, several problems such as registration accuracy, portability and cost still remain.

CONCLUSION

AR-based NS contributed to accurate and effective surgical resection in pancreatectomy. The pancreas appears to be a suitable organ for further investigations. This technology is promising to improve surgical quality, training, and education.

1D-3D registration for intra-operative nuclear imaging in radio-guided surgery

3D functional nuclear imaging modalities like SPECT or PET provide valuable information, as small structures can be marked with radioactive tracers to be localized before surgery. This positional information is valuable during surgery as well, for example when locating potentially cancerous lymph nodes in the case of breast cancer. However, the volumetric information provided by pre-operative SPECT scans loses validity quickly due to posture changes and manipulation of the soft tissue during surgery. During the intervention, the surgeon has to rely on the acoustic feedback provided by handheld gamma-detectors in order to localize the marked structures. In this paper, we present a method that allows updating the pre-operative image with a very limited number of tracked readings. A previously acquired 3D functional volume serves as prior knowledge and a limited number of new 1D detector readings is used in order to update the prior knowledge. This update is performed by a 1D-3D registration algorithm that registers the volume to the detector readings. This enables the rapid update of the visual guidance provided to the surgeon during a radio-guided surgery without slowing down the surgical workflow. We evaluate the performance of this approach using Monte-Carlo simulations, phantom experiments and patient data, resulting in a positional error of less than 8 mm which is acceptable for surgery. The 1D-3D registration is also compared to a volumetric reconstruction using the tracked detector measurements without taking prior information into account, and achieves a comparable accuracy with significantly less measurements.

Automated anatomical labeling of abdominal arteries and hepatic portal system extracted from abdominal CT volumes

This paper proposes a method for automated anatomical labeling of abdominal arteries and a hepatic portal system. In abdominal surgeries, understanding blood vessel structure is critical since it is very complicated. The input of the proposed method is the blood vessel region extracted from the CT volume. The blood vessel region is expressed as a tree structure by applying a thinning process to it and compute the mapping from the branches in the tree structure to the anatomical names. First, several characteristic anatomical names are assigned by rule-based pre-processing. The branches assigned to these names are used as references. The remaining blood vessel names are assigned using a likelihood function trained by a machine-learning technique. Simple rule-based postprocessing can correct several blood vessel names. The output of the proposed method is a tree structure with anatomical names. In an experiment using 50 blood vessel regions manually extracted from abdominal CT volumes, the recall and precision rates of the abdominal arteries were 86.2% and 85.3%, and they were 86.5% and 79.5% for the hepatic portal system.

Clinical application of navigation surgery using augmented reality in the abdominal field

This article presents general principles and recent advancements in the clinical application of augmented reality-based navigation surgery (AR based NS) for abdominal procedures and includes a description of our clinical trial and subsequent outcomes. Moreover, current problems and future aspects are discussed. The development of AR-based NS in the abdomen is delayed compared with another field because of the problem of intraoperative organ deformations or the existence of established modalities. Although there are a few reports on the clinical use of AR-based NS for digestive surgery, sophisticated technologies in urology have often been reported. However, the rapid widespread use of video- or robot assisted surgeries requires this technology. We have worked to develop a system of AR-based NS for hepatobiliary and pancreatic surgery. Then we developed a short rigid scope that enables surgeons to obtain 3D view. We recently focused on pancreatic surgery, because intraoperative organ shifting is minimal. The position of each organ in overlaid image almost corresponded with that of the actual organ with about 5 mm of mean registration errors. Intraoperative information generated from this system provided us with useful navigation. However, AR-based NS has several problems to overcome such as organ deformity, evaluation of utility, portability or cost.

Liver deformation in an animal model due to pneumoperitoneum assessed by a vessel-based deformable registration

PURPOSE

Surgical navigation based on preoperative images partly overcomes some of the drawbacks of minimally invasive interventions - reduction of free sight, lack of dexterity and tactile feedback. The usefulness of preoperative images is limited in laparoscopic liver surgery, as the liver shifts due to respiration, induction of pneumoperitoneum and surgical manipulation. In this study, we evaluated the shift and deformation in an animal liver caused by respiration and pneumopertioneum using intraoperative cone beam CT.

MATERIAL AND METHODS

3D cone beam CT scans were acquired with arterial contrast. The centerlines of the segmented vessels were extracted from the images taken at different respiration and pressure settings. A non-rigid registration method was used to measure the shift and deformation. The mean Euclidean distance between the annotated landmarks was used for evaluation.

RESULTS

A shift and deformation of 44.6 mm on average was introduced due to the combined effect of respiration and pneumoperitoneum. On average 91% of the deformations caused by the respiration and pneumoperitoneum were recovered.

CONCLUSION

The results can contribute to the use of intraoperative imaging to correct for anatomic shift so that preoperative data can be used with greater confidence and accuracy during guidance of laparoscopic liver procedures.

A novel 3-dimensional image analysis system for case-specific kidney anatomy and surgical simulation to facilitate clampless partial nephrectomy

OBJECTIVE

To report our initial experience with the novel 3-dimensional (3D) image analysis system Synapse Vincent in clampless partial nephrectomy (PN), describing its advantages with regard to short-term surgical outcomes and its usefulness as an informed consent tool.

METHODS

Twenty-six patients with renal cell carcinoma underwent clampless PN navigated with the aid of case-specific 3D anatomic video images of the kidney, after surgical simulation using the same video system. Baseline characteristics were reviewed, and short-term surgical outcomes were recorded. Of the 26, 6 had imperative indications, and 22 were treated with a minimally invasive approach. Before tumor excision, the renal hilar vessels were meticulously dissected, and definite tumor feeders were selectively ligated. Before patients consented to PN, the surgical procedure and perioperative risks were explained to each patient using case-specific 3D video images; subsequently, surgeons asked patients whether the 3D images had helped them understand PN more clearly than 2D images would have.

RESULTS

All operations were successfully completed without clamping, with negative surgical margins. No patients required blood transfusions. During PN, the surgeons confirmed the accuracy of the reconstructed 3D images and surgical simulations in all cases. All patients answered that the 3D images had helped them understand their disease status and surgical risks.

CONCLUSION

This is the first report on the Synapse Vincent 3D image analysis system for kidney surgery. Its 3D images and surgical simulation helped not only surgeons in their performance of clampless PN but also patients in their understanding of the operation.

Simultaneous evaluation of wall motion and blood perfusion of a beating heart using stereoscopic fluorescence camera system

In this study, we aimed to develop a stereoscopic fluorescence camera system for simultaneous evaluation of wall motion and tissue perfusion using indocyanine green (ICG) fluorescence imaging. The system consists of two high-speed stereo cameras, an excitation lamp, and a computer for image processing. Evaluation experiments demonstrated that the stereoscopic fluorescence camera system successfully performed the simultaneous measurement of wall motion and tissue perfusion based on ICG fluorescence imaging. Our system can be applied to intraoperative evaluation of cardiac status, leading to an improvement in surgical outcomes.

Surgical planning and manual image fusion based on 3D model facilitate laparoscopic partial nephrectomy for intrarenal tumors

PURPOSE

Construction of three-dimensional (3D) model of renal tumor facilitated surgical planning and imaging guidance of manual image fusion in laparoscopic partial nephrectomy (LPN) for intrarenal tumors.

MATERIALS AND METHODS

Fifteen patients with intrarenal tumors underwent LPN between January and December 2012. Computed tomography-based reconstruction of the 3D models of renal tumors was performed using Mimics 12.1 software. Surgical planning was performed through morphometry and multi-angle visual views of the tumor model. Two-step manual image fusion superimposed 3D model images onto 2D laparoscopic images. The image fusion was verified by intraoperative ultrasound. Imaging-guided laparoscopic hilar clamping and tumor excision was performed. Manual fusion time, patient demographics, surgical details, and postoperative treatment parameters were analyzed.

RESULTS

The reconstructed 3D tumor models accurately represented the patient's physiological anatomical landmarks. The surgical planning markers were marked successfully. Manual image fusion was flexible and feasible with fusion time of 6 min (5-7 min). All surgeries were completed laparoscopically. The median tumor excision time was 5.4 min (3.5-10 min), whereas the median warm ischemia time was 25.5 min (16-32 min). Twelve patients (80 %) demonstrated renal cell carcinoma on final pathology, and all surgical margins were negative. No tumor recurrence was detected after a media follow-up of 1 year (3-15 months).

CONCLUSIONS

The surgical planning and two-step manual image fusion based on 3D model of renal tumor facilitated visible-imaging-guided tumor resection with negative margin in LPN for intrarenal tumor. It is promising and moves us one step closer to imaging-guided surgery.