Intraoperative localized constrained registration in navigated bronchoscopy

Accuracy of instrument tip position using fiber optic shape sensing for navigated bronchoscopy

The purpose of this study was to evaluate the accuracy of a method for estimating the tip position of a fiber optic shape-sensing (FOSS) integrated instrument being inserted through a bronchoscope. A modified guidewire with a multicore optical fiber was inserted into the working channel of a custom-made catheter with three electromagnetic (EM) sensors. The displacement between the instruments was manually set, and a point-based method was applied to match the position of the EM sensors to corresponding points on the shape. The accuracy was evaluated in a realistic bronchial model. An additional EM sensor was used to sample the tip of the guidewire, and the absolute deviation between this position and the estimated tip position was calculated. For small displacements between the tip of the FOSS integrated tool and the catheter, the median deviation in estimated tip position was ≤5 mm. For larger displacements, deviations exceeding 10 mm were observed. The deviations increased when the shape sensor had sharp curvatures relative to more straight shapes. The method works well for clinically relevant displacements of a biopsy tool from the bronchoscope tip, and when the path to the lesion has limited curvatures. However, improvements must be made to our configuration before pursuing further clinical testing.

Augmented Reality Navigation System for Biliary Interventional Procedures With Dynamic Respiratory Motion Correction

OBJECTIVE

Biliary interventional procedures require physicians to track the interventional instrument tip (Tip) precisely with X-ray image. However, Tip positioning relies heavily on the physicians' experience due to the limitations of X-ray imaging and the respiratory interference, which leads to biliary damage, prolonged operation time, and increased X-ray radiation.

METHODS

We construct an augmented reality (AR) navigation system for biliary interventional procedures. It includes system calibration, respiratory motion correction and fusion navigation. Firstly, the magnetic and 3D computed tomography (CT) coordinates are aligned through system calibration. Secondly, a respiratory motion correction method based on manifold regularization is proposed to correct the misalignment of the two coordinates caused by respiratory motion. Thirdly, the virtual biliary, liver and Tip from CT are overlapped to the corresponding position of the patient for dynamic virtual-real fusion.

RESULTS

Our system is respectively evaluated and achieved an average alignment error of 0.75 ± 0.17 mm and 2.79 ± 0.46 mm on phantoms and patients. The navigation experiments conducted on phantoms achieve an average Tip positioning error of 0.98 ± 0.15 mm and an average fusion error of 1.67 ± 0.34 mm after correction.

CONCLUSION

Our system can automatically register the Tip to the corresponding location in CT, and dynamically overlap the 3D virtual model onto patients to provide accurate and intuitive AR navigation.

SIGNIFICANCE

This study demonstrates the clinical potential of our system by assisting physicians during biliary interventional procedures. Our system enables dynamic visualization of virtual model on patients, reducing the reliance on contrast agents and X-ray usage.

A cGAN-based network for depth estimation from bronchoscopic images

PURPOSE

Depth estimation is the basis of 3D reconstruction of airway structure from 2D bronchoscopic scenes, which can be further used to develop a vision-based bronchoscopic navigation system. This work aims to improve the performance of depth estimation directly from bronchoscopic images by training a depth estimation network on both synthetic and real datasets.

METHODS

We propose a cGAN-based network Bronchoscopic-Depth-GAN (BronchoDep-GAN) to estimate depth from bronchoscopic images by translating bronchoscopic images into depth maps. The network is trained in a supervised way learning from synthetic textured bronchoscopic image-depth pairs and virtual bronchoscopic image-depth pairs, and simultaneously, also in an unsupervised way learning from unpaired real bronchoscopic images and depth maps to adapt the model to real bronchoscopic scenes.

RESULTS

Our method is tested on both synthetic data and real data. However, the tests on real data are only qualitative, as no ground truth is available. The results show that our network obtains better accuracy in all cases in estimating depth from bronchoscopic images compared to the well-known cGANs pix2pix.

CONCLUSIONS

Including virtual and real bronchoscopic images in the training phase of the depth estimation networks can improve depth estimation's performance on both synthetic and real scenes. Further validation of this work is planned on 3D clinical phantoms. Based on the depth estimation results obtained in this work, the accuracy of locating bronchoscopes with corresponding pre-operative CTs will also be evaluated in comparison with the current clinical status.

Texture synthesis for generating realistic-looking bronchoscopic videos

PURPOSE

Synthetic realistic-looking bronchoscopic videos are needed to develop and evaluate depth estimation methods as part of investigating vision-based bronchoscopic navigation system. To generate these synthetic videos under the circumstance where access to real bronchoscopic images/image sequences is limited, we need to create various realistic-looking image textures of the airway inner surface with large size using a small number of real bronchoscopic image texture patches.

METHODS

A generative adversarial networks-based method is applied to create realistic-looking textures of the airway inner surface by learning from a limited number of small texture patches from real bronchoscopic images. By applying a purely convolutional architecture without any fully connected layers, this method allows the production of textures with arbitrary size.

RESULTS

Authentic image textures of airway inner surface are created. An example of the synthesized textures and two frames of the thereby generated bronchoscopic video are shown. The necessity and sufficiency of the generated textures as image features for further depth estimation methods are demonstrated.

CONCLUSIONS

The method can generate textures of the airway inner surface that meet the requirements for the texture itself and for the thereby generated bronchoscopic videos, including "realistic-looking," "long-term temporal consistency," "sufficient image features for depth estimation," and "large size and variety of synthesized textures." Besides, it also shows advantages with respect to the easy accessibility to required data source. A further validation of this approach is planned by utilizing the realistic-looking bronchoscopic videos with textures generated by this method as training and test data for some depth estimation networks.

Landmark Based Bronchoscope Localization for Needle Insertion Under Respiratory Deformation

Bronchoscopy is currently the least invasive method for definitively diagnosing lung cancer, which kills more people in the United States than any other form of cancer. Successfully diagnosing suspicious lung nodules requires accurate localization of the bronchoscope relative to a planned biopsy site in the airways. This task is challenging because the lung deforms intraoperatively due to respiratory motion, the airways lack photometric features, and the anatomy's appearance is repetitive. In this paper, we introduce a real-time camera-based method for accurately localizing a bronchoscope with respect to a planned needle insertion pose. Our approach uses deep learning and accounts for deformations and overcomes limitations of global pose estimation by estimating pose relative to anatomical landmarks. Specifically, our learned model considers airway bifurcations along the airway wall as landmarks because they are distinct geometric features that do not vary significantly with respiratory motion. We evaluate our method in a simulated dataset of lungs undergoing respiratory motion. The results show that our method generalizes across patients and localizes the bronchoscope with accuracy sufficient to access the smallest clinically-relevant nodules across all levels of respiratory deformation, even in challenging distal airways. Our method could enable physicians to perform more accurate biopsies and serve as a key building block toward accurate autonomous robotic bronchoscopy.

Bronchoscope motion tracking using centerline-guided Gaussian mixture model in navigated bronchoscopy

Electromagnetic-based navigation bronchoscopy requires accurate and robust estimation of the bronchoscope position inside the bronchial tree. However, respiratory motion, coughing, patient movement, and airway deformation inflicted by bronchoscope significantly hinder the accuracy of intraoperative bronchoscopic localization. In this study, a real-time and automatic registration procedure was proposed to superimpose the current location of the bronchoscope to corresponding locations on a centerline extracted from bronchial computed tomography (CT) images. A centerline-guided Gaussian mixture model (CG-GMM) was introduced to register a bronchoscope's position concerning extracted centerlines. A GMM was fitted to bronchoscope positions where the orientation likelihood was chosen to assign the membership probabilities of the mixture model, which led to preserving the global and local structures. The problem was formulated and solved under the expectation maximization framework, where the feature correspondence and spatial transformation are estimated iteratively. Validation was performed on a dynamic phantom with four different respiratory motions and four human real bronchoscopy (RB) datasets. Results of the experiments conducted on the bronchial phantom showed that the average positional tracking error using the proposed approach was equal to 1.98 [Formula: see text] 0.98 mm that was reduced in comparison with independent electromagnetic tracking (EMT), iterative closest point (ICP), and coherent point drift (CPD) methods by 64%, 58%, and 53%, respectively. In the patient assessment part of the study, the average positional tracking error was 4.73 [Formula: see text] 4.76 mm and compared to ICP, and CPD methods showed 31.4% improvement of successfully tracked frames. Our approach introduces a novel method for real-time respiratory motion compensation that provides reliable guidance during bronchoscopic interventions and, thus could increase the diagnostic yield of transbronchial biopsy.

Use of electromagnetic navigational bronchoscopy in robotic pulmonary resection

Robotic resection of pulmonary lesions has become a more common approach in the field of thoracic surgery. The greatest drawback of robotic resection is the lack of tactile feedback as compared to open approaches, making identification of intrapulmonary lesion difficult. Electromagnetic navigational bronchoscopy (navibronch) enables pre-incisional marking of pulmonary lesions for intraoperative identification. We sought to determine how effective navibronch was in our institution's robotic cases. Thirty-one patients underwent robotically assisted resection of 35 lesions with the assistance of navibronch from 7/2014 to 9/2015. Retrospective demographic and operative data were collected on these patients, and statistical analysis was conducted using ANOVA means testing, Chi-square, and non-parametric tests. The average age in this patient population was 63.7 ± 13.5 years. Eight patients (25.8%) were male. Twenty-five (80.6%) of the patients had pathology involving one lobe, with six (19.4%) in two lobes. 34 of the resections (97.1%) resulted in dye being localized to the first specimen; 34 (97.1%) were found to have the target pathology in the initial specimen. Further resection was carried out in 22 (62.9%) cases, with the final resection resulting in a segment in 2 (5.7%) and a lobe in 14 (40.0%). The mean number of lung specimens collected was 1.94 ± 0.13. The mean number of tumors in each target resection was 1.46 ± 0.66 in final pathology. Malignancy was found in 19 (54.3%) of final specimens. There were no complications related to navibronch. Navibronch is an effective technique in the identification and localization of pulmonary lesions in robotically assisted lung resections.