1
|
Acar A, Lu D, Wu Y, Oguz I, Kavoussi N, Wu JY. Towards navigation in endoscopic kidney surgery based on preoperative imaging. Healthc Technol Lett 2024; 11:67-75. [PMID: 38638503 PMCID: PMC11022214 DOI: 10.1049/htl2.12059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2023] [Accepted: 11/21/2023] [Indexed: 04/20/2024] Open
Abstract
Endoscopic renal surgeries have high re-operation rates, particularly for lower volume surgeons. Due to the limited field and depth of view of current endoscopes, mentally mapping preoperative computed tomography (CT) images of patient anatomy to the surgical field is challenging. The inability to completely navigate the intrarenal collecting system leads to missed kidney stones and tumors, subsequently raising recurrence rates. A guidance system is proposed to estimate the endoscope positions within the CT to reduce re-operation rates. A Structure from Motion algorithm is used to reconstruct the kidney collecting system from the endoscope videos. In addition, the kidney collecting system is segmented from CT scans using 3D U-Net to create a 3D model. The two collecting system representations can then be registered to provide information on the relative endoscope position. Correct reconstruction and localization of intrarenal anatomy and endoscope position is demonstrated. Furthermore, a 3D map is created supported by the RGB endoscope images to reduce the burden of mental mapping during surgery. The proposed reconstruction pipeline has been validated for guidance. It can reduce the mental burden for surgeons and is a step towards the long-term goal of reducing re-operation rates in kidney stone surgery.
Collapse
Affiliation(s)
- Ayberk Acar
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
- Present address:
Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| | - Daiwei Lu
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| | - Yifan Wu
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| | - Ipek Oguz
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| | - Nicholas Kavoussi
- Department of UrologyVanderbilt University Medical CenterNashvilleTennesseeUSA
| | - Jie Ying Wu
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| |
Collapse
|
2
|
Lu D, Wu Y, Acar A, Yao X, Wu JY, Kavoussi N, Oguz I. ASSIST-U: A system for segmentation and image style transfer for ureteroscopy. Healthc Technol Lett 2024; 11:40-47. [PMID: 38638492 PMCID: PMC11022208 DOI: 10.1049/htl2.12065] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2023] [Accepted: 11/22/2023] [Indexed: 04/20/2024] Open
Abstract
Kidney stones require surgical removal when they grow too large to be broken up externally or to pass on their own. Upper tract urothelial carcinoma is also sometimes treated endoscopically in a similar procedure. These surgeries are difficult, particularly for trainees who often miss tumours, stones or stone fragments, requiring re-operation. Furthermore, there are no patient-specific simulators to facilitate training or standardized visualization tools for ureteroscopy despite its high prevalence. Here a system ASSIST-U is proposed to create realistic ureteroscopy images and videos solely using preoperative computerized tomography (CT) images to address these unmet needs. A 3D UNet model is trained to automatically segment CT images and construct 3D surfaces. These surfaces are then skeletonized for rendering. Finally, a style transfer model is trained using contrastive unpaired translation (CUT) to synthesize realistic ureteroscopy images. Cross validation on the CT segmentation model achieved a Dice score of 0.853 ± 0.084. CUT style transfer produced visually plausible images; the kernel inception distance to real ureteroscopy images was reduced from 0.198 (rendered) to 0.089 (synthesized). The entire pipeline from CT to synthesized ureteroscopy is also qualitatively demonstrated. The proposed ASSIST-U system shows promise for aiding surgeons in the visualization of kidney ureteroscopy.
Collapse
Affiliation(s)
- Daiwei Lu
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| | - Yifan Wu
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| | - Ayberk Acar
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| | - Xing Yao
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| | - Jie Ying Wu
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| | - Nicholas Kavoussi
- Department of UrologyVanderbilt University Medical CenterNashvilleTennesseeUSA
| | - Ipek Oguz
- Department of Computer ScienceVanderbilt UniversityNashvilleTennesseeUSA
| |
Collapse
|
3
|
Zhang J, Liu L, Xiang P, Fang Q, Nie X, Ma H, Hu J, Xiong R, Wang Y, Lu H. AI co-pilot bronchoscope robot. Nat Commun 2024; 15:241. [PMID: 38172095 PMCID: PMC10764930 DOI: 10.1038/s41467-023-44385-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2023] [Accepted: 12/12/2023] [Indexed: 01/05/2024] Open
Abstract
The unequal distribution of medical resources and scarcity of experienced practitioners confine access to bronchoscopy primarily to well-equipped hospitals in developed regions, contributing to the unavailability of bronchoscopic services in underdeveloped areas. Here, we present an artificial intelligence (AI) co-pilot bronchoscope robot that empowers novice doctors to conduct lung examinations as safely and adeptly as experienced colleagues. The system features a user-friendly, plug-and-play catheter, devised for robot-assisted steering, facilitating access to bronchi beyond the fifth generation in average adult patients. Drawing upon historical bronchoscopic videos and expert imitation, our AI-human shared control algorithm enables novice doctors to achieve safe steering in the lung, mitigating misoperations. Both in vitro and in vivo results underscore that our system equips novice doctors with the skills to perform lung examinations as expertly as seasoned practitioners. This study offers innovative strategies to address the pressing issue of medical resource disparities through AI assistance.
Collapse
Affiliation(s)
- Jingyu Zhang
- State Key Laboratory of Industrial Control and Technology, Zhejiang University, 310027, Hangzhou, China
- Institute of Cyber-Systems and Control, Department of Control Science and Engineering, Zhejiang University, 310027, Hangzhou, China
| | - Lilu Liu
- State Key Laboratory of Industrial Control and Technology, Zhejiang University, 310027, Hangzhou, China
- Institute of Cyber-Systems and Control, Department of Control Science and Engineering, Zhejiang University, 310027, Hangzhou, China
| | - Pingyu Xiang
- State Key Laboratory of Industrial Control and Technology, Zhejiang University, 310027, Hangzhou, China
- Institute of Cyber-Systems and Control, Department of Control Science and Engineering, Zhejiang University, 310027, Hangzhou, China
| | - Qin Fang
- State Key Laboratory of Industrial Control and Technology, Zhejiang University, 310027, Hangzhou, China
- Institute of Cyber-Systems and Control, Department of Control Science and Engineering, Zhejiang University, 310027, Hangzhou, China
| | - Xiuping Nie
- State Key Laboratory of Industrial Control and Technology, Zhejiang University, 310027, Hangzhou, China
- Institute of Cyber-Systems and Control, Department of Control Science and Engineering, Zhejiang University, 310027, Hangzhou, China
| | - Honghai Ma
- Department of Thoracic Surgery, First Affiliated Hospital, School of Medicine, Zhejiang University, 310009, Hangzhou, China
| | - Jian Hu
- Department of Thoracic Surgery, First Affiliated Hospital, School of Medicine, Zhejiang University, 310009, Hangzhou, China
| | - Rong Xiong
- State Key Laboratory of Industrial Control and Technology, Zhejiang University, 310027, Hangzhou, China.
- Institute of Cyber-Systems and Control, Department of Control Science and Engineering, Zhejiang University, 310027, Hangzhou, China.
| | - Yue Wang
- State Key Laboratory of Industrial Control and Technology, Zhejiang University, 310027, Hangzhou, China.
- Institute of Cyber-Systems and Control, Department of Control Science and Engineering, Zhejiang University, 310027, Hangzhou, China.
| | - Haojian Lu
- State Key Laboratory of Industrial Control and Technology, Zhejiang University, 310027, Hangzhou, China.
- Institute of Cyber-Systems and Control, Department of Control Science and Engineering, Zhejiang University, 310027, Hangzhou, China.
| |
Collapse
|
4
|
Liu S, Fan J, Zang L, Yang Y, Fu T, Song H, Wang Y, Yang J. Pose estimation via structure-depth information from monocular endoscopy images sequence. BIOMEDICAL OPTICS EXPRESS 2024; 15:460-478. [PMID: 38223180 PMCID: PMC10783895 DOI: 10.1364/boe.498262] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/16/2023] [Revised: 12/08/2023] [Accepted: 12/14/2023] [Indexed: 01/16/2024]
Abstract
Image-based endoscopy pose estimation has been shown to significantly improve the visualization and accuracy of minimally invasive surgery (MIS). This paper proposes a method for pose estimation based on structure-depth information from a monocular endoscopy image sequence. Firstly, the initial frame location is constrained using the image structure difference (ISD) network. Secondly, endoscopy image depth information is used to estimate the pose of sequence frames. Finally, adaptive boundary constraints are used to optimize continuous frame endoscopy pose estimation, resulting in more accurate intraoperative endoscopy pose estimation. Evaluations were conducted on publicly available datasets, with the pose estimation error in bronchoscopy and colonoscopy datasets reaching 1.43 mm and 3.64 mm, respectively. These results meet the real-time requirements of various scenarios, demonstrating the capability of this method to generate reliable pose estimation results for endoscopy images and its meaningful applications in clinical practice. This method enables accurate localization of endoscopy images during surgery, assisting physicians in performing safer and more effective procedures.
Collapse
Affiliation(s)
- Shiyuan Liu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
- China Center for Information Industry Development, Beijing 100081, China
| | - Jingfan Fan
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
| | - Liugeng Zang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
| | - Yun Yang
- Department of General Surgery, Beijing Friendship Hospital, Capital Medical University; National Clinical Research Center for Digestive Diseases, Beijing 100050, China
| | - Tianyu Fu
- Institute of Engineering Medicine, Beijing Institute of Technology, Beijing 100081, China
| | - Hong Song
- School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China
| | - Yongtian Wang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
| | - Jian Yang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
| |
Collapse
|
5
|
Zhao W, Ahmad D, Toth J, Bascom R, Higgins WE. Endobronchial Ultrasound Image Simulation for Image-Guided Bronchoscopy. IEEE Trans Biomed Eng 2023; 70:318-330. [PMID: 35819999 PMCID: PMC9927880 DOI: 10.1109/tbme.2022.3190165] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
BACKGROUND/OBJECTIVE Accurate disease diagnosis and staging are essential for patients suspected of having lung cancer. The state-of-the-art minimally invasive tools used by physicians to perform these operations are bronchoscopy, for navigating the lung airways, and endobronchial ultrasound (EBUS), for localizing suspect extraluminal cancer lesions. While new image-guided systems enable accurate bronchoscope navigation close to a lesion, no means exists for guiding the final EBUS localization of an extraluminal lesion. We propose an EBUS simulation method to assist with EBUS localization. METHODS The method draws on a patient's chest computed-tomography (CT) scan to model the ultrasound signal propagation through the tissue media. The method, which is suitable for simulating EBUS images for both radial-probe and convex-probe EBUS devices, entails three steps: 1) image preprocessing, which generates a 2D CT equivalent of the EBUS scan plane; 2) EBUS scan-line computation, which models ultrasound transmission to map the CT plane into a preliminary simulated EBUS image; and 3) image post-processing, which increases realism by introducing simulated EBUS imaging effects and artifacts. RESULTS Results show that the method produces simulated EBUS images that strongly resemble images generated live by a real device and compares favorably to an existing ultrasound simulation method. It also produces images at a rate greater than real time (i.e., 53 frames/sec). We also demonstrate a successful integration of the method into an image-guided EBUS bronchoscopy system. CONCLUSION/SIGNIFICANCE The method is effective and practical for procedure planning/preview and follow-on live guidance of EBUS bronchoscopy.
Collapse
Affiliation(s)
- Wennan Zhao
- Wennan Zhao is with the School of Electrical Engineering and Computer Science, Pennsylvania State University, University Park, PA 16802 USA. D. Ahmad, J. Toth, and R. Bascom are with the College of Medicine, Penn State Milton S. Hershey Medical Center, Hershey, PA, 17033 USA
| | - Danish Ahmad
- Wennan Zhao is with the School of Electrical Engineering and Computer Science, Pennsylvania State University, University Park, PA 16802 USA. D. Ahmad, J. Toth, and R. Bascom are with the College of Medicine, Penn State Milton S. Hershey Medical Center, Hershey, PA, 17033 USA
| | - Jennifer Toth
- Wennan Zhao is with the School of Electrical Engineering and Computer Science, Pennsylvania State University, University Park, PA 16802 USA. D. Ahmad, J. Toth, and R. Bascom are with the College of Medicine, Penn State Milton S. Hershey Medical Center, Hershey, PA, 17033 USA
| | - Rebecca Bascom
- Wennan Zhao is with the School of Electrical Engineering and Computer Science, Pennsylvania State University, University Park, PA 16802 USA. D. Ahmad, J. Toth, and R. Bascom are with the College of Medicine, Penn State Milton S. Hershey Medical Center, Hershey, PA, 17033 USA
| | - William E. Higgins
- Wennan Zhao is with the School of Electrical Engineering and Computer Science, Pennsylvania State University, University Park, PA 16802 USA. D. Ahmad, J. Toth, and R. Bascom are with the College of Medicine, Penn State Milton S. Hershey Medical Center, Hershey, PA, 17033 USA
| |
Collapse
|
6
|
Borrego-Carazo J, Sanchez C, Castells-Rufas D, Carrabina J, Gil D. BronchoPose: an analysis of data and model configuration for vision-based bronchoscopy pose estimation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 228:107241. [PMID: 36434960 DOI: 10.1016/j.cmpb.2022.107241] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/22/2022] [Revised: 10/07/2022] [Accepted: 11/08/2022] [Indexed: 06/16/2023]
Abstract
BACKGROUND AND OBJECTIVE Recent advances in neural networks and temporal image processing have provided new results and opportunities for vision-based bronchoscopy tracking. However, such progress has been hindered by the lack of comparative experimental data conditions. We address the issue by sharing a novel synthetic dataset, which allows for a fair comparison of methods. Moreover, as incorporating deep learning advances in temporal structures is not yet explored in bronchoscopy navigation, we investigate several neural network architectures for learning temporal information at different levels of subject personalization, providing new insights and results. METHODS Using our own shared synthetic dataset for bronchoscopy navigation and tracking, we explore deep learning temporal information architectures (Recurrent Neural Networks and 3D convolutions), which have not been fully explored on bronchoscopy tracking, putting a special focus on network efficiency by using a modern backbone (EfficientNet-B0) and ShuffleNet blocks. Finally, we provide a study of different losses for rotation tracking and population modeling schemes (personalized vs. population) for bronchoscopy tracking. RESULTS Temporal information architectures provide performance improvements, both in position and angle estimation. Additionally, population scheme analysis illustrates the benefits of offering a personalized model, while loss analysis indicates the benefits of using an adequate metric, improving results. We finally compare with a state-of-the-art model obtaining better results both in performance, with 12.2% and 18.7% improvement for position and rotation respectively, and around 67.6% reduction in memory consumption. CONCLUSIONS Proposed advances in temporal information architectures, loss configuration, and population scheme definition allow for improving the current state of the art in bronchoscopy analysis. Moreover, the publication of the first synthetic dataset allows for further improving bronchoscopy research by enabling proper comparison grounds among methods.
Collapse
Affiliation(s)
- Juan Borrego-Carazo
- Computer Vision Center, Universitat Autònoma de Barcelona, Cerdanyola del Vallès 08193, Spain; Department of Microelectronics & Electronic Systems, Universitat Autònoma de Barcelona, Cerdanyola del Vallès 08193, Spain.
| | - Carles Sanchez
- Computer Vision Center, Universitat Autònoma de Barcelona, Cerdanyola del Vallès 08193, Spain
| | - David Castells-Rufas
- Department of Microelectronics & Electronic Systems, Universitat Autònoma de Barcelona, Cerdanyola del Vallès 08193, Spain
| | - Jordi Carrabina
- Department of Microelectronics & Electronic Systems, Universitat Autònoma de Barcelona, Cerdanyola del Vallès 08193, Spain
| | - Débora Gil
- Computer Vision Center, Universitat Autònoma de Barcelona, Cerdanyola del Vallès 08193, Spain
| |
Collapse
|
7
|
Wang C, Oda M, Hayashi Y, Kitasaka T, Itoh H, Honma H, Takebatake H, Mori M, Natori H, Mori K. Anatomy aware-based 2.5D bronchoscope tracking for image-guided bronchoscopic navigation. COMPUTER METHODS IN BIOMECHANICS AND BIOMEDICAL ENGINEERING: IMAGING & VISUALIZATION 2022. [DOI: 10.1080/21681163.2022.2152728] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Affiliation(s)
- Cheng Wang
- Graduate School of Informatics, Nagoya University, Nagoya, Japan
| | - Masahiro Oda
- Graduate School of Informatics, Nagoya University, Nagoya, Japan
- Information and Communications, Nagoya University, Nagoya, Japan
| | - Yuichiro Hayashi
- Graduate School of Informatics, Nagoya University, Nagoya, Japan
| | - Takayuki Kitasaka
- School of Information Science, Aichi Institute of Technology, Toyota, Japan
| | - Hayato Itoh
- Graduate School of Informatics, Nagoya University, Nagoya, Japan
| | - Hirotoshi Honma
- Medical Examination Department, Seamen’s Insurance Hokkaido Healthcare Center, Sapporo, Japan
| | - Hirotsugu Takebatake
- Department of Respiratory Medicine, Sapporo Minami-Sanjo Hospital, Sapporo, Japan
| | - Masaki Mori
- Department of Respiratory Medicine, Sapporo-Kosei General Hospital, Sapporo, Japan
| | - Hiroshi Natori
- Department of Internal Medicine, Keiwakai Nishioka Hospital, Sapporo, Japan
| | - Kensaku Mori
- Graduate School of Informatics, Nagoya University, Nagoya, Japan
- Information Technology Center, Nagoya University, Nagoya, Japan
- Research Center for Medical Bigdata, National Institute of Informatics, Tokyo, Japan
| |
Collapse
|
8
|
Gu Y, Gu C, Yang J, Sun J, Yang GZ. Vision-Kinematics Interaction for Robotic-Assisted Bronchoscopy Navigation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:3600-3610. [PMID: 35839186 DOI: 10.1109/tmi.2022.3191317] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Endobronchial intervention is increasingly used as a minimally invasive means for the treatment of pulmonary diseases. In order to acquire the position of bronchoscopy, vision-based localization approaches are clinically preferable but are sensitive to visual variations. The static nature of pre-operative planning makes mapping of intraoperative anatomical features challenging for learning-based methods using visual features alone. In this work, we propose a robust navigation framework based on Vision Kinematic Interaction (VKI) for monocular bronchoscopic videos. To address visual-imbalance between the virtual and real views of bronchoscopy images, a Visual Similarity Network (VSN) is proposed to extract domain-invariant features to represent the lumen structure from endoscopic views, as well as domain-specific features to characterize the surface texture and visual artefacts. To improve the robustness of online estimation of camera pose, we also introduce a Kinematic Refinement Network (KRN) that allows progressive refinement of camera pose estimation based on network prediction and robot control signals. The accuracy of camera localization is validated on phantom and porcine lung datasets from a robotically controlled endobronchial intervention system, with both quantitative and qualitative results demonstrating the performance of the techniques. Results show that the features extracted by the proposed method can preserve the structural information of small airways in the presence of large visual variations along with the much-improved camera localization accuracy. The absolute trajectory errors (ATE) on phantom data and porcine data are 8.01 mm and 8.62 mm respectively.
Collapse
|
9
|
Multimodal Registration for Image-Guided EBUS Bronchoscopy. J Imaging 2022; 8:jimaging8070189. [PMID: 35877633 PMCID: PMC9320860 DOI: 10.3390/jimaging8070189] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2022] [Revised: 06/27/2022] [Accepted: 06/29/2022] [Indexed: 12/24/2022] Open
Abstract
The state-of-the-art procedure for examining the lymph nodes in a lung cancer patient involves using an endobronchial ultrasound (EBUS) bronchoscope. The EBUS bronchoscope integrates two modalities into one device: (1) videobronchoscopy, which gives video images of the airway walls; and (2) convex-probe EBUS, which gives 2D fan-shaped views of extraluminal structures situated outside the airways. During the procedure, the physician first employs videobronchoscopy to navigate the device through the airways. Next, upon reaching a given node’s approximate vicinity, the physician probes the airway walls using EBUS to localize the node. Due to the fact that lymph nodes lie beyond the airways, EBUS is essential for confirming a node’s location. Unfortunately, it is well-documented that EBUS is difficult to use. In addition, while new image-guided bronchoscopy systems provide effective guidance for videobronchoscopic navigation, they offer no assistance for guiding EBUS localization. We propose a method for registering a patient’s chest CT scan to live surgical EBUS views, thereby facilitating accurate image-guided EBUS bronchoscopy. The method entails an optimization process that registers CT-based virtual EBUS views to live EBUS probe views. Results using lung cancer patient data show that the method correctly registered 28/28 (100%) lymph nodes scanned by EBUS, with a mean registration time of 3.4 s. In addition, the mean position and direction errors of registered sites were 2.2 mm and 11.8∘, respectively. In addition, sensitivity studies show the method’s robustness to parameter variations. Lastly, we demonstrate the method’s use in an image-guided system designed for guiding both phases of EBUS bronchoscopy.
Collapse
|
10
|
Banach A, King F, Masaki F, Tsukada H, Hata N. Visually Navigated Bronchoscopy using three cycle-Consistent generative adversarial network for depth estimation. Med Image Anal 2021; 73:102164. [PMID: 34314953 PMCID: PMC8453111 DOI: 10.1016/j.media.2021.102164] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2021] [Revised: 06/29/2021] [Accepted: 07/06/2021] [Indexed: 11/30/2022]
Abstract
[Background] Electromagnetically Navigated Bronchoscopy (ENB) is currently the state-of-the art diagnostic and interventional bronchoscopy. CT-to-body divergence is a critical hurdle in ENB, causing navigation error and ultimately limiting the clinical efficacy of diagnosis and treatment. In this study, Visually Navigated Bronchoscopy (VNB) is proposed to address the aforementioned issue of CT-to-body divergence. [Materials and Methods] We extended and validated an unsupervised learning method to generate a depth map directly from bronchoscopic images using a Three Cycle-Consistent Generative Adversarial Network (3cGAN) and registering the depth map to preprocedural CTs. We tested the working hypothesis that the proposed VNB can be integrated to the navigated bronchoscopic system based on 3D Slicer, and accurately register bronchoscopic images to pre-procedural CTs to navigate transbronchial biopsies. The quantitative metrics to asses the hypothesis we set was Absolute Tracking Error (ATE) of the tracking and the Target Registration Error (TRE) of the total navigation system. We validated our method on phantoms produced from the pre-procedural CTs of five patients who underwent ENB and on two ex-vivo pig lung specimens. [Results] The ATE using 3cGAN was 6.2 +/- 2.9 [mm]. The ATE of 3cGAN was statistically significantly lower than that of cGAN, particularly in the trachea and lobar bronchus (p < 0.001). The TRE of the proposed method had a range of 11.7 to 40.5 [mm]. The TRE computed by 3cGAN was statistically significantly smaller than those computed by cGAN in two of the five cases enrolled (p < 0.05). [Conclusion] VNB, using 3cGAN to generate the depth maps was technically and clinically feasible. While the accuracy of tracking by cGAN was acceptable, the TRE warrants further investigation and improvement.
Collapse
Affiliation(s)
- Artur Banach
- National Center for Image-guided Therapy, Department of Radiology, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, United States; QUT Centre for Robotics, Queensland University of Technology, Brisbane, Australia.
| | - Franklin King
- National Center for Image-guided Therapy, Department of Radiology, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, United States
| | - Fumitaro Masaki
- National Center for Image-guided Therapy, Department of Radiology, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, United States; Healthcare Optics Research Laboratory, Canon U.S.A., Cambridge, MA, United States
| | - Hisashi Tsukada
- Division of Thoracic Surgery, Department of Surgery, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, United States
| | - Nobuhiko Hata
- National Center for Image-guided Therapy, Department of Radiology, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, United States
| |
Collapse
|
11
|
Abstract
The staging of the central-chest lymph nodes is a major step in the management of lung-cancer patients. For this purpose, the physician uses a device that integrates videobronchoscopy and an endobronchial ultrasound (EBUS) probe. To biopsy a lymph node, the physician first uses videobronchoscopy to navigate through the airways and then invokes EBUS to localize and biopsy the node. Unfortunately, this process proves difficult for many physicians, with the choice of biopsy site found by trial and error. We present a complete image-guided EBUS bronchoscopy system tailored to lymph-node staging. The system accepts a patient’s 3D chest CT scan, an optional PET scan, and the EBUS bronchoscope’s video sources as inputs. System workflow follows two phases: (1) procedure planning and (2) image-guided EBUS bronchoscopy. Procedure planning derives airway guidance routes that facilitate optimal EBUS scanning and nodal biopsy. During the live procedure, the system’s graphical display suggests a series of device maneuvers to perform and provides multimodal visual cues for locating suitable biopsy sites. To this end, the system exploits data fusion to drive a multimodal virtual bronchoscope and other visualization tools that lead the physician through the process of device navigation and localization. A retrospective lung-cancer patient study and follow-on prospective patient study, performed within the standard clinical workflow, demonstrate the system’s feasibility and functionality. For the prospective study, 60/60 selected lymph nodes (100%) were correctly localized using the system, and 30/33 biopsied nodes (91%) gave adequate tissue samples. Also, the mean procedure time including all user interactions was 6 min 43 s All of these measures improve upon benchmarks reported for other state-of-the-art systems and current practice. Overall, the system enabled safe, efficient EBUS-based localization and biopsy of lymph nodes.
Collapse
|
12
|
Depth-based branching level estimation for bronchoscopic navigation. Int J Comput Assist Radiol Surg 2021; 16:1795-1804. [PMID: 34392469 DOI: 10.1007/s11548-021-02460-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2020] [Accepted: 07/12/2021] [Indexed: 10/20/2022]
Abstract
PURPOSE Bronchoscopists rely on navigation systems during bronchoscopy to reduce the risk of getting lost in the complex bronchial tree-like structure and the homogeneous bronchus lumens. We propose a patient-specific branching level estimation method for bronchoscopic navigation because it is vital to identify the branches being examined in the bronchus tree during examination. METHODS We estimate the branching level by integrating the changes in the number of bronchial orifices and the camera motions among the frames. We extract the bronchial orifice regions from a depth image, which is generated using a cycle generative adversarial network (CycleGAN) from real bronchoscopic images. We calculate the number of orifice regions using the vertical and horizontal projection profiles of the depth images and obtain the camera-moving direction using the feature point-based camera motion estimation. The changes in the number of bronchial orifices are combined with the camera-moving direction to estimate the branching level. RESULTS We used three in vivo and one phantom case to train the CycleGAN model and four in vivo cases to validate the proposed method. We manually created the ground truth of the branching level. The experimental results showed that the proposed method can estimate the branching level with an average accuracy of 87.6%. The processing time per frame was about 61 ms. CONCLUSION Experimental results show that it is feasible to estimate the branching level using the number of bronchial orifices and camera-motion estimation from real bronchoscopic images.
Collapse
|
13
|
Ramalhinho J, Tregidgo HFJ, Gurusamy K, Hawkes DJ, Davidson B, Clarkson MJ. Registration of Untracked 2D Laparoscopic Ultrasound to CT Images of the Liver Using Multi-Labelled Content-Based Image Retrieval. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:1042-1054. [PMID: 33326379 DOI: 10.1109/tmi.2020.3045348] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Laparoscopic Ultrasound (LUS) is recommended as a standard-of-care when performing laparoscopic liver resections as it images sub-surface structures such as tumours and major vessels. Given that LUS probes are difficult to handle and some tumours are iso-echoic, registration of LUS images to a pre-operative CT has been proposed as an image-guidance method. This registration problem is particularly challenging due to the small field of view of LUS, and usually depends on both a manual initialisation and tracking to compose a volume, hindering clinical translation. In this paper, we extend a proposed registration approach using Content-Based Image Retrieval (CBIR), removing the requirement for tracking or manual initialisation. Pre-operatively, a set of possible LUS planes is simulated from CT and a descriptor generated for each image. Then, a Bayesian framework is employed to estimate the most likely sequence of CT simulations that matches a series of LUS images. We extend our CBIR formulation to use multiple labelled objects and constrain the registration by separating liver vessels into portal vein and hepatic vein branches. The value of this new labeled approach is demonstrated in retrospective data from 5 patients. Results show that, by including a series of 5 untracked images in time, a single LUS image can be registered with accuracies ranging from 5.7 to 16.4 mm with a success rate of 78%. Initialisation of the LUS to CT registration with the proposed framework could potentially enable the clinical translation of these image fusion techniques.
Collapse
|
14
|
A visual SLAM-based bronchoscope tracking scheme for bronchoscopic navigation. Int J Comput Assist Radiol Surg 2020; 15:1619-1630. [DOI: 10.1007/s11548-020-02241-9] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2020] [Accepted: 07/21/2020] [Indexed: 10/23/2022]
|
15
|
Shi RB, Mirza S, Martinez D, Douglas C, Cho J, Irish JC, Jaffray DA, Weersink RA. Cost-function testing methodology for image-based registration of endoscopy to CT images in the head and neck. Phys Med Biol 2020; 65. [PMID: 32702685 DOI: 10.1088/1361-6560/aba8b3] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2020] [Accepted: 07/23/2020] [Indexed: 11/11/2022]
Abstract
One of the largest geometric uncertainties in designing radiotherapy treatment plans for squamous cell cancers of the head and neck is contouring the gross tumour volume. We have previously described a method of projecting mucosal disease contours, visible on endoscopy, to volumetrically reconstructed planning CT datasets, using electromagnetic (EM) tracking of a flexible endoscope, enabling rigid registration between endoscopic and CT images. However, to achieve better accuracy for radiotherapy planning, we propose refining this initial registration with image-based registration methods. In this paper, several types of cost functions are evaluated based on accuracy and robustness. Three phantoms and eight clinical cases are used to test each cost function, with initial registration of endoscopy to CT provided by the pose of the flexible endoscope recovered from EM tracking. Cost function classes include: cross correlation, mutual information and gradient methods. For each test case, a ground truth virtual camera pose was first defined by manual registration of anatomical features visible in both real and virtual endoscope images. A new set of evenly spaced fiducial points and a sample contour were created and projected onto the CT image to be used in assessing image registration quality. A new set of 5000 displaced poses was generated by random sampling displacements along each translational and rotational dimension. At each pose, fiducial and contour points in the real image were again projected on the CT image. The cost function, fiducial registration error and contouring error values were then calculated. While all cost functions performed well in select cases, only the normalized gradient field function consistently had registration errors less than 2 mm, which is the accuracy needed if this application of registering mucosal disease identified on optical image to CT images is to be used in the clinical practice of radiation treatment planning. (Registration: ClinicalTrials.gov NCT02704169).
Collapse
Affiliation(s)
| | - Souzan Mirza
- University of Toronto Institute of Biomaterials and Biomedical Engineering, Toronto, Ontario, CANADA
| | - Diego Martinez
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Ontario, CANADA
| | - Catriona Douglas
- Surgical Oncology, University of Toronto Department of Surgery, Toronto, Ontario, CANADA
| | - John Cho
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Ontario, CANADA
| | - Jonathan C Irish
- Surgical Oncology, University of Toronto Department of Surgery, Toronto, Ontario, CANADA
| | - David A Jaffray
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Ontario, CANADA
| | - Robert A Weersink
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Ontario, CANADA
| |
Collapse
|
16
|
Zang X, Gibbs JD, Cheirsilp R, Byrnes PD, Toth J, Bascom R, Higgins WE. Optimal route planning for image-guided EBUS bronchoscopy. Comput Biol Med 2019; 112:103361. [PMID: 31362107 PMCID: PMC6820695 DOI: 10.1016/j.compbiomed.2019.103361] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2019] [Revised: 07/16/2019] [Accepted: 07/16/2019] [Indexed: 12/25/2022]
Abstract
The staging of the central-chest lymph nodes is a major lung-cancer management procedure. To perform a staging procedure, the physician first uses a patient's 3D X-ray computed-tomography (CT) chest scan to interactively plan airway routes leading to selected target lymph nodes. Next, using an integrated EBUS bronchoscope (EBUS = endobronchial ultrasound), the physician uses videobronchoscopy to navigate through the airways toward a target node's general vicinity and then invokes EBUS to localize the node for biopsy. Unfortunately, during the procedure, the physician has difficulty in translating the preplanned airway routes into safe, effective biopsy sites. We propose an automatic route-planning method for EBUS bronchoscopy that gives optimal localization of safe, effective nodal biopsy sites. To run the method, a 3D chest model is first computed from a patient's chest CT scan. Next, an optimization method derives feasible airway routes that enables maximal tissue sampling of target lymph nodes while safely avoiding major blood vessels. In a lung-cancer patient study entailing 31 nodes (long axis range: [9.0 mm, 44.5 mm]), 25/31 nodes yielded safe airway routes having an optimal tissue sample size = 8.4 mm (range: [1.0 mm, 18.6 mm]) and sample adequacy = 0.42 (range: [0.05, 0.93]). Quantitative results indicate that the method potentially enables successful biopsies in essentially 100% of selected lymph nodes versus the 70-94% success rate of other approaches. The method also potentially facilitates adequate tissue biopsies for nearly 100% of selected nodes, as opposed to the 55-77% tissue adequacy rates of standard methods. The remaining nodes did not yield a safe route within the preset safety-margin constraints, with 3 nodes never yielding a route even under the most lenient safety-margin conditions. Thus, the method not only helps determine effective airway routes and expected sample quality for nodal biopsy, but it also helps point out situations where biopsy may not be advisable. We also demonstrate the methodology in an image-guided EBUS bronchoscopy system, used successfully in live lung-cancer patient studies. During a live procedure, the method provides dynamic real-time sample size visualization in an enhanced virtual bronchoscopy viewer. In this way, the physician vividly sees the most promising biopsy sites along the airway walls as the bronchoscope moves through the airways.
Collapse
Affiliation(s)
- Xiaonan Zang
- School of Electrical Engineering and Computer Science, USA; EDDA Technologies, Princeton, NJ, 08540, USA
| | - Jason D Gibbs
- School of Electrical Engineering and Computer Science, USA; X-Nav Technologies, Lansdale, PA, 19446, USA
| | - Ronnarit Cheirsilp
- School of Electrical Engineering and Computer Science, USA; Broncus Medical, San Jose, CA, USA
| | | | - Jennifer Toth
- Department of Medicine, Division of Pulmonary, Allergy, and Critical Care Penn State University, University Park and Hershey, PA, USA
| | - Rebecca Bascom
- Department of Medicine, Division of Pulmonary, Allergy, and Critical Care Penn State University, University Park and Hershey, PA, USA
| | | |
Collapse
|
17
|
A Hybrid Method for Real-Time Bronchoscope Tracking Using Contour Registration and Synchronous EMT Data. IRANIAN JOURNAL OF RADIOLOGY 2019. [DOI: 10.5812/iranjradiol.66994] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
|
18
|
Ramírez E, Sánchez C, Borràs A, Diez-Ferrer M, Rosell A, Gil D. BronchoX: bronchoscopy exploration software for biopsy intervention planning. Healthc Technol Lett 2018; 5:177-182. [PMID: 30464850 PMCID: PMC6222182 DOI: 10.1049/htl.2018.5074] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2018] [Accepted: 08/20/2018] [Indexed: 11/19/2022] Open
Abstract
Virtual bronchoscopy (VB) is a non-invasive exploration tool for intervention planning and navigation of possible pulmonary lesions (PLs). A VB software involves the location of a PL and the calculation of a route, starting from the trachea, to reach it. The selection of a VB software might be a complex process, and there is no consensus in the community of medical software developers in which is the best-suited system to use or framework to choose. The authors present Bronchoscopy Exploration (BronchoX), a VB software to plan biopsy interventions that generate physician-readable instructions to reach the PLs. The authors' solution is open source, multiplatform, and extensible for future functionalities, designed by their multidisciplinary research and development group. BronchoX is a compound of different algorithms for segmentation, visualisation, and navigation of the respiratory tract. Performed results are a focus on the test the effectiveness of their proposal as an exploration software, also to measure its accuracy as a guiding system to reach PLs. Then, 40 different virtual planning paths were created to guide physicians until distal bronchioles. These results provide a functional software for BronchoX and demonstrate how following simple instructions is possible to reach distal lesions from the trachea.
Collapse
Affiliation(s)
- Esmitt Ramírez
- Computer Vision Center, Autonomous University of Barcelona, Bellaterra 08193, Spain
| | - Carles Sánchez
- Computer Vision Center, Autonomous University of Barcelona, Bellaterra 08193, Spain
| | - Agnés Borràs
- Computer Vision Center, Autonomous University of Barcelona, Bellaterra 08193, Spain
| | - Marta Diez-Ferrer
- Bellvitge University Hospital, L'Hospitalet de Llobregat, Barcelona 08907, Spain
| | - Antoni Rosell
- Bellvitge University Hospital, L'Hospitalet de Llobregat, Barcelona 08907, Spain
| | - Debora Gil
- Computer Vision Center, Autonomous University of Barcelona, Bellaterra 08193, Spain
| |
Collapse
|
19
|
Augmented Reality of the Middle Ear Combining Otoendoscopy and Temporal Bone Computed Tomography. Otol Neurotol 2018; 39:931-939. [DOI: 10.1097/mao.0000000000001922] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
20
|
Abstract
Bronchoscopy enables many minimally invasive chest procedures for diseases such as lung cancer and asthma. Guided by the bronchoscope's video stream, a physician can navigate the complex three-dimensional (3-D) airway tree to collect tissue samples or administer a disease treatment. Unfortunately, physicians currently discard procedural video because of the overwhelming amount of data generated. Hence, they must rely on memory and anecdotal snapshots to document a procedure. We propose a robust automatic method for summarizing an endobronchial video stream. Inspired by the multimedia concept of the video summary and by research in other endoscopy domains, our method consists of three main steps: 1) shot segmentation, 2) motion analysis, and 3) keyframe selection. Overall, the method derives a true hierarchical decomposition, consisting of a shot set and constituent keyframe set, for a given procedural video. No other method to our knowledge gives such a structured summary for the raw, unscripted, unedited videos arising in endoscopy. Results show that our method more efficiently covers the observed endobronchial regions than other keyframe-selection approaches and is robust to parameter variations. Over a wide range of video sequences, our method required on average only 6.5% of available video frames to achieve a video coverage = 92.7%. We also demonstrate how the derived video summary facilitates direct fusion with a patient's 3-D chest computed-tomography scan in a system under development, thereby enabling efficient video browsing and retrieval through the complex airway tree.
Collapse
|
21
|
Luo X, Mori K, Peters TM. Advanced Endoscopic Navigation: Surgical Big Data, Methodology, and Applications. Annu Rev Biomed Eng 2018; 20:221-251. [PMID: 29505729 DOI: 10.1146/annurev-bioeng-062117-120917] [Citation(s) in RCA: 33] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2022]
Abstract
Interventional endoscopy (e.g., bronchoscopy, colonoscopy, laparoscopy, cystoscopy) is a widely performed procedure that involves either diagnosis of suspicious lesions or guidance for minimally invasive surgery in a variety of organs within the body cavity. Endoscopy may also be used to guide the introduction of certain items (e.g., stents) into the body. Endoscopic navigation systems seek to integrate big data with multimodal information (e.g., computed tomography, magnetic resonance images, endoscopic video sequences, ultrasound images, external trackers) relative to the patient's anatomy, control the movement of medical endoscopes and surgical tools, and guide the surgeon's actions during endoscopic interventions. Nevertheless, it remains challenging to realize the next generation of context-aware navigated endoscopy. This review presents a broad survey of various aspects of endoscopic navigation, particularly with respect to the development of endoscopic navigation techniques. First, we investigate big data with multimodal information involved in endoscopic navigation. Next, we focus on numerous methodologies used for endoscopic navigation. We then review different endoscopic procedures in clinical applications. Finally, we discuss novel techniques and promising directions for the development of endoscopic navigation.
Collapse
Affiliation(s)
- Xiongbiao Luo
- Department of Computer Science, Fujian Key Laboratory of Computing and Sensing for Smart City, Xiamen University, Xiamen 361005, China;
| | - Kensaku Mori
- Department of Intelligent Systems, Graduate School of Informatics, Nagoya University, Nagoya 464-8601, Japan;
| | - Terry M Peters
- Robarts Research Institute, Western University, London, Ontario N6A 3K7, Canada;
| |
Collapse
|
22
|
Hofstad EF, Sorger H, Bakeng JBL, Gruionu L, Leira HO, Amundsen T, Langø T. Intraoperative localized constrained registration in navigated bronchoscopy. Med Phys 2017; 44:4204-4212. [DOI: 10.1002/mp.12361] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2016] [Revised: 05/17/2017] [Accepted: 05/17/2017] [Indexed: 11/11/2022] Open
Affiliation(s)
| | - Hanne Sorger
- Department of Thoracic Medicine St. Olav's Hospital Trondheim Norway
- Department of Circulation and Medical Imaging Faculty of Medicine Norwegian University of Science and Technology (NTNU) Trondheim Norway
| | | | - Lucian Gruionu
- Department of Automotive Transportation and Industrial Engineering Faculty of Mechanics University of Craiova Craiova Romania
| | - Håkon Olav Leira
- Department of Thoracic Medicine St. Olav's Hospital Trondheim Norway
- Department of Circulation and Medical Imaging Faculty of Medicine Norwegian University of Science and Technology (NTNU) Trondheim Norway
| | - Tore Amundsen
- Department of Thoracic Medicine St. Olav's Hospital Trondheim Norway
- Department of Circulation and Medical Imaging Faculty of Medicine Norwegian University of Science and Technology (NTNU) Trondheim Norway
| | - Thomas Langø
- Department of Medical Technology SINTEF Technology and Society Trondheim Norway
| |
Collapse
|
23
|
Visentini-Scarzanella M, Sugiura T, Kaneko T, Koto S. Deep monocular 3D reconstruction for assisted navigation in bronchoscopy. Int J Comput Assist Radiol Surg 2017; 12:1089-1099. [PMID: 28508345 DOI: 10.1007/s11548-017-1609-2] [Citation(s) in RCA: 34] [Impact Index Per Article: 4.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2017] [Accepted: 05/05/2017] [Indexed: 11/25/2022]
Abstract
PURPOSE In bronchoschopy, computer vision systems for navigation assistance are an attractive low-cost solution to guide the endoscopist to target peripheral lesions for biopsy and histological analysis. We propose a decoupled deep learning architecture that projects input frames onto the domain of CT renderings, thus allowing offline training from patient-specific CT data. METHODS A fully convolutional network architecture is implemented on GPU and tested on a phantom dataset involving 32 video sequences and [Formula: see text]60k frames with aligned ground truth and renderings, which is made available as the first public dataset for bronchoscopy navigation. RESULTS An average estimated depth accuracy of 1.5 mm was obtained, outperforming conventional direct depth estimation from input frames by 60%, and with a computational time of [Formula: see text]30 ms on modern GPUs. Qualitatively, the estimated depth and renderings closely resemble the ground truth. CONCLUSIONS The proposed method shows a novel architecture to perform real-time monocular depth estimation without losing patient specificity in bronchoscopy. Future work will include integration within SLAM systems and collection of in vivo datasets.
Collapse
Affiliation(s)
- Marco Visentini-Scarzanella
- Multimedia Laboratory, Toshiba Corporate Research and Development Center, 1, Komukai-Toshiba-cho, Kawasaki, 212-8582, Japan.
| | - Takamasa Sugiura
- Multimedia Laboratory, Toshiba Corporate Research and Development Center, 1, Komukai-Toshiba-cho, Kawasaki, 212-8582, Japan
| | - Toshimitsu Kaneko
- Multimedia Laboratory, Toshiba Corporate Research and Development Center, 1, Komukai-Toshiba-cho, Kawasaki, 212-8582, Japan
| | - Shinichiro Koto
- Multimedia Laboratory, Toshiba Corporate Research and Development Center, 1, Komukai-Toshiba-cho, Kawasaki, 212-8582, Japan
| |
Collapse
|
24
|
Pre-clinical validation of virtual bronchoscopy using 3D Slicer. Int J Comput Assist Radiol Surg 2016; 12:25-38. [PMID: 27325238 DOI: 10.1007/s11548-016-1447-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2016] [Accepted: 06/11/2016] [Indexed: 10/21/2022]
Abstract
PURPOSE Lung cancer still represents the leading cause of cancer-related death, and the long-term survival rate remains low. Computed tomography (CT) is currently the most common imaging modality for lung diseases recognition. The purpose of this work was to develop a simple and easily accessible virtual bronchoscopy system to be coupled with a customized electromagnetic (EM) tracking system for navigation in the lung and which requires as little user interaction as possible, while maintaining high usability. METHODS The proposed method has been implemented as an extension to the open-source platform, 3D Slicer. It creates a virtual reconstruction of the airways starting from CT images for virtual navigation. It provides tools for pre-procedural planning and virtual navigation, and it has been optimized for use in combination with a [Formula: see text] of freedom EM tracking sensor. Performance of the algorithm has been evaluated in ex vivo and in vivo testing. RESULTS During ex vivo testing, nine volunteer physicians tested the implemented algorithm to navigate three separate targets placed inside a breathing pig lung model. In general, the system proved easy to use and accurate in replicating the clinical setting and seemed to help choose the correct path without any previous experience or image analysis. Two separate animal studies confirmed technical feasibility and usability of the system. CONCLUSIONS This work describes an easily accessible virtual bronchoscopy system for navigation in the lung. The system provides the user with a complete set of tools that facilitate navigation towards user-selected regions of interest. Results from ex vivo and in vivo studies showed that the system opens the way for potential future work with virtual navigation for safe and reliable airway disease diagnosis.
Collapse
|
25
|
Wang J, Suenaga H, Yang L, Kobayashi E, Sakuma I. Video see-through augmented reality for oral and maxillofacial surgery. Int J Med Robot 2016; 13. [PMID: 27283505 DOI: 10.1002/rcs.1754] [Citation(s) in RCA: 50] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2016] [Revised: 03/26/2016] [Accepted: 04/29/2016] [Indexed: 11/11/2022]
Abstract
BACKGROUND Oral and maxillofacial surgery has not been benefitting from image guidance techniques owing to the limitations in image registration. METHODS A real-time markerless image registration method is proposed by integrating a shape matching method into a 2D tracking framework. The image registration is performed by matching the patient's teeth model with intraoperative video to obtain its pose. The resulting pose is used to overlay relevant models from the same CT space on the camera video for augmented reality. RESULTS The proposed system was evaluated on mandible/maxilla phantoms, a volunteer and clinical data. Experimental results show that the target overlay error is about 1 mm, and the frame rate of registration update yields 3-5 frames per second with a 4 K camera. CONCLUSIONS The significance of this work lies in its simplicity in clinical setting and the seamless integration into the current medical procedure with satisfactory response time and overlay accuracy. Copyright © 2016 John Wiley & Sons, Ltd.
Collapse
Affiliation(s)
- Junchen Wang
- School of Mechanical Engineering and Automation, Beihang University, Beijing, China.,Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
| | - Hideyuki Suenaga
- Department of Oral-Maxillofacial Surgery, Dentistry and Orthodontics, The University of Tokyo Hospital, Tokyo, Japan
| | - Liangjing Yang
- Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
| | - Etsuko Kobayashi
- Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
| | - Ichiro Sakuma
- Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
26
|
Markerless registration for image-guided endoscopic retrograde cholangiopancreatography (ERCP). ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2016; 2015:2932-5. [PMID: 26736906 DOI: 10.1109/embc.2015.7319006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
This paper proposes methods for markerless registration, which enable tracking pose of the endoscope camera in real time for implementation of the image-guided ERCP. Edge-based initialization is developed to determine the initial pose of the endoscope camera. Images of virtual endoscope are rendered from the virtual 3D organ model constructed from the patient's CT images. The similarity between edges on the image of the virtual and real endoscope is exploited for registration. An optical-flow-based tracking method is developed to track the changes starting from the initial pose of the endoscope camera in real time. The redefinition method is proposed to prevent the accumulation of the tracking error. Accuracy of the proposed methods is compared with the previous methods. The initialization method reduces 5.2 mm, 33.1 degrees, and 10.9 degrees of the position, direction, and roll angle error, on average, respectively. The tracking method reduces 3.5 degrees and 1.7 degrees of the hysteresis error in the direction angle and roll angle, respectively, with 15% faster update rate.
Collapse
|
27
|
Cheirsilp R, Bascom R, Allen TW, Higgins WE. Thoracic cavity definition for 3D PET/CT analysis and visualization. Comput Biol Med 2015; 62:222-38. [PMID: 25957746 PMCID: PMC4429311 DOI: 10.1016/j.compbiomed.2015.04.018] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2015] [Revised: 04/10/2015] [Accepted: 04/11/2015] [Indexed: 12/25/2022]
Abstract
X-ray computed tomography (CT) and positron emission tomography (PET) serve as the standard imaging modalities for lung-cancer management. CT gives anatomical details on diagnostic regions of interest (ROIs), while PET gives highly specific functional information. During the lung-cancer management process, a patient receives a co-registered whole-body PET/CT scan pair and a dedicated high-resolution chest CT scan. With these data, multimodal PET/CT ROI information can be gleaned to facilitate disease management. Effective image segmentation of the thoracic cavity, however, is needed to focus attention on the central chest. We present an automatic method for thoracic cavity segmentation from 3D CT scans. We then demonstrate how the method facilitates 3D ROI localization and visualization in patient multimodal imaging studies. Our segmentation method draws upon digital topological and morphological operations, active-contour analysis, and key organ landmarks. Using a large patient database, the method showed high agreement to ground-truth regions, with a mean coverage=99.2% and leakage=0.52%. Furthermore, it enabled extremely fast computation. For PET/CT lesion analysis, the segmentation method reduced ROI search space by 97.7% for a whole-body scan, or nearly 3 times greater than that achieved by a lung mask. Despite this reduction, we achieved 100% true-positive ROI detection, while also reducing the false-positive (FP) detection rate by >5 times over that achieved with a lung mask. Finally, the method greatly improved PET/CT visualization by eliminating false PET-avid obscurations arising from the heart, bones, and liver. In particular, PET MIP views and fused PET/CT renderings depicted unprecedented clarity of the lesions and neighboring anatomical structures truly relevant to lung-cancer assessment.
Collapse
Affiliation(s)
- Ronnarit Cheirsilp
- School of Electrical Engineering and Computer Science, Penn State University, University Park, PA, United States
| | - Rebecca Bascom
- Department of Medicine, Division of Pulmonary, Allergy, and Critical Care, Penn State University, Milton S. Hershey Medical Center, Hershey, PA, United States
| | - Thomas W Allen
- Department of Radiology, Penn State University, Milton S. Hershey Medical Center, Hershey, PA, United States
| | - William E Higgins
- School of Electrical Engineering and Computer Science, Penn State University, University Park, PA, United States.
| |
Collapse
|
28
|
Abstract
Bronchoscopy is a commonly used minimally invasive procedure for lung-cancer staging. In standard practice, however, physicians differ greatly in their levels of performance. To address this concern, image-guided intervention (IGI) systems have been devised to improve procedure success. Current IGI bronchoscopy systems based on virtual bronchoscopic navigation (VBN), however, require involvement from the attending technician. This lessens physician control and hinders the overall acceptance of such systems. We propose a hands-free VBN system for planning and guiding bronchoscopy. The system introduces two major contributions. First, it incorporates a new procedure-planning method that automatically computes airway navigation plans conforming to the physician's bronchoscopy training and manual dexterity. Second, it incorporates a guidance strategy for bronchoscope navigation that enables user-friendly system control via a foot switch, coupled with a novel position-verification mechanism. Phantom studies verified that the system enables smooth operation under physician control, while also enabling faster navigation than an existing technician-assisted VBN system. In a clinical human study, we noted a 97% bronchoscopy navigation success rate, in line with existing VBN systems, and a mean guidance time per diagnostic site = 52 s. This represents a guidance time often nearly 3 min faster per diagnostic site than guidance times reported for other technician-assisted VBN systems. Finally, an ergonomic study further asserts the system's acceptability to the physician and long-term potential.
Collapse
|
29
|
Luo X, Wan Y, He X, Mori K. Adaptive marker-free registration using a multiple point strategy for real-time and robust endoscope electromagnetic navigation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2015; 118:147-157. [PMID: 25547498 DOI: 10.1016/j.cmpb.2014.11.008] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/29/2014] [Revised: 10/04/2014] [Accepted: 11/26/2014] [Indexed: 06/04/2023]
Abstract
Registration of pre-clinical images to physical space is indispensable for computer-assisted endoscopic interventions in operating rooms. Electromagnetically navigated endoscopic interventions are increasingly performed at current diagnoses and treatments. Such interventions use an electromagnetic tracker with a miniature sensor that is usually attached at an endoscope distal tip to real time track endoscope movements in a pre-clinical image space. Spatial alignment between the electromagnetic tracker (or sensor) and pre-clinical images must be performed to navigate the endoscope to target regions. This paper proposes an adaptive marker-free registration method that uses a multiple point selection strategy. This method seeks to address an assumption that the endoscope is operated along the centerline of an intraluminal organ which is easily violated during interventions. We introduce an adaptive strategy that generates multiple points in terms of sensor measurements and endoscope tip center calibration. From these generated points, we adaptively choose the optimal point, which is the closest to its assigned the centerline of the hollow organ, to perform registration. The experimental results demonstrate that our proposed adaptive strategy significantly reduced the target registration error from 5.32 to 2.59 mm in static phantoms validation, as well as from at least 7.58 mm to 4.71 mm in dynamic phantom validation compared to current available methods.
Collapse
Affiliation(s)
- Xiongbiao Luo
- Information and Communications Headquarters, Nagoya University, Japan.
| | - Ying Wan
- School of Computing and Communications, University of Technology, Sydney, Australia.
| | - Xiangjian He
- School of Computing and Communications, University of Technology, Sydney, Australia.
| | - Kensaku Mori
- Information and Communications Headquarters, Nagoya University, Japan.
| |
Collapse
|
30
|
Multi-view stereo and advanced navigation for transanal endoscopic microsurgery. ACTA ACUST UNITED AC 2015. [PMID: 25485396 DOI: 10.1007/978-3-319-10470-6_42] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register]
Abstract
Transanal endoscopic microsurgery (TEM), i.e., the local excision of rectal carcinomas by way of a bimanual operating system with magnified binocular vision, is gaining acceptance in lieu of more radical total interventions. A major issue with this approach is the lack of information on submucosal anatomical structures. This paper presents an advanced navigation system, wherein the intraoperative 3D structure is stably estimated from multiple stereoscopic views. It is registered to a preoperatively acquired anatomical volume based on subject-specific priors. The endoscope motion is tracked based on the 3D scene and its field-of-view is visualised jointly with the preoperative information. Based on in vivo data, this paper demonstrates how the proposed navigation system provides intraoperative navigation for TEM1.
Collapse
|
31
|
Real-time bronchoscope three-dimensional motion estimation using multiple sensor-driven alignment of CT images and electromagnetic measurements. Comput Med Imaging Graph 2014; 38:540-8. [PMID: 25002104 DOI: 10.1016/j.compmedimag.2014.06.013] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2014] [Revised: 04/22/2014] [Accepted: 06/13/2014] [Indexed: 11/21/2022]
Abstract
Bronchoscope three-dimensional motion estimation plays a key role in developing bronchoscopic navigation systems. Currently external tracking devices, particularly electromagnetic trackers with electromagnetic sensors, are increasingly introduced to navigate surgical tools in pre-clinical images. An unavoidable problem, which is to align the electromagnetic tracker to pre-clinical images, must be solved before navigation. This paper proposes a multiple sensor-driven registration method to establish this alignment without using any anatomical fiducials. Although current fiducially free registration methods work well, they limit to the initialization of optimization and manipulating the bronchoscope along the bronchial centerlines, which could be failed easily during clinical interventions. To address these limitations, we utilize measurements of multiple electromagnetic sensors to calculate bronchoscope geometric center positions that are usually closer to the bronchial centerlines than the sensor itself measured positions. We validated our method on a bronchial phantom. The experimental results demonstrate that our idea of using multiple sensors to determine bronchoscope geometric center positions for fiducial-free registration was very effective. Compared to currently available methods in bronchoscope three-dimensional motion estimation, our method reduced fiducial alignment error from at least 6.79 to 4.68-5.26 mm and significantly improved motion estimation or tracking accuracy from at least 5.42 to 3.78-4.53 mm.
Collapse
|
32
|
Luo X, Mori K. A discriminative structural similarity measure and its application to video-volume registration for endoscope three-dimensional motion tracking. IEEE TRANSACTIONS ON MEDICAL IMAGING 2014; 33:1248-1261. [PMID: 24893255 DOI: 10.1109/tmi.2014.2307052] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
Endoscope 3-D motion tracking, which seeks to synchronize pre- and intra-operative images in endoscopic interventions, is usually performed as video-volume registration that optimizes the similarity between endoscopic video and pre-operative images. The tracking performance, in turn, depends significantly on whether a similarity measure can successfully characterize the difference between video sequences and volume rendering images driven by pre-operative images. The paper proposes a discriminative structural similarity measure, which uses the degradation of structural information and takes image correlation or structure, luminance, and contrast into consideration, to boost video-volume registration. By applying the proposed similarity measure to endoscope tracking, it was demonstrated to be more accurate and robust than several available similarity measures, e.g., local normalized cross correlation, normalized mutual information, modified mean square error, or normalized sum squared difference. Based on clinical data evaluation, the tracking error was reduced significantly from at least 14.6 mm to 4.5 mm. The processing time was accelerated more than 30 frames per second using graphics processing unit.
Collapse
|
33
|
Luo X. A bronchoscopic navigation system using bronchoscope center calibration for accurate registration of electromagnetic tracker and CT volume without markers. Med Phys 2014; 41:061913. [DOI: 10.1118/1.4876381] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022] Open
|
34
|
Hofstad EF, Sorger H, Leira HO, Amundsen T, Langø T. Automatic registration of CT images to patient during the initial phase of bronchoscopy: A clinical pilot study. Med Phys 2014; 41:041903. [DOI: 10.1118/1.4866884] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022] Open
|
35
|
Gibbs JD, Graham MW, Bascom R, Cornish DC, Khare R, Higgins WE. Optimal procedure planning and guidance system for peripheral bronchoscopy. IEEE Trans Biomed Eng 2013; 61:638-57. [PMID: 24235246 DOI: 10.1109/tbme.2013.2285627] [Citation(s) in RCA: 18] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2022]
Abstract
With the development of multidetector computed-tomography (MDCT) scanners and ultrathin bronchoscopes, the use of bronchoscopy for diagnosing peripheral lung-cancer nodules is becoming a viable option. The work flow for assessing lung cancer consists of two phases: 1) 3-D MDCT analysis and 2) live bronchoscopy. Unfortunately, the yield rates for peripheral bronchoscopy have been reported to be as low as 14%, and bronchoscopy performance varies considerably between physicians. Recently, proposed image-guided systems have shown promise for assisting with peripheral bronchoscopy. Yet, MDCT-based route planning to target sites has relied on tedious error-prone techniques. In addition, route planning tends not to incorporate known anatomical, device, and procedural constraints that impact a feasible route. Finally, existing systems do not effectively integrate MDCT-derived route information into the live guidance process. We propose a system that incorporates an automatic optimal route-planning method, which integrates known route constraints. Furthermore, our system offers a natural translation of the MDCT-based route plan into the live guidance strategy via MDCT/video data fusion. An image-based study demonstrates the route-planning method's functionality. Next, we present a prospective lung-cancer patient study in which our system achieved a successful navigation rate of 91% to target sites. Furthermore, when compared to a competing commercial system, our system enabled bronchoscopy over two airways deeper into the airway-tree periphery with a sample time that was nearly 2 min shorter on average. Finally, our system's ability to almost perfectly predict the depth of a bronchoscope's navigable route in advance represents a substantial benefit of optimal route planning.
Collapse
|