Stomach 3D Reconstruction Using Virtual Chromoendoscopic Images

3D reconstruction from endoscopy images: A survey

Three-dimensional reconstruction of images acquired through endoscopes is playing a vital role in an increasing number of medical applications. Endoscopes used in the clinic are commonly classified as monocular endoscopes and binocular endoscopes. We have reviewed the classification of methods for depth estimation according to the type of endoscope. Basically, depth estimation relies on feature matching of images and multi-view geometry theory. However, these traditional techniques have many problems in the endoscopic environment. With the increasing development of deep learning techniques, there is a growing number of works based on learning methods to address challenges such as inconsistent illumination and texture sparsity. We have reviewed over 170 papers published in the 10 years from 2013 to 2023. The commonly used public datasets and performance metrics are summarized. We also give a taxonomy of methods and analyze the advantages and drawbacks of algorithms. Summary tables and result atlas are listed to facilitate the comparison of qualitative and quantitative performance of different methods in each category. In addition, we summarize commonly used scene representation methods in endoscopy and speculate on the prospects of deep estimation research in medical applications. We also compare the robustness performance, processing time, and scene representation of the methods to facilitate doctors and researchers in selecting appropriate methods based on surgical applications.

Autonomous Magnetic Navigation in Endoscopic Image Mosaics

Endoscopes navigate within the human body to observe anatomical structures with minimal invasiveness. A major shortcoming of their use is their narrow field-of-view during navigation in large, hollow anatomical regions. Mosaics of endoscopic images can provide surgeons with a map of the tool's environment. This would facilitate procedures, improve their efficiency, and potentially generate better patient outcomes. The emergence of magnetically steered endoscopes opens the way to safer procedures and creates an opportunity to provide robotic assistance both in the generation of the mosaic map and in navigation within this map. This paper proposes methods to autonomously navigate magnetic endoscopes to 1) generate endoscopic image mosaics and 2) use these mosaics as user interfaces to navigate throughout the explored area. These are the first strategies, which allow autonomous magnetic navigation in large, hollow organs during minimally invasive surgeries. The feasibility of these methods is demonstrated experimentally both in vitro and ex vivo in the context of the treatment of twin-to-twin transfusion syndrome. This minimally invasive procedure is performed in utero and necessitates coagulating shared vessels of twin fetuses on the placenta. A mosaic of the vasculature in combination with autonomous navigation has the potential to significantly facilitate this challenging surgery.

Diagnostic performance of deep-learning-based virtual chromoendoscopy in gastric neoplasms

BACKGROUNDS

Cycle-consistent generative adversarial network (CycleGAN) is a deep neural network model that performs image-to-image translations. We generated virtual indigo carmine (IC) chromoendoscopy images of gastric neoplasms using CycleGAN and compared their diagnostic performance with that of white light endoscopy (WLE).

METHODS

WLE and IC images of 176 patients with gastric neoplasms who underwent endoscopic resection were obtained. We used 1,633 images (911 WLE and 722 IC) of 146 cases in the training dataset to develop virtual IC images using CycleGAN. The remaining 30 WLE images were translated into 30 virtual IC images using the trained CycleGAN and used for validation. The lesion borders were evaluated by 118 endoscopists from 22 institutions using the 60 paired virtual IC and WLE images. The lesion area concordance rate and successful whole-lesion diagnosis were compared.

RESULTS

The lesion area concordance rate based on the pathological diagnosis in virtual IC was lower than in WLE (44.1% vs. 48.5%, p < 0.01). The successful whole-lesion diagnosis was higher in the virtual IC than in WLE images; however, the difference was insignificant (28.2% vs. 26.4%, p = 0.11). Conversely, subgroup analyses revealed a significantly higher diagnosis in virtual IC than in WLE for depressed morphology (41.9% vs. 36.9%, p = 0.02), differentiated histology (27.6% vs. 24.8%, p = 0.02), smaller lesion size (42.3% vs. 38.3%, p = 0.01), and assessed by expert endoscopists (27.3% vs. 23.6%, p = 0.03).

CONCLUSIONS

The diagnostic ability of virtual IC was higher for some lesions, but not completely superior to that of WLE. Adjustments are required to improve the imaging system's performance.

Towards navigation in endoscopic kidney surgery based on preoperative imaging

Endoscopic renal surgeries have high re-operation rates, particularly for lower volume surgeons. Due to the limited field and depth of view of current endoscopes, mentally mapping preoperative computed tomography (CT) images of patient anatomy to the surgical field is challenging. The inability to completely navigate the intrarenal collecting system leads to missed kidney stones and tumors, subsequently raising recurrence rates. A guidance system is proposed to estimate the endoscope positions within the CT to reduce re-operation rates. A Structure from Motion algorithm is used to reconstruct the kidney collecting system from the endoscope videos. In addition, the kidney collecting system is segmented from CT scans using 3D U-Net to create a 3D model. The two collecting system representations can then be registered to provide information on the relative endoscope position. Correct reconstruction and localization of intrarenal anatomy and endoscope position is demonstrated. Furthermore, a 3D map is created supported by the RGB endoscope images to reduce the burden of mental mapping during surgery. The proposed reconstruction pipeline has been validated for guidance. It can reduce the mental burden for surgeons and is a step towards the long-term goal of reducing re-operation rates in kidney stone surgery.

Virtual indigo carmine dyeing: New artificial intelligence-based chromoendoscopy technique

Watch a video of this article.

The diagnostic ability to classify neoplasias occurring in inflammatory bowel disease by artificial intelligence and endoscopists: A pilot study

BACKGROUND AND AIM

Although endoscopic resection with careful surveillance instead of total proctocolectomy become to be permitted for visible low-grade dysplasia, it is unclear how accurately endoscopists can differentiate these lesions, as classifying neoplasias occurring in inflammatory bowel disease (IBDN) is exceedingly challenging due to background chronic inflammation. We evaluated a pilot model of an artificial intelligence (AI) system for classifying IBDN and compared it with the endoscopist's ability.

METHODS

This study used a deep convolutional neural network, the EfficientNet-B3. Among patients who underwent treatment for IBDN at two hospitals between 2003 and 2021, we selected 862 non-magnified endoscopic images from 99 IBDN lesions and utilized 6 375 352 images that were increased by data augmentation for the development of AI. We evaluated the diagnostic ability of AI using two classifications: the "adenocarcinoma/high-grade dysplasia" and "low-grade dysplasia/sporadic adenoma/normal mucosa" groups. We compared the diagnostic accuracy between AI and endoscopists (three non-experts and four experts) using 186 test set images.

RESULTS

The diagnostic ability of the experts/non-experts/AI for the two classifications in the test set images had a sensitivity of 60.5% (95% confidence interval [CI]: 54.5-66.3)/70.5% (95% CI: 63.8-76.6)/72.5% (95% CI: 60.4-82.5), specificity of 88.0% (95% CI: 84.7-90.8)/78.8% (95% CI: 74.3-83.1)/82.9% (95% CI: 74.8-89.2), and accuracy of 77.8% (95% CI: 74.7-80.8)/75.8% (95% CI: 72-79.3)/79.0% (95% CI: 72.5-84.6), respectively.

CONCLUSIONS

The diagnostic accuracy of the two classifications of IBDN was higher than that of the experts. Our AI system is valuable enough to contribute to the next generation of clinical practice.

Learning-Based Depth and Pose Estimation for Monocular Endoscope with Loss Generalization

Gastroendoscopy has been a clinical standard for diagnosing and treating conditions that affect a part of a patient's digestive system, such as the stomach. Despite the fact that gastroendoscopy has a lot of advantages for patients, there exist some challenges for practitioners, such as the lack of 3D perception, including the depth and the endoscope pose information. Such challenges make navigating the endoscope and localizing any found lesion in a digestive tract difficult. To tackle these problems, deep learning-based approaches have been proposed to provide monocular gastroendoscopy with additional yet important depth and pose information. In this paper, we propose a novel supervised approach to train depth and pose estimation networks using consecutive endoscopy images to assist the endoscope navigation in the stomach. We firstly generate real depth and pose training data using our previously proposed whole stomach 3D reconstruction pipeline to avoid poor generalization ability between computer-generated (CG) models and real data for the stomach. In addition, we propose a novel generalized photometric loss function to avoid the complicated process of finding proper weights for balancing the depth and the pose loss terms, which is required for existing direct depth and pose supervision approaches. We then experimentally show that our proposed generalized loss performs better than existing direct supervision losses.