1
|
Bobrow TL, Golhar M, Vijayan R, Akshintala VS, Garcia JR, Durr NJ. Colonoscopy 3D video dataset with paired depth from 2D-3D registration. Med Image Anal 2023; 90:102956. [PMID: 37713764 PMCID: PMC10591895 DOI: 10.1016/j.media.2023.102956] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2022] [Revised: 06/29/2023] [Accepted: 09/04/2023] [Indexed: 09/17/2023]
Abstract
Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a Colonoscopy 3D Video Dataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at https://durr.jhu.edu/C3VD.
Collapse
Affiliation(s)
- Taylor L Bobrow
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Mayank Golhar
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Rohan Vijayan
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Venkata S Akshintala
- Division of Gastroenterology and Hepatology, Johns Hopkins Medicine, Baltimore, MD 21287, USA
| | - Juan R Garcia
- Department of Art as Applied to Medicine, Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Nicholas J Durr
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA.
| |
Collapse
|
2
|
Ozyoruk KB, Gokceler GI, Bobrow TL, Coskun G, Incetan K, Almalioglu Y, Mahmood F, Curto E, Perdigoto L, Oliveira M, Sahin H, Araujo H, Alexandrino H, Durr NJ, Gilbert HB, Turan M. EndoSLAM dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videos. Med Image Anal 2021; 71:102058. [PMID: 33930829 DOI: 10.1016/j.media.2021.102058] [Citation(s) in RCA: 35] [Impact Index Per Article: 11.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2020] [Revised: 01/23/2021] [Accepted: 03/29/2021] [Indexed: 02/07/2023]
Abstract
Deep learning techniques hold promise to develop dense topography reconstruction and pose estimation methods for endoscopic videos. However, currently available datasets do not support effective quantitative benchmarking. In this paper, we introduce a comprehensive endoscopic SLAM dataset consisting of 3D point cloud data for six porcine organs, capsule and standard endoscopy recordings, synthetically generated data as well as clinically in use conventional endoscope recording of the phantom colon with computed tomography(CT) scan ground truth. A Panda robotic arm, two commercially available capsule endoscopes, three conventional endoscopes with different camera properties, two high precision 3D scanners, and a CT scanner were employed to collect data from eight ex-vivo porcine gastrointestinal (GI)-tract organs and a silicone colon phantom model. In total, 35 sub-datasets are provided with 6D pose ground truth for the ex-vivo part: 18 sub-datasets for colon, 12 sub-datasets for stomach, and 5 sub-datasets for small intestine, while four of these contain polyp-mimicking elevations carried out by an expert gastroenterologist. To verify the applicability of this data for use with real clinical systems, we recorded a video sequence with a state-of-the-art colonoscope from a full representation silicon colon phantom. Synthetic capsule endoscopy frames from stomach, colon, and small intestine with both depth and pose annotations are included to facilitate the study of simulation-to-real transfer learning algorithms. Additionally, we propound Endo-SfMLearner, an unsupervised monocular depth and pose estimation method that combines residual networks with a spatial attention module in order to dictate the network to focus on distinguishable and highly textured tissue regions. The proposed approach makes use of a brightness-aware photometric loss to improve the robustness under fast frame-to-frame illumination changes that are commonly seen in endoscopic videos. To exemplify the use-case of the EndoSLAM dataset, the performance of Endo-SfMLearner is extensively compared with the state-of-the-art: SC-SfMLearner, Monodepth2, and SfMLearner. The codes and the link for the dataset are publicly available at https://github.com/CapsuleEndoscope/EndoSLAM. A video demonstrating the experimental setup and procedure is accessible as Supplementary Video 1.
Collapse
Affiliation(s)
| | | | - Taylor L Bobrow
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
| | - Gulfize Coskun
- Institute of Biomedical Engineering, Bogazici University, Turkey
| | - Kagan Incetan
- Institute of Biomedical Engineering, Bogazici University, Turkey
| | | | - Faisal Mahmood
- Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Cancer Data Science, Dana Farber Cancer Institute, Boston, MA, USA; Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
| | - Eva Curto
- Institute for Systems and Robotics, University of Coimbra, Portugal
| | - Luis Perdigoto
- Institute for Systems and Robotics, University of Coimbra, Portugal
| | - Marina Oliveira
- Institute for Systems and Robotics, University of Coimbra, Portugal
| | - Hasan Sahin
- Institute of Biomedical Engineering, Bogazici University, Turkey
| | - Helder Araujo
- Institute for Systems and Robotics, University of Coimbra, Portugal
| | - Henrique Alexandrino
- Faculty of Medicine, Clinical Academic Center of Coimbra, University of Coimbra, Coimbra, Portugal
| | - Nicholas J Durr
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
| | - Hunter B Gilbert
- Department of Mechanical and Industrial Engineering, Louisiana State University, Baton Rouge, LA, USA
| | - Mehmet Turan
- Institute of Biomedical Engineering, Bogazici University, Turkey.
| |
Collapse
|
3
|
Golhar M, Bobrow TL, Khoshknab MP, Jit S, Ngamruengphong S, Durr NJ. Improving Colonoscopy Lesion Classification Using Semi-Supervised Deep Learning. IEEE Access 2021; 9:631-640. [PMID: 33747680 PMCID: PMC7978231 DOI: 10.1109/access.2020.3047544] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/07/2023]
Abstract
While data-driven approaches excel at many image analysis tasks, the performance of these approaches is often limited by a shortage of annotated data available for training. Recent work in semi-supervised learning has shown that meaningful representations of images can be obtained from training with large quantities of unlabeled data, and that these representations can improve the performance of supervised tasks. Here, we demonstrate that an unsupervised jigsaw learning task, in combination with supervised training, results in up to a 9.8% improvement in correctly classifying lesions in colonoscopy images when compared to a fully-supervised baseline. We additionally benchmark improvements in domain adaptation and out-of-distribution detection, and demonstrate that semi-supervised learning outperforms supervised learning in both cases. In colonoscopy applications, these metrics are important given the skill required for endoscopic assessment of lesions, the wide variety of endoscopy systems in use, and the homogeneity that is typical of labeled datasets.
Collapse
Affiliation(s)
- Mayank Golhar
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Taylor L Bobrow
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | | | - Simran Jit
- Division of Gastroenterology and Hepatology, Johns Hopkins Hospital, Baltimore, MD 21287, USA
| | - Saowanee Ngamruengphong
- Division of Gastroenterology and Hepatology, Johns Hopkins Hospital, Baltimore, MD 21287, USA
| | - Nicholas J Durr
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| |
Collapse
|
4
|
Gowda PC, Chen VX, Sobral MC, Bobrow TL, Romer TG, Palepu AK, Guo JY, Kim DJ, Tsai AS, Chen S, Weiss CR, Durr NJ. Establishing a Quantitative Endpoint for Transarterial Embolization From Real-Time Pressure Measurements. J Med Device 2020. [DOI: 10.1115/1.4049056] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022] Open
Abstract
Abstract
Transarterial embolization (TAE) is a standard-of-care treatment for tumors in which embolic particles are locally injected via a catheter to occlude blood flow and induce ischemia in the target tissue. Physicians currently rely on subjective visual cues from fluoroscopy in order to determine the procedural endpoint relative to the injection site. This contributes to highly variable treatment outcomes, including the accumulation of embolic particles in healthy tissue, called off-target embolization. To address this concern, we describe a novel, multilumen catheter that 1) measures real-time pressure upstream of the tumor site during TAE injection; and 2) associates that measurement with the volume of embolic particles injected. Using an in vitro silicon vascular model, we characterize the relationship between blood flow, intravascular pressure, and injection pressure. Furthermore, we identify a predictive pressure curve based on the volume of embolic particles injected. This approach has the potential to standardize and optimize TAE, reducing the likelihood of incomplete or off-target embolization, and improving patient outcomes.
Collapse
Affiliation(s)
- Prateek C. Gowda
- Department of Biomedical Engineering, Johns Hopkins University, 1800 Orleans Street, Sheikh Zayed Tower Ste 7203, Baltimore, MD 21218
| | - Victoria X. Chen
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N Charles Street, Baltimore, MD 21218
| | - Miguel C. Sobral
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N Charles Street, Baltimore, MD 21218
| | - Taylor L. Bobrow
- Department of Biomedical Engineering, Johns Hopkins University, 733 N Broadway, Traylor 606, Baltimore, MD 21218
| | - Tatiana Gelaf Romer
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N Charles Street, Baltimore, MD 21218
| | - Anil K. Palepu
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N Charles Street, Baltimore, MD 21218
| | - Joanna Y. Guo
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N Charles Street, Baltimore, MD 21218
| | - Dohyung J. Kim
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N Charles Street, Baltimore, MD 21218
| | - Andrew S. Tsai
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N Charles Street, Baltimore, MD 21218
| | - Steven Chen
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N Charles Street, Baltimore, MD 21218
| | - Clifford R. Weiss
- Department of Radiology and Radiologic Science, The Johns Hopkins University School of Medicine, 1800 Orleans Street, Sheikh Zayed Tower Ste 7203, Baltimore, MD 21287
| | - Nicholas J. Durr
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N Charles Street, Baltimore, MD 21218
| |
Collapse
|
5
|
Bobrow TL, Mahmood F, Inserni M, Durr NJ. DeepLSR: a deep learning approach for laser speckle reduction. Biomed Opt Express 2019; 10:2869-2882. [PMID: 31259057 PMCID: PMC6583356 DOI: 10.1364/boe.10.002869] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/19/2019] [Revised: 05/08/2019] [Accepted: 05/08/2019] [Indexed: 05/06/2023]
Abstract
Speckle artifacts degrade image quality in virtually all modalities that utilize coherent energy, including optical coherence tomography, reflectance confocal microscopy, ultrasound, and widefield imaging with laser illumination. We present an adversarial deep learning framework for laser speckle reduction, called DeepLSR (https://durr.jhu.edu/DeepLSR), that transforms images from a source domain of coherent illumination to a target domain of speckle-free, incoherent illumination. We apply this method to widefield images of objects and tissues illuminated with a multi-wavelength laser, using light emitting diode-illuminated images as ground truth. In images of gastrointestinal tissues, DeepLSR reduces laser speckle noise by 6.4 dB, compared to a 2.9 dB reduction from optimized non-local means processing, a 3.0 dB reduction from BM3D, and a 3.7 dB reduction from an optical speckle reducer utilizing an oscillating diffuser. Further, DeepLSR can be combined with optical speckle reduction to reduce speckle noise by 9.4 dB. This dramatic reduction in speckle noise may enable the use of coherent light sources in applications that require small illumination sources and high-quality imaging, including medical endoscopy.
Collapse
|
6
|
Bobrow TL, Durr NJ. An adaptive-coherence light source for hyperspectral, topographic, and flow-contrast imaging. Proc SPIE Int Soc Opt Eng 2019; 10871:108710Y. [PMID: 34168392 PMCID: PMC8221261 DOI: 10.1117/12.2510632] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
Colorectal cancer accounts for an estimated 8% of cancer deaths in the United States with a five-year survival rate of 55-75%. The early detection and removal of precancerous lesions is critical for reducing mortality, but subtle neoplastic growths, such as non-polypoid lesions, often go undetected during routine colonoscopy. Current approaches to flat or depressed lesion detection are ineffective due to the poor contrast of subtle features in white light endoscopy. Towards improving colorectal lesion contrast, we present an endoscopic light source with custom laser channels for multimodal color, topographic, and speckle contrast flow imaging. Three red-green-blue laser units, paired with laser speckle reducers, are coupled into endoscopic fiber optic light guides in a benchtop endoscope. Tissue phantom topography is reconstructed using alternating illumination of the laser units and a photometric stereo endoscopy algorithm. The contrast of flow regions is enhanced in an optical flow phantom using laser speckle contrast imaging. Further, the system retains the ability to offer white light and narrow band illumination modes with improved power efficiency, a reduced size, and longer lifetimes compared to conventional endoscopic arc lamp sources. This novel endoscopic light source design shows promise for increasing the detection of subtle lesions in routine colonoscopy screening.
Collapse
Affiliation(s)
- Taylor L Bobrow
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N. Charles St., Baltimore, MD 21218. USA
| | - Nicholas J Durr
- Department of Biomedical Engineering, Johns Hopkins University, 3400 N. Charles St., Baltimore, MD 21218. USA
| |
Collapse
|
7
|
Abstract
Inattentional blindness is a failure to notice an unexpected event when attention is directed elsewhere. The current study examined participants’ awareness of an unexpected object that maintained luminance contrast, switched the luminance once, or repetitively flashed. One hundred twenty participants performed a dynamic tracking task on a computer monitor for which they were instructed to count the number of movement deflections of an attended set of objects while ignoring other objects. On the critical trial, an unexpected cross that did not change its luminance (control condition), switched its luminance once (switch condition), or repetitively flashed (flash condition) traveled across the stimulus display. Participants noticed the unexpected cross more frequently when the luminance feature matched their attention set than when it did not match. Unexpectedly, however, a proportion of the participants who noticed the cross in the switch and flash conditions were statistically comparable. The results suggest that an unexpected object with even a single luminance change can break inattentional blindness in a multi-object tracking task.
Collapse
Affiliation(s)
| | - Yusuke Yamani
- Department of Psychology, Old Dominion University, VA, USA
| | - Taylor L Bobrow
- Department of Electrical & Computer Engineering, Old Dominion University, VA, USA
| | | | - Dean J Krusienski
- Department of Electrical & Computer Engineering, Old Dominion University, VA, USA
| |
Collapse
|