1
|
Teufel T, Shu H, Soberanis-Mukul RD, Mangulabnan JE, Sahu M, Vedula SS, Ishii M, Hager G, Taylor RH, Unberath M. OneSLAM to map them all: a generalized approach to SLAM for monocular endoscopic imaging based on tracking any point. Int J Comput Assist Radiol Surg 2024; 19:1259-1266. [PMID: 38775904 DOI: 10.1007/s11548-024-03171-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Accepted: 04/30/2024] [Indexed: 07/10/2024]
Abstract
PURPOSE Monocular SLAM algorithms are the key enabling technology for image-based surgical navigation systems for endoscopic procedures. Due to the visual feature scarcity and unique lighting conditions encountered in endoscopy, classical SLAM approaches perform inconsistently. Many of the recent approaches to endoscopic SLAM rely on deep learning models. They show promising results when optimized on singular domains such as arthroscopy, sinus endoscopy, colonoscopy or laparoscopy, but are limited by an inability to generalize to different domains without retraining. METHODS To address this generality issue, we propose OneSLAM a monocular SLAM algorithm for surgical endoscopy that works out of the box for several endoscopic domains, including sinus endoscopy, colonoscopy, arthroscopy and laparoscopy. Our pipeline builds upon robust tracking any point (TAP) foundation models to reliably track sparse correspondences across multiple frames and runs local bundle adjustment to jointly optimize camera poses and a sparse 3D reconstruction of the anatomy. RESULTS We compare the performance of our method against three strong baselines previously proposed for monocular SLAM in endoscopy and general scenes. OneSLAM presents better or comparable performance over existing approaches targeted to that specific data in all four tested domains, generalizing across domains without the need for retraining. CONCLUSION OneSLAM benefits from the convincing performance of TAP foundation models but generalizes to endoscopic sequences of different anatomies all while demonstrating better or comparable performance over domain-specific SLAM approaches. Future research on global loop closure will investigate how to reliably detect loops in endoscopic scenes to reduce accumulated drift and enhance long-term navigation capabilities.
Collapse
Affiliation(s)
- Timo Teufel
- Johns Hopkins University, Baltimore, MD, 21211, USA.
| | - Hongchao Shu
- Johns Hopkins University, Baltimore, MD, 21211, USA
| | | | | | - Manish Sahu
- Johns Hopkins University, Baltimore, MD, 21211, USA
| | | | - Masaru Ishii
- Johns Hopkins Medical Institutions, Baltimore, MD, 21287, USA
| | | | - Russell H Taylor
- Johns Hopkins University, Baltimore, MD, 21211, USA
- Johns Hopkins Medical Institutions, Baltimore, MD, 21287, USA
| | - Mathias Unberath
- Johns Hopkins University, Baltimore, MD, 21211, USA
- Johns Hopkins Medical Institutions, Baltimore, MD, 21287, USA
| |
Collapse
|
2
|
Yang Z, Dai J, Pan J. 3D reconstruction from endoscopy images: A survey. Comput Biol Med 2024; 175:108546. [PMID: 38704902 DOI: 10.1016/j.compbiomed.2024.108546] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Revised: 01/05/2024] [Accepted: 04/28/2024] [Indexed: 05/07/2024]
Abstract
Three-dimensional reconstruction of images acquired through endoscopes is playing a vital role in an increasing number of medical applications. Endoscopes used in the clinic are commonly classified as monocular endoscopes and binocular endoscopes. We have reviewed the classification of methods for depth estimation according to the type of endoscope. Basically, depth estimation relies on feature matching of images and multi-view geometry theory. However, these traditional techniques have many problems in the endoscopic environment. With the increasing development of deep learning techniques, there is a growing number of works based on learning methods to address challenges such as inconsistent illumination and texture sparsity. We have reviewed over 170 papers published in the 10 years from 2013 to 2023. The commonly used public datasets and performance metrics are summarized. We also give a taxonomy of methods and analyze the advantages and drawbacks of algorithms. Summary tables and result atlas are listed to facilitate the comparison of qualitative and quantitative performance of different methods in each category. In addition, we summarize commonly used scene representation methods in endoscopy and speculate on the prospects of deep estimation research in medical applications. We also compare the robustness performance, processing time, and scene representation of the methods to facilitate doctors and researchers in selecting appropriate methods based on surgical applications.
Collapse
Affiliation(s)
- Zhuoyue Yang
- State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, 37 Xueyuan Road, Haidian District, Beijing, 100191, China; Peng Cheng Lab, 2 Xingke 1st Street, Nanshan District, Shenzhen, Guangdong Province, 518000, China
| | - Ju Dai
- Peng Cheng Lab, 2 Xingke 1st Street, Nanshan District, Shenzhen, Guangdong Province, 518000, China
| | - Junjun Pan
- State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, 37 Xueyuan Road, Haidian District, Beijing, 100191, China; Peng Cheng Lab, 2 Xingke 1st Street, Nanshan District, Shenzhen, Guangdong Province, 518000, China.
| |
Collapse
|
3
|
Cao Q, Deng R, Pan Y, Liu R, Chen Y, Gong G, Zou J, Yang H, Han D. Robotic wireless capsule endoscopy: recent advances and upcoming technologies. Nat Commun 2024; 15:4597. [PMID: 38816464 PMCID: PMC11139981 DOI: 10.1038/s41467-024-49019-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2023] [Accepted: 05/21/2024] [Indexed: 06/01/2024] Open
Abstract
Wireless capsule endoscopy (WCE) offers a non-invasive evaluation of the digestive system, eliminating the need for sedation and the risks associated with conventional endoscopic procedures. Its significance lies in diagnosing gastrointestinal tissue irregularities, especially in the small intestine. However, existing commercial WCE devices face limitations, such as the absence of autonomous lesion detection and treatment capabilities. Recent advancements in micro-electromechanical fabrication and computational methods have led to extensive research in sophisticated technology integration into commercial capsule endoscopes, intending to supersede wired endoscopes. This Review discusses the future requirements for intelligent capsule robots, providing a comparative evaluation of various methods' merits and disadvantages, and highlighting recent developments in six technologies relevant to WCE. These include near-field wireless power transmission, magnetic field active drive, ultra-wideband/intrabody communication, hybrid localization, AI-based autonomous lesion detection, and magnetic-controlled diagnosis and treatment. Moreover, we explore the feasibility for future "capsule surgeons".
Collapse
Affiliation(s)
- Qing Cao
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China
- School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China
| | - Runyi Deng
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China
- School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China
| | - Yue Pan
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China
- School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China
| | - Ruijie Liu
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China
- School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China
| | - Yicheng Chen
- Sir Run-Run Shaw Hospital, College of Medicine, Zhejiang University, Hangzhou, 310016, China
| | - Guofang Gong
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China
- School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China
| | - Jun Zou
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China
- School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China
| | - Huayong Yang
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China
- School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China
| | - Dong Han
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China.
- School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China.
| |
Collapse
|
4
|
Furukawa R, Chen E, Sagawa R, Oka S, Kawasaki H. Calibration-free structured-light-based 3D scanning system in laparoscope for robotic surgery. Healthc Technol Lett 2024; 11:196-205. [PMID: 38638488 PMCID: PMC11022229 DOI: 10.1049/htl2.12083] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Accepted: 02/15/2024] [Indexed: 04/20/2024] Open
Abstract
Accurate 3D shape measurement is crucial for surgical support and alignment in robotic surgery systems. Stereo cameras in laparoscopes offer a potential solution; however, their accuracy in stereo image matching diminishes when the target image has few textures. Although stereo matching with deep learning has gained significant attention, supervised learning requires a large dataset of images with depth annotations, which are scarce for laparoscopes. Thus, there is a strong demand to explore alternative methods for depth reconstruction or annotation for laparoscopes. Active stereo techniques are a promising approach for achieving 3D reconstruction without textures. In this study, a 3D shape reconstruction method is proposed using an ultra-small patterned projector attached to a laparoscopic arm to address these issues. The pattern projector emits a structured light with a grid-like pattern that features node-wise modulation for positional encoding. To scan the target object, multiple images are taken while the projector is in motion, and the relative poses of the projector and a camera are auto-calibrated using a differential rendering technique. In the experiment, the proposed method is evaluated by performing 3D reconstruction using images obtained from a surgical robot and comparing the results with a ground-truth shape obtained from X-ray CT.
Collapse
Affiliation(s)
- Ryo Furukawa
- Department of InformaticsKindai UniversityHigashihiroshimaJapan
| | | | - Ryusuke Sagawa
- Artificial Intelligence Research CenterNational Institute of Anvanced Industrial Science and Technology (AIST)TsukubaJapan
| | | | - Hiroshi Kawasaki
- Faculty of Information Science and Electrical EngineeringKyushu UniversityFukuokaJapan
| |
Collapse
|
5
|
Liu S, Fan J, Yang Y, Xiao D, Ai D, Song H, Wang Y, Yang J. Monocular endoscopy images depth estimation with multi-scale residual fusion. Comput Biol Med 2024; 169:107850. [PMID: 38145602 DOI: 10.1016/j.compbiomed.2023.107850] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2023] [Revised: 11/16/2023] [Accepted: 12/11/2023] [Indexed: 12/27/2023]
Abstract
BACKGROUND Monocular depth estimation plays a fundamental role in clinical endoscopy surgery. However, the coherent illumination, smooth surfaces, and texture-less nature of endoscopy images present significant challenges to traditional depth estimation methods. Existing approaches struggle to accurately perceive depth in such settings. METHOD To overcome these challenges, this paper proposes a novel multi-scale residual fusion method for estimating the depth of monocular endoscopy images. Specifically, we address the issue of coherent illumination by leveraging image frequency domain component space transformation, thereby enhancing the stability of the scene's light source. Moreover, we employ an image radiation intensity attenuation model to estimate the initial depth map. Finally, to refine the accuracy of depth estimation, we utilize a multi-scale residual fusion optimization technique. RESULTS To evaluate the performance of our proposed method, extensive experiments were conducted on public datasets. The structural similarity measures for continuous frames in three distinct clinical data scenes reached impressive values of 0.94, 0.82, and 0.84, respectively. These results demonstrate the effectiveness of our approach in capturing the intricate details of endoscopy images. Furthermore, the depth estimation accuracy achieved remarkable levels of 89.3 % and 91.2 % for the two models' data, respectively, underscoring the robustness of our method. CONCLUSIONS Overall, the promising results obtained on public datasets highlight the significant potential of our method for clinical applications, facilitating reliable depth estimation and enhancing the quality of endoscopy surgical procedures.
Collapse
Affiliation(s)
- Shiyuan Liu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China; China Center for Information Industry Development, Beijing, 100081, China
| | - Jingfan Fan
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China.
| | - Yun Yang
- Department of General Surgery, Beijing Friendship Hospital, Capital Medical University, National Clinical Research Center for Digestive Diseases, Beijing 100050, China
| | - Deqiang Xiao
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Danni Ai
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Hong Song
- School of Computer Science and Technology, Beijing Institute of Technology, Beijing, 100081, China
| | - Yongtian Wang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China.
| | - Jian Yang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| |
Collapse
|
6
|
Lu T, Ji S, Jin W, Yang Q, Luo Q, Ren TL. Biocompatible and Long-Term Monitoring Strategies of Wearable, Ingestible and Implantable Biosensors: Reform the Next Generation Healthcare. SENSORS (BASEL, SWITZERLAND) 2023; 23:2991. [PMID: 36991702 PMCID: PMC10054135 DOI: 10.3390/s23062991] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/16/2022] [Revised: 12/31/2022] [Accepted: 01/04/2023] [Indexed: 06/19/2023]
Abstract
Sensors enable the detection of physiological indicators and pathological markers to assist in the diagnosis, treatment, and long-term monitoring of diseases, in addition to playing an essential role in the observation and evaluation of physiological activities. The development of modern medical activities cannot be separated from the precise detection, reliable acquisition, and intelligent analysis of human body information. Therefore, sensors have become the core of new-generation health technologies along with the Internet of Things (IoTs) and artificial intelligence (AI). Previous research on the sensing of human information has conferred many superior properties on sensors, of which biocompatibility is one of the most important. Recently, biocompatible biosensors have developed rapidly to provide the possibility for the long-term and in-situ monitoring of physiological information. In this review, we summarize the ideal features and engineering realization strategies of three different types of biocompatible biosensors, including wearable, ingestible, and implantable sensors from the level of sensor designing and application. Additionally, the detection targets of the biosensors are further divided into vital life parameters (e.g., body temperature, heart rate, blood pressure, and respiratory rate), biochemical indicators, as well as physical and physiological parameters based on the clinical needs. In this review, starting from the emerging concept of next-generation diagnostics and healthcare technologies, we discuss how biocompatible sensors revolutionize the state-of-art healthcare system unprecedentedly, as well as the challenges and opportunities faced in the future development of biocompatible health sensors.
Collapse
Affiliation(s)
- Tian Lu
- School of Integrated Circuit and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
| | - Shourui Ji
- School of Integrated Circuit and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
| | - Weiqiu Jin
- Shanghai Lung Cancer Center, Shanghai Chest Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai 200030, China
| | - Qisheng Yang
- School of Integrated Circuit and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
| | - Qingquan Luo
- Shanghai Lung Cancer Center, Shanghai Chest Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai 200030, China
| | - Tian-Ling Ren
- School of Integrated Circuit and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
| |
Collapse
|
7
|
Abstract
Visual SLAM (VSLAM) has been developing rapidly due to its advantages of low-cost sensors, the easy fusion of other sensors, and richer environmental information. Traditional visionbased SLAM research has made many achievements, but it may fail to achieve wished results in challenging environments. Deep learning has promoted the development of computer vision, and the combination of deep learning and SLAM has attracted more and more attention. Semantic information, as high-level environmental information, can enable robots to better understand the surrounding environment. This paper introduces the development of VSLAM technology from two aspects: traditional VSLAM and semantic VSLAM combined with deep learning. For traditional VSLAM, we summarize the advantages and disadvantages of indirect and direct methods in detail and give some classical VSLAM open-source algorithms. In addition, we focus on the development of semantic VSLAM based on deep learning. Starting with typical neural networks CNN and RNN, we summarize the improvement of neural networks for the VSLAM system in detail. Later, we focus on the help of target detection and semantic segmentation for VSLAM semantic information introduction. We believe that the development of the future intelligent era cannot be without the help of semantic technology. Introducing deep learning into the VSLAM system to provide semantic information can help robots better perceive the surrounding environment and provide people with higher-level help.
Collapse
|
8
|
SelfVIO: Self-supervised deep monocular Visual–Inertial Odometry and depth estimation. Neural Netw 2022; 150:119-136. [DOI: 10.1016/j.neunet.2022.03.005] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2022] [Revised: 03/01/2022] [Accepted: 03/03/2022] [Indexed: 01/31/2023]
|
9
|
Yang M, Sun X, Jia F, Rushworth A, Dong X, Zhang S, Fang Z, Yang G, Liu B. Sensors and Sensor Fusion Methodologies for Indoor Odometry: A Review. Polymers (Basel) 2022; 14:polym14102019. [PMID: 35631899 PMCID: PMC9143447 DOI: 10.3390/polym14102019] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2022] [Revised: 05/05/2022] [Accepted: 05/11/2022] [Indexed: 02/04/2023] Open
Abstract
Although Global Navigation Satellite Systems (GNSSs) generally provide adequate accuracy for outdoor localization, this is not the case for indoor environments, due to signal obstruction. Therefore, a self-contained localization scheme is beneficial under such circumstances. Modern sensors and algorithms endow moving robots with the capability to perceive their environment, and enable the deployment of novel localization schemes, such as odometry, or Simultaneous Localization and Mapping (SLAM). The former focuses on incremental localization, while the latter stores an interpretable map of the environment concurrently. In this context, this paper conducts a comprehensive review of sensor modalities, including Inertial Measurement Units (IMUs), Light Detection and Ranging (LiDAR), radio detection and ranging (radar), and cameras, as well as applications of polymers in these sensors, for indoor odometry. Furthermore, analysis and discussion of the algorithms and the fusion frameworks for pose estimation and odometry with these sensors are performed. Therefore, this paper straightens the pathway of indoor odometry from principle to application. Finally, some future prospects are discussed.
Collapse
Affiliation(s)
- Mengshen Yang
- Department of Mechanical, Materials and Manufacturing Engineering, The Faculty of Science and Engineering, University of Nottingham Ningbo China, Ningbo 315100, China; (M.Y.); (F.J.); (B.L.)
- Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo 315201, China;
- Zhejiang Key Laboratory of Robotics and Intelligent Manufacturing Equipment Technology, Ningbo 315201, China
| | - Xu Sun
- Department of Mechanical, Materials and Manufacturing Engineering, The Faculty of Science and Engineering, University of Nottingham Ningbo China, Ningbo 315100, China; (M.Y.); (F.J.); (B.L.)
- Nottingham Ningbo China Beacons of Excellence Research and Innovation Institute, University of Nottingham Ningbo China, Ningbo 315100, China
- Correspondence: (X.S.); (A.R.); (G.Y.)
| | - Fuhua Jia
- Department of Mechanical, Materials and Manufacturing Engineering, The Faculty of Science and Engineering, University of Nottingham Ningbo China, Ningbo 315100, China; (M.Y.); (F.J.); (B.L.)
| | - Adam Rushworth
- Department of Mechanical, Materials and Manufacturing Engineering, The Faculty of Science and Engineering, University of Nottingham Ningbo China, Ningbo 315100, China; (M.Y.); (F.J.); (B.L.)
- Correspondence: (X.S.); (A.R.); (G.Y.)
| | - Xin Dong
- Department of Mechanical, Materials and Manufacturing Engineering, University of Nottingham, Nottingham NG7 2RD, UK;
| | - Sheng Zhang
- Ningbo Research Institute, Zhejiang University, Ningbo 315100, China;
| | - Zaojun Fang
- Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo 315201, China;
- Zhejiang Key Laboratory of Robotics and Intelligent Manufacturing Equipment Technology, Ningbo 315201, China
| | - Guilin Yang
- Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo 315201, China;
- Zhejiang Key Laboratory of Robotics and Intelligent Manufacturing Equipment Technology, Ningbo 315201, China
- Correspondence: (X.S.); (A.R.); (G.Y.)
| | - Bingjian Liu
- Department of Mechanical, Materials and Manufacturing Engineering, The Faculty of Science and Engineering, University of Nottingham Ningbo China, Ningbo 315100, China; (M.Y.); (F.J.); (B.L.)
| |
Collapse
|
10
|
Liu S, Fan J, Song D, Fu T, Lin Y, Xiao D, Song H, Wang Y, Yang J. Joint estimation of depth and motion from a monocular endoscopy image sequence using a multi-loss rebalancing network. BIOMEDICAL OPTICS EXPRESS 2022; 13:2707-2727. [PMID: 35774318 PMCID: PMC9203100 DOI: 10.1364/boe.457475] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/01/2022] [Revised: 04/01/2022] [Accepted: 04/01/2022] [Indexed: 06/15/2023]
Abstract
Building an in vivo three-dimensional (3D) surface model from a monocular endoscopy is an effective technology to improve the intuitiveness and precision of clinical laparoscopic surgery. This paper proposes a multi-loss rebalancing-based method for joint estimation of depth and motion from a monocular endoscopy image sequence. The feature descriptors are used to provide monitoring signals for the depth estimation network and motion estimation network. The epipolar constraints of the sequence frame is considered in the neighborhood spatial information by depth estimation network to enhance the accuracy of depth estimation. The reprojection information of depth estimation is used to reconstruct the camera motion by motion estimation network with a multi-view relative pose fusion mechanism. The relative response loss, feature consistency loss, and epipolar consistency loss function are defined to improve the robustness and accuracy of the proposed unsupervised learning-based method. Evaluations are implemented on public datasets. The error of motion estimation in three scenes decreased by 42.1%,53.6%, and 50.2%, respectively. And the average error of 3D reconstruction is 6.456 ± 1.798mm. This demonstrates its capability to generate reliable depth estimation and trajectory reconstruction results for endoscopy images and meaningful applications in clinical.
Collapse
Affiliation(s)
- Shiyuan Liu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Jingfan Fan
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Dengpan Song
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Tianyu Fu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Yucong Lin
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Deqiang Xiao
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Hong Song
- School of Computer Science and Technology, Beijing Institute of Technology, Beijing, 100081, China
| | - Yongtian Wang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Jian Yang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| |
Collapse
|
11
|
|
12
|
Detection of elusive polyps using a large-scale artificial intelligence system (with videos). Gastrointest Endosc 2021; 94:1099-1109.e10. [PMID: 34216598 DOI: 10.1016/j.gie.2021.06.021] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/25/2021] [Accepted: 06/22/2021] [Indexed: 12/11/2022]
Abstract
BACKGROUND AND AIMS Colorectal cancer is a leading cause of death. Colonoscopy is the criterion standard for detection and removal of precancerous lesions and has been shown to reduce mortality. The polyp miss rate during colonoscopies is 22% to 28%. DEEP DEtection of Elusive Polyps (DEEP2) is a new polyp detection system based on deep learning that alerts the operator in real time to the presence and location of polyps. The primary outcome was the performance of DEEP2 on the detection of elusive polyps. METHODS The DEEP2 system was trained on 3611 hours of colonoscopy videos derived from 2 sources and was validated on a set comprising 1393 hours from a third unrelated source. Ground truth labeling was provided by offline gastroenterologist annotators who were able to watch the video in slow motion and pause and rewind as required. To assess applicability, stability, and user experience and to obtain some preliminary data on performance in a real-life scenario, a preliminary prospective clinical validation study was performed comprising 100 procedures. RESULTS DEEP2 achieved a sensitivity of 97.1% at 4.6 false alarms per video for all polyps and of 88.5% and 84.9% for polyps in the field of view for less than 5 and 2 seconds, respectively. DEEP2 was able to detect polyps not seen by live real-time endoscopists or offline annotators in an average of .22 polyps per sequence. In the clinical validation study, the system detected an average of .89 additional polyps per procedure. No adverse events occurred. CONCLUSIONS DEEP2 has a high sensitivity for polyp detection and was effective in increasing the detection of polyps both in colonoscopy videos and in real procedures with a low number of false alarms. (Clinical trial registration number: NCT04693078.).
Collapse
|
13
|
Widya AR, Monno Y, Okutomi M, Suzuki S, Gotoda T, Miki K. Learning-Based Depth and Pose Estimation for Monocular Endoscope with Loss Generalization. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:3547-3552. [PMID: 34892005 DOI: 10.1109/embc46164.2021.9630156] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Gastroendoscopy has been a clinical standard for diagnosing and treating conditions that affect a part of a patient's digestive system, such as the stomach. Despite the fact that gastroendoscopy has a lot of advantages for patients, there exist some challenges for practitioners, such as the lack of 3D perception, including the depth and the endoscope pose information. Such challenges make navigating the endoscope and localizing any found lesion in a digestive tract difficult. To tackle these problems, deep learning-based approaches have been proposed to provide monocular gastroendoscopy with additional yet important depth and pose information. In this paper, we propose a novel supervised approach to train depth and pose estimation networks using consecutive endoscopy images to assist the endoscope navigation in the stomach. We firstly generate real depth and pose training data using our previously proposed whole stomach 3D reconstruction pipeline to avoid poor generalization ability between computer-generated (CG) models and real data for the stomach. In addition, we propose a novel generalized photometric loss function to avoid the complicated process of finding proper weights for balancing the depth and the pose loss terms, which is required for existing direct depth and pose supervision approaches. We then experimentally show that our proposed generalized loss performs better than existing direct supervision losses.
Collapse
|
14
|
Motion-based camera localization system in colonoscopy videos. Med Image Anal 2021; 73:102180. [PMID: 34303888 DOI: 10.1016/j.media.2021.102180] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2020] [Revised: 07/09/2021] [Accepted: 07/13/2021] [Indexed: 11/20/2022]
Abstract
Optical colonoscopy is an essential diagnostic and prognostic tool for many gastrointestinal diseases, including cancer screening and staging, intestinal bleeding, diarrhea, abdominal symptom evaluation, and inflammatory bowel disease assessment. However, the evaluation, classification, and quantification of findings from colonoscopy are subject to inter-observer variation. Automated assessment of colonoscopy is of interest considering the subjectivity present in qualitative human interpretations of colonoscopy findings. Localization of the camera is essential to interpreting the meaning and context of findings for diseases evaluated by colonoscopy. In this study, we propose a camera localization system to estimate the relative location of the camera and classify the colon into anatomical segments. The camera localization system begins with non-informative frame detection and removal. Then a self-training end-to-end convolutional neural network is built to estimate the camera motion, where several strategies are proposed to improve its robustness and generalization on endoscopic videos. Using the estimated camera motion a camera trajectory can be derived and a relative location index calculated. Based on the estimated location index, anatomical colon segment classification is performed by constructing a colon template. The proposed motion estimation algorithm was evaluated on an external dataset containing the ground truth for camera pose. The experimental results show that the performance of the proposed method is superior to other published methods. The relative location index estimation and anatomical region classification were further validated using colonoscopy videos collected from routine clinical practice. This validation yielded an average accuracy in classification of 0.754, which is substantially higher than the performances obtained using location indices built from other methods.
Collapse
|
15
|
Vedaei SS, Wahid KA. A localization method for wireless capsule endoscopy using side wall cameras and IMU sensor. Sci Rep 2021; 11:11204. [PMID: 34045554 PMCID: PMC8160358 DOI: 10.1038/s41598-021-90523-w] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2020] [Accepted: 05/11/2021] [Indexed: 02/08/2023] Open
Abstract
Localizing the endoscopy capsule inside gastrointestinal (GI) system provides key information which leads to GI abnormality tracking and precision medical delivery. In this paper, we have proposed a new method to localize the capsule inside human GI track. We propose to equip the capsule with four side wall cameras and an Inertial Measurement Unit (IMU), that consists of 9 Degree-Of-Freedom (DOF) including a gyroscope, an accelerometer and a magnetometer to monitor the capsule’s orientation and direction of travel. The low resolution mono-chromatic cameras, installed along the wide wall, are responsible to measure the actual capsule movement, not the involuntary motion of the small intestine. Finally, a fusion algorithm is used to combine all data to derive the traveled path and plot the trajectory. Compared to other methods, the presented system is resistive to surrounding conditions, such as GI nonhomogeneous structure and involuntary small bowel movements. In addition, it does not require external antenna or arrays. Therefore, GI tracking can be achieved without disturbing patients’ daily activities.
Collapse
Affiliation(s)
- Seyed Shahim Vedaei
- Department of Electrical and Computer Engineering, University of Saskatchewan, Saskatoon, SK, S7N 5A9, Canada.
| | - Khan A Wahid
- Department of Electrical and Computer Engineering, University of Saskatchewan, Saskatoon, SK, S7N 5A9, Canada
| |
Collapse
|
16
|
Chang KP, Lin SH, Chu YW. Artificial intelligence in gastrointestinal radiology: A review with special focus on recent development of magnetic resonance and computed tomography. Artif Intell Gastroenterol 2021; 2:27-41. [DOI: 10.35712/aig.v2.i2.27] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Revised: 03/21/2021] [Accepted: 04/20/2021] [Indexed: 02/06/2023] Open
Abstract
Artificial intelligence (AI), particularly the deep learning technology, have been proven influential to radiology in the recent decade. Its ability in image classification, segmentation, detection and reconstruction tasks have substantially assisted diagnostic radiology, and has even been viewed as having the potential to perform better than radiologists in some tasks. Gastrointestinal radiology, an important subspecialty dealing with complex anatomy and various modalities including endoscopy, have especially attracted the attention of AI researchers and engineers worldwide. Consequently, recently many tools have been developed for lesion detection and image construction in gastrointestinal radiology, particularly in the fields for which public databases are available, such as diagnostic abdominal magnetic resonance imaging (MRI) and computed tomography (CT). This review will provide a framework for understanding recent advancements of AI in gastrointestinal radiology, with a special focus on hepatic and pancreatobiliary diagnostic radiology with MRI and CT. For fields where AI is less developed, this review will also explain the difficulty in AI model training and possible strategies to overcome the technical issues. The authors’ insights of possible future development will be addressed in the last section.
Collapse
Affiliation(s)
- Kai-Po Chang
- PhD Program in Medical Biotechnology, National Chung Hsing University, Taichung 40227, Taiwan
- Department of Pathology, China Medical University Hospital, Taichung 40447, Taiwan
| | - Shih-Huan Lin
- PhD Program in Medical Biotechnology, National Chung Hsing University, Taichung 40227, Taiwan
| | - Yen-Wei Chu
- PhD Program in Medical Biotechnology, National Chung Hsing University, Taichung 40227, Taiwan
- Institute of Genomics and Bioinformatics, National Chung Hsing University, Taichung 40227, Taiwan
- Institute of Molecular Biology, National Chung Hsing University, Taichung 40227, Taiwan
- Agricultural Biotechnology Center, National Chung Hsing University, Taichung 40227, Taiwan
- Biotechnology Center, National Chung Hsing University, Taichung 40227, Taiwan
- PhD Program in Translational Medicine, National Chung Hsing University, Taichung 40227, Taiwan
- Rong Hsing Research Center for Translational Medicine, Taichung 40227, Taiwan
| |
Collapse
|
17
|
Tariverdi A, Venkiteswaran VK, Richter M, Elle OJ, Tørresen J, Mathiassen K, Misra S, Martinsen ØG. A Recurrent Neural-Network-Based Real-Time Dynamic Model for Soft Continuum Manipulators. Front Robot AI 2021; 8:631303. [PMID: 33869294 PMCID: PMC8044932 DOI: 10.3389/frobt.2021.631303] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2020] [Accepted: 02/05/2021] [Indexed: 11/25/2022] Open
Abstract
This paper introduces and validates a real-time dynamic predictive model based on a neural network approach for soft continuum manipulators. The presented model provides a real-time prediction framework using neural-network-based strategies and continuum mechanics principles. A time-space integration scheme is employed to discretize the continuous dynamics and decouple the dynamic equations for translation and rotation for each node of a soft continuum manipulator. Then the resulting architecture is used to develop distributed prediction algorithms using recurrent neural networks. The proposed RNN-based parallel predictive scheme does not rely on computationally intensive algorithms; therefore, it is useful in real-time applications. Furthermore, simulations are shown to illustrate the approach performance on soft continuum elastica, and the approach is also validated through an experiment on a magnetically-actuated soft continuum manipulator. The results demonstrate that the presented model can outperform classical modeling approaches such as the Cosserat rod model while also shows possibilities for being used in practice.
Collapse
Affiliation(s)
| | | | - Michiel Richter
- Department of Biomechanical Engineering, University of Twente, Enschede, Netherlands
| | - Ole J Elle
- The Intervention Centre, Oslo University Hospital, Oslo, Norway.,Department of Informatics, University of Oslo, Oslo, Norway
| | - Jim Tørresen
- Department of Informatics, University of Oslo, Oslo, Norway
| | - Kim Mathiassen
- Department of Technology Systems, University of Oslo, Oslo, Norway
| | - Sarthak Misra
- Department of Biomechanical Engineering, University of Twente, Enschede, Netherlands.,Department of Biomedical Engineering, University of Groningen and University Medical Centre Groningen, Groningen, Netherlands
| | - Ørjan G Martinsen
- Department of Physics, University of Oslo, Oslo, Norway.,Department of Clinical and Biomedical Engineering, Oslo University Hospital, Oslo, Norway
| |
Collapse
|
18
|
İncetan K, Celik IO, Obeid A, Gokceler GI, Ozyoruk KB, Almalioglu Y, Chen RJ, Mahmood F, Gilbert H, Durr NJ, Turan M. VR-Caps: A Virtual Environment for Capsule Endoscopy. Med Image Anal 2021; 70:101990. [PMID: 33609920 DOI: 10.1016/j.media.2021.101990] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2020] [Revised: 02/01/2021] [Accepted: 02/02/2021] [Indexed: 02/06/2023]
Abstract
Current capsule endoscopes and next-generation robotic capsules for diagnosis and treatment of gastrointestinal diseases are complex cyber-physical platforms that must orchestrate complex software and hardware functions. The desired tasks for these systems include visual localization, depth estimation, 3D mapping, disease detection and segmentation, automated navigation, active control, path realization and optional therapeutic modules such as targeted drug delivery and biopsy sampling. Data-driven algorithms promise to enable many advanced functionalities for capsule endoscopes, but real-world data is challenging to obtain. Physically-realistic simulations providing synthetic data have emerged as a solution to the development of data-driven algorithms. In this work, we present a comprehensive simulation platform for capsule endoscopy operations and introduce VR-Caps, a virtual active capsule environment that simulates a range of normal and abnormal tissue conditions (e.g., inflated, dry, wet etc.) and varied organ types, capsule endoscope designs (e.g., mono, stereo, dual and 360∘ camera), and the type, number, strength, and placement of internal and external magnetic sources that enable active locomotion. VR-Caps makes it possible to both independently or jointly develop, optimize, and test medical imaging and analysis software for the current and next-generation endoscopic capsule systems. To validate this approach, we train state-of-the-art deep neural networks to accomplish various medical image analysis tasks using simulated data from VR-Caps and evaluate the performance of these models on real medical data. Results demonstrate the usefulness and effectiveness of the proposed virtual platform in developing algorithms that quantify fractional coverage, camera trajectory, 3D map reconstruction, and disease classification. All of the code, pre-trained weights and created 3D organ models of the virtual environment with detailed instructions how to setup and use the environment are made publicly available at https://github.com/CapsuleEndoscope/VirtualCapsuleEndoscopy and a video demonstration can be seen in the supplementary videos (Video-I).
Collapse
Affiliation(s)
- Kağan İncetan
- Institute of Biomedical Engineering, Bogazici University, Istanbul, Turkey
| | - Ibrahim Omer Celik
- Department of Computer Engineering, Bogazici University, Istanbul, Turkey
| | - Abdulhamid Obeid
- Institute of Biomedical Engineering, Bogazici University, Istanbul, Turkey
| | | | | | | | - Richard J Chen
- Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Faisal Mahmood
- Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Cancer Data Science, Dana Farber Cancer Institute, Boston, MA, USA; Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
| | - Hunter Gilbert
- Deparment of Mechanical and Industrial Engineering, Louisiana State University, Baton Rouge, LA USA
| | - Nicholas J Durr
- Department of Biomedical Engineering, Johns Hopkins University (JHU), Baltimore, MD, USA
| | - Mehmet Turan
- Institute of Biomedical Engineering, Bogazici University, Istanbul, Turkey.
| |
Collapse
|
19
|
Almalioglu Y, Bengisu Ozyoruk K, Gokce A, Incetan K, Irem Gokceler G, Ali Simsek M, Ararat K, Chen RJ, Durr NJ, Mahmood F, Turan M. EndoL2H: Deep Super-Resolution for Capsule Endoscopy. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:4297-4309. [PMID: 32795966 DOI: 10.1109/tmi.2020.3016744] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Although wireless capsule endoscopy is the preferred modality for diagnosis and assessment of small bowel diseases, the poor camera resolution is a substantial limitation for both subjective and automated diagnostics. Enhanced-resolution endoscopy has shown to improve adenoma detection rate for conventional endoscopy and is likely to do the same for capsule endoscopy. In this work, we propose and quantitatively validate a novel framework to learn a mapping from low-to-high-resolution endoscopic images. We combine conditional adversarial networks with a spatial attention block to improve the resolution by up to factors of 8× , 10× , 12× , respectively. Quantitative and qualitative studies demonstrate the superiority of EndoL2H over state-of-the-art deep super-resolution methods Deep Back-Projection Networks (DBPN), Deep Residual Channel Attention Networks (RCAN) and Super Resolution Generative Adversarial Network (SRGAN). Mean Opinion Score (MOS) tests were performed by 30 gastroenterologists qualitatively assess and confirm the clinical relevance of the approach. EndoL2H is generally applicable to any endoscopic capsule system and has the potential to improve diagnosis and better harness computational approaches for polyp detection and characterization. Our code and trained models are available at https://github.com/CapsuleEndoscope/EndoL2H.
Collapse
|
20
|
Freedman D, Blau Y, Katzir L, Aides A, Shimshoni I, Veikherman D, Golany T, Gordon A, Corrado G, Matias Y, Rivlin E. Detecting Deficient Coverage in Colonoscopies. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:3451-3462. [PMID: 32746092 DOI: 10.1109/tmi.2020.2994221] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Colonoscopy is tool of choice for preventing Colorectal Cancer, by detecting and removing polyps before they become cancerous. However, colonoscopy is hampered by the fact that endoscopists routinely miss 22-28% of polyps. While some of these missed polyps appear in the endoscopist's field of view, others are missed simply because of substandard coverage of the procedure, i.e. not all of the colon is seen. This paper attempts to rectify the problem of substandard coverage in colonoscopy through the introduction of the C2D2 (Colonoscopy Coverage Deficiency via Depth) algorithm which detects deficient coverage, and can thereby alert the endoscopist to revisit a given area. More specifically, C2D2 consists of two separate algorithms: the first performs depth estimation of the colon given an ordinary RGB video stream; while the second computes coverage given these depth estimates. Rather than compute coverage for the entire colon, our algorithm computes coverage locally, on a segment-by-segment basis; C2D2 can then indicate in real-time whether a particular area of the colon has suffered from deficient coverage, and if so the endoscopist can return to that area. Our coverage algorithm is the first such algorithm to be evaluated in a large-scale way; while our depth estimation technique is the first calibration-free unsupervised method applied to colonoscopies. The C2D2 algorithm achieves state of the art results in the detection of deficient coverage. On synthetic sequences with ground truth, it is 2.4 times more accurate than human experts; while on real sequences, C2D2 achieves a 93.0% agreement with experts.
Collapse
|
21
|
Zhou M, Feng Y, Xue C, Han F. Deep convolutional neural network based fractional-order terminal sliding-mode control for robotic manipulators. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2019.04.087] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|
22
|
Dai X, Mao Y, Huang T, Qin N, Huang D, Li Y. Automatic obstacle avoidance of quadrotor UAV via CNN-based learning. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2020.04.020] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
23
|
Application of artificial intelligence in surgery. Front Med 2020; 14:417-430. [DOI: 10.1007/s11684-020-0770-0] [Citation(s) in RCA: 34] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2019] [Accepted: 03/05/2020] [Indexed: 12/14/2022]
|
24
|
|
25
|
Jones A, Rush A, Merkel C, Herrmann E, Jacob AP, Thiem C, Jha R. A neuromorphic SLAM architecture using gated-memristive synapses. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2019.09.098] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
|
26
|
Ali H, Sharif M, Yasmin M, Rehmani MH, Riaz F. A survey of feature extraction and fusion of deep learning for detection of abnormalities in video endoscopy of gastrointestinal-tract. Artif Intell Rev 2019. [DOI: 10.1007/s10462-019-09743-2] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022]
|
27
|
Wang M, Shi Q, Song S, Hu C, Meng MQH. A Novel Relative Position Estimation Method for Capsule Robot Moving in Gastrointestinal Tract. SENSORS 2019; 19:s19122746. [PMID: 31248092 PMCID: PMC6630487 DOI: 10.3390/s19122746] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/16/2019] [Revised: 06/14/2019] [Accepted: 06/16/2019] [Indexed: 12/19/2022]
Abstract
Recently, a variety of positioning and tracking methods have been proposed for capsule robots moving in the gastrointestinal (GI) tract to provide real-time unobstructed spatial pose results. However, the current absolute position-based result cannot match the GI structure due to its unstructured environment. To overcome this disadvantage and provide a proper position description method to match the GI tract, we here present a relative position estimation method for tracking the capsule robot, which uses the moving distance of the robot along the GI tract to indicate the position result. The procedure of the proposed method is as follows: firstly, the absolute position results of the capsule robot are obtained with the magnetic tracking method; then, the moving status of the robot along the GI tract is determined according to the moving direction; and finally, the movement trajectory of the capsule robot is fitted with the Bézier curve, where the moving distance can then be evaluated using the integral method. Compared to state-of-the-art capsule tracking methods, the proposed method can directly help to guide medical instruments by providing physicians the insertion distance in patients’ bodies, which cannot be done based on absolute position results. Moreover, as relative distance information was used, no reference tracking objects needed to be mounted onto the human body. The experimental results prove that the proposed method achieves a good distance estimation of the capsule robot moving in the simulation platform.
Collapse
Affiliation(s)
- Min Wang
- Shenzhen Engineering Lab for Medical Intelligent Wireless Ultrasonic Imaging Technology, Harbin Institute of Technolgoy, Shenzhen 518055, China.
| | - Qinyuan Shi
- Shenzhen Engineering Lab for Medical Intelligent Wireless Ultrasonic Imaging Technology, Harbin Institute of Technolgoy, Shenzhen 518055, China.
| | - Shuang Song
- Shenzhen Engineering Lab for Medical Intelligent Wireless Ultrasonic Imaging Technology, Harbin Institute of Technolgoy, Shenzhen 518055, China.
| | - Chao Hu
- Ningbo Institute of Technology, Zhejiang University, Ningbo 315000, China.
| | - Max Q-H Meng
- Robotics, Perception and Artificial Intelligence Lab, The Chinese University of Hong Kong, N.T., Hong Kong 999077, China.
| |
Collapse
|
28
|
Vasilakakis M, Koulaouzidis A, Yung DE, Plevris JN, Toth E, Iakovidis DK. Follow-up on: optimizing lesion detection in small bowel capsule endoscopy and beyond: from present problems to future solutions. Expert Rev Gastroenterol Hepatol 2019; 13:129-141. [PMID: 30791780 DOI: 10.1080/17474124.2019.1553616] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/16/2022]
Abstract
This review presents noteworthy advances in clinical and experimental Capsule Endoscopy (CE), focusing on the progress that has been reported over the last 5 years since our previous review on the subject. Areas covered: This study presents the commercially available CE platforms, as well as the advances made in optimizing the diagnostic capabilities of CE. The latter includes recent concept and prototype capsule endoscopes, medical approaches to improve diagnostic yield, and progress in software for enhancing visualization, abnormality detection, and lesion localization. Expert commentary: Currently, moving through the second decade of CE evolution, there are still several open issues and remarkable challenges to overcome.
Collapse
Affiliation(s)
- Michael Vasilakakis
- a Department of Computer Science and Biomedical Informatics , University of Thessaly , Lamia , Greece
| | - Anastasios Koulaouzidis
- b Endoscopy Unit , The Royal Infirmary of Edinburgh , Edinburgh , Scotland.,c Department of Clinical Sciences , Lund University , Malmö , Sweden
| | - Diana E Yung
- b Endoscopy Unit , The Royal Infirmary of Edinburgh , Edinburgh , Scotland
| | - John N Plevris
- b Endoscopy Unit , The Royal Infirmary of Edinburgh , Edinburgh , Scotland
| | - Ervin Toth
- c Department of Clinical Sciences , Lund University , Malmö , Sweden.,d Section of Gastroenterology, Department of Clinical Sciences , Skåne University Hospital Malmö , Malmö , Sweden
| | - Dimitris K Iakovidis
- a Department of Computer Science and Biomedical Informatics , University of Thessaly , Lamia , Greece
| |
Collapse
|
29
|
High Precision Detection of Salient Objects Based on Deep Convolutional Networks with Proper Combinations of Shallow and Deep Connections. Symmetry (Basel) 2018. [DOI: 10.3390/sym11010005] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
In this paper, a high precision detection method of salient objects is presented based on deep convolutional networks with proper combinations of shallow and deep connections. In order to achieve better performance in the extraction of deep semantic features of salient objects, based on a symmetric encoder and decoder architecture, an upgrade of backbone networks is carried out with a transferable model on the ImageNet pre-trained ResNet50. Moreover, by introducing shallow and deep connections on multiple side outputs, feature maps generated from various layers of the deep neural network (DNN) model are well fused so as to describe salient objects from local and global aspects comprehensively. Afterwards, based on a holistically nested edge detector (HED) architecture, multiple fused side outputs with various sizes of receptive fields are integrated to form detection results of salient objects accordingly. A series of experiments and assessments on extensive benchmark datasets demonstrate the dominant performance of our DNN model for the detection of salient objects in accuracy, which has outperformed those of other published works.
Collapse
|
30
|
Tabak AF. Hydrodynamic Impedance Correction for Reduced‐Order Modeling of Spermatozoa‐Like Soft Micro‐Robots. ADVANCED THEORY AND SIMULATIONS 2018. [DOI: 10.1002/adts.201800130] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Ahmet Fatih Tabak
- Mechatronics Engineering DepartmentFaculty of EngineeringOkan University Akfirat‐Tuzla/Istanbul 34959 Turkey
| |
Collapse
|
31
|
Li R, Wang S, Gu D. Ongoing Evolution of Visual SLAM from Geometry to Deep Learning: Challenges and Opportunities. Cognit Comput 2018. [DOI: 10.1007/s12559-018-9591-8] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
32
|
Wu Y, Tang F, Li H. Image-based camera localization: an overview. Vis Comput Ind Biomed Art 2018; 1:8. [PMID: 32240389 PMCID: PMC7099558 DOI: 10.1186/s42492-018-0008-z] [Citation(s) in RCA: 32] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2018] [Accepted: 07/06/2018] [Indexed: 11/22/2022] Open
Abstract
Virtual reality, augmented reality, robotics, and autonomous driving, have recently attracted much attention from both academic and industrial communities, in which image-based camera localization is a key task. However, there has not been a complete review on image-based camera localization. It is urgent to map this topic to enable individuals enter the field quickly. In this paper, an overview of image-based camera localization is presented. A new and complete classification of image-based camera localization approaches is provided and the related techniques are introduced. Trends for future development are also discussed. This will be useful not only to researchers, but also to engineers and other individuals interested in this field.
Collapse
Affiliation(s)
- Yihong Wu
- National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences, Beijing, China.
| | - Fulin Tang
- National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences, Beijing, China
| | - Heping Li
- National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences, Beijing, China
| |
Collapse
|