1
|
Liu R, Liu Z, Lu J, Zhang G, Zuo Z, Sun B, Zhang J, Sheng W, Guo R, Zhang L, Hua X. Sparse-to-dense coarse-to-fine depth estimation for colonoscopy. Comput Biol Med 2023; 160:106983. [PMID: 37187133 DOI: 10.1016/j.compbiomed.2023.106983] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Revised: 04/17/2023] [Accepted: 04/27/2023] [Indexed: 05/17/2023]
Abstract
Colonoscopy, as the golden standard for screening colon cancer and diseases, offers considerable benefits to patients. However, it also imposes challenges on diagnosis and potential surgery due to the narrow observation perspective and limited perception dimension. Dense depth estimation can overcome the above limitations and offer doctors straightforward 3D visual feedback. To this end, we propose a novel sparse-to-dense coarse-to-fine depth estimation solution for colonoscopic scenes based on the direct SLAM algorithm. The highlight of our solution is that we utilize the scattered 3D points obtained from SLAM to generate accurate and dense depth in full resolution. This is done by a deep learning (DL)-based depth completion network and a reconstruction system. The depth completion network effectively extracts texture, geometry, and structure features from sparse depth along with RGB data to recover the dense depth map. The reconstruction system further updates the dense depth map using a photometric error-based optimization and a mesh modeling approach to reconstruct a more accurate 3D model of colons with detailed surface texture. We show the effectiveness and accuracy of our depth estimation method on near photo-realistic challenging colon datasets. Experiments demonstrate that the strategy of sparse-to-dense coarse-to-fine can significantly improve the performance of depth estimation and smoothly fuse direct SLAM and DL-based depth estimation into a complete dense reconstruction system.
Collapse
Affiliation(s)
- Ruyu Liu
- School of Information Science and Technology, Hangzhou Normal University, Hangzhou, 311121, China; Haixi Institutes, Chinese Academy of Sciences Quanzhou Institute of Equipment Manufacturing, Quanzhou, 362000, China
| | - Zhengzhe Liu
- School of Information Science and Technology, Hangzhou Normal University, Hangzhou, 311121, China
| | - Jiaming Lu
- School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, 300384, China
| | - Guodao Zhang
- Department of Digital Media Technology, Hangzhou Dianzi University, Hangzhou, 310018, China
| | - Zhigui Zuo
- Department of Colorectal Surgery, the First Affiliated Hospital of Wenzhou Medical University, Wenzhou, 325035, China
| | - Bo Sun
- Haixi Institutes, Chinese Academy of Sciences Quanzhou Institute of Equipment Manufacturing, Quanzhou, 362000, China
| | - Jianhua Zhang
- School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, 300384, China
| | - Weiguo Sheng
- School of Information Science and Technology, Hangzhou Normal University, Hangzhou, 311121, China
| | - Ran Guo
- Cyberspace Institute Advanced Technology, Guangzhou University, Guangzhou, 510006, China.
| | - Lejun Zhang
- Cyberspace Institute Advanced Technology, Guangzhou University, Guangzhou, 510006, China; College of Information Engineering, Yangzhou University, Yangzhou, 225127, China; Research and Development Center for E-Learning, Ministry of Education, Beijing, 100039, China
| | - Xiaozhen Hua
- Department of Pediatrics, Cangnan Affiliated Hospital of Wenzhou Medical University, Wenzhou, 325800, China.
| |
Collapse
|
2
|
Li W, Yin Ng W, Zhang X, Huang Y, Li Y, Song C, Chiu PWY, Li Z. A Kinematic Modeling and Control Scheme for Different Robotic Endoscopes: A Rudimentary Research Prototype. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3186758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Weibing Li
- School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
| | - Wing Yin Ng
- Department of Surgery and the Chow Yuk Ho Technology Centre for Innovative Medicine, The Chinese University of Hong Kong, Hong Kong, China
| | - Xue Zhang
- Department of Surgery and the Chow Yuk Ho Technology Centre for Innovative Medicine, The Chinese University of Hong Kong, Hong Kong, China
| | - Yisen Huang
- Department of Surgery and the Chow Yuk Ho Technology Centre for Innovative Medicine, The Chinese University of Hong Kong, Hong Kong, China
| | - Yehui Li
- Department of Surgery and the Chow Yuk Ho Technology Centre for Innovative Medicine, The Chinese University of Hong Kong, Hong Kong, China
| | - Chengzhi Song
- Shenzhen Cornerstone Technology Co., Ltd., Shenzhen, China
| | - Philip Wai-Yan Chiu
- Department of Surgery and the Chow Yuk Ho Technology Centre for Innovative Medicine, The Chinese University of Hong Kong, Hong Kong, China
| | - Zheng Li
- Department of Surgery, Chow Yuk Ho Technology Centre for Innovative Medicine, Li Ka Shing Institute of Health Science, and Multi-Scale Medical Robotics Centre, The Chinese University of Hong Kong, Hong Kong, China
| |
Collapse
|