1
|
Shao S, Pei Z, Chen W, Zhu W, Wu X, Sun D, Zhang B. Self-Supervised monocular depth and ego-Motion estimation in endoscopy: Appearance flow to the rescue. Med Image Anal 2021; 77:102338. [PMID: 35016079 DOI: 10.1016/j.media.2021.102338] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2021] [Revised: 10/24/2021] [Accepted: 12/14/2021] [Indexed: 11/25/2022]
Abstract
Recently, self-supervised learning technology has been applied to calculate depth and ego-motion from monocular videos, achieving remarkable performance in autonomous driving scenarios. One widely adopted assumption of depth and ego-motion self-supervised learning is that the image brightness remains constant within nearby frames. Unfortunately, the endoscopic scene does not meet this assumption because there are severe brightness fluctuations induced by illumination variations, non-Lambertian reflections and interreflections during data collection, and these brightness fluctuations inevitably deteriorate the depth and ego-motion estimation accuracy. In this work, we introduce a novel concept referred to as appearance flow to address the brightness inconsistency problem. The appearance flow takes into consideration any variations in the brightness pattern and enables us to develop a generalized dynamic image constraint. Furthermore, we build a unified self-supervised framework to estimate monocular depth and ego-motion simultaneously in endoscopic scenes, which comprises a structure module, a motion module, an appearance module and a correspondence module, to accurately reconstruct the appearance and calibrate the image brightness. Extensive experiments are conducted on the SCARED dataset and EndoSLAM dataset, and the proposed unified framework exceeds other self-supervised approaches by a large margin. To validate our framework's generalization ability on different patients and cameras, we train our model on SCARED but test it on the SERV-CT and Hamlyn datasets without any fine-tuning, and the superior results reveal its strong generalization ability. Code is available at: https://github.com/ShuweiShao/AF-SfMLearner.
Collapse
Affiliation(s)
- Shuwei Shao
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China
| | - Zhongcai Pei
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Weihai Chen
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, China.
| | | | - Xingming Wu
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China
| | - Dianmin Sun
- Shandong Cancer Hospital Affiliated to Shandong University, Shandong First Medical University and Shandong Academy of Medical Sciences, Jinan, China
| | - Baochang Zhang
- Institute of Artificial Intelligence, Beihang University, Beijing, China.
| |
Collapse
|
2
|
Lerner U, Yacobi T, Levy I, Moltchanov SA, Cole-Hunter T, Fishbain B. The effect of ego-motion on environmental monitoring. Sci Total Environ 2015; 533:8-16. [PMID: 26150302 DOI: 10.1016/j.scitotenv.2015.06.066] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/30/2015] [Accepted: 06/17/2015] [Indexed: 06/04/2023]
Abstract
Air pollution has a proven impact on public health. Currently, pollutant levels are obtained by high-priced, sizeable, stationary Air Quality Monitoring (AQM) stations. Recent developments in sensory and communication technologies have made relatively low-cost, micro-sensing units (MSUs) feasible. Their lower power consumption and small size enable mobile sensing, deploying single or multiple units simultaneously. Recent studies have reported on measurements acquired by mobile MSUs, mounted on cars, bicycles and pedestrians. While these modes of transportation inherently present different velocity and acceleration regimes, the effect of the sensors' varying movement characteristics have not been previously accounted for. This research assesses the impact of sensor's motion on its functionality through laboratory measurements and a field campaign. The laboratory setup consists of a wind tunnel to assess the effect of air flow on the measurements of nitrogen dioxide and ozone at different velocities in a controlled environment, while the field campaign is based on three cars mounted with MSUs, measuring pollutants and environmental variables at different traveling speeds. In both experimental designs we can regard the MSUs as a moving object in the environment, i.e. having a distinct ego-motion. The results show that MSU's behavior is highly affected by variation in speed and sensor placement with respect to direction of movement, mainly due to the physical properties of installed sensors. This strongly suggests that any future design of MSU must account for the speed effect from the design stage all the way through deployment and results analysis. This is the first report examining the influence of airflow variations on MSU's ability to accurately measure pollutant levels.
Collapse
Affiliation(s)
- Uri Lerner
- Technion Enviromatics Lab. (TechEL), Dept. of Environmental, Water and Agricultural Engineering, Faculty of Civil & Environmental Engineering, The Technion - Israel Institute of Technology, Israel; Technion Center of Excellence in Exposure Science and Environmental Health (TCEEH), Dept. of Environmental, Water and Agricultural Engineering, Faculty of Civil & Environmental Engineering, The Technion - Israel Institute of Technology, Israel
| | - Tamar Yacobi
- Technion Enviromatics Lab. (TechEL), Dept. of Environmental, Water and Agricultural Engineering, Faculty of Civil & Environmental Engineering, The Technion - Israel Institute of Technology, Israel; Technion Center of Excellence in Exposure Science and Environmental Health (TCEEH), Dept. of Environmental, Water and Agricultural Engineering, Faculty of Civil & Environmental Engineering, The Technion - Israel Institute of Technology, Israel
| | - Ilan Levy
- Technion Center of Excellence in Exposure Science and Environmental Health (TCEEH), Dept. of Environmental, Water and Agricultural Engineering, Faculty of Civil & Environmental Engineering, The Technion - Israel Institute of Technology, Israel
| | - Sharon A Moltchanov
- Technion Enviromatics Lab. (TechEL), Dept. of Environmental, Water and Agricultural Engineering, Faculty of Civil & Environmental Engineering, The Technion - Israel Institute of Technology, Israel; Technion Center of Excellence in Exposure Science and Environmental Health (TCEEH), Dept. of Environmental, Water and Agricultural Engineering, Faculty of Civil & Environmental Engineering, The Technion - Israel Institute of Technology, Israel
| | - Tom Cole-Hunter
- Centre for Research in Environmental Epidemiology (CREAL), Barcelona, Spain; Centro de Investigación Biomédica en Red de Epidemiología y Salud Pública (CIBERESP), Madrid, Spain; Universitat Pompeu Fabra (UPF), Barcelona, Spain
| | - Barak Fishbain
- Technion Enviromatics Lab. (TechEL), Dept. of Environmental, Water and Agricultural Engineering, Faculty of Civil & Environmental Engineering, The Technion - Israel Institute of Technology, Israel; Technion Center of Excellence in Exposure Science and Environmental Health (TCEEH), Dept. of Environmental, Water and Agricultural Engineering, Faculty of Civil & Environmental Engineering, The Technion - Israel Institute of Technology, Israel.
| |
Collapse
|