1
|
Chen J, Li W, Hou P, Yang Z, Zhao H. An Appearance-Semantic Descriptor with Coarse-to-Fine Matching for Robust VPR. SENSORS (BASEL, SWITZERLAND) 2024; 24:2203. [PMID: 38610414 PMCID: PMC11014388 DOI: 10.3390/s24072203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/25/2024] [Revised: 03/23/2024] [Accepted: 03/26/2024] [Indexed: 04/14/2024]
Abstract
In recent years, semantic segmentation has made significant progress in visual place recognition (VPR) by using semantic information that is relatively invariant to appearance and viewpoint, demonstrating great potential. However, in some extreme scenarios, there may be semantic occlusion and semantic sparsity, which can lead to confusion when relying solely on semantic information for localization. Therefore, this paper proposes a novel VPR framework that employs a coarse-to-fine image matching strategy, combining semantic and appearance information to improve algorithm performance. First, we construct SemLook global descriptors using semantic contours, which can preliminarily screen images to enhance the accuracy and real-time performance of the algorithm. Based on this, we introduce SemLook local descriptors for fine screening, combining robust appearance information extracted by deep learning with semantic information. These local descriptors can address issues such as semantic overlap and sparsity in urban environments, further improving the accuracy of the algorithm. Through this refined screening process, we can effectively handle the challenges of complex image matching in urban environments and obtain more accurate results. The performance of SemLook descriptors is evaluated on three public datasets (Extended-CMU Season, Robot-Car Seasons v2, and SYNTHIA) and compared with six state-of-the-art VPR algorithms (HOG, CoHOG, AlexNet_VPR, Region VLAD, Patch-NetVLAD, Forest). In the experimental comparison, considering both real-time performance and evaluation metrics, the SemLook descriptors are found to outperform the other six algorithms. Evaluation metrics include the area under the curve (AUC) based on the precision-recall curve, Recall@100%Precision, and Precision@100%Recall. On the Extended-CMU Season dataset, SemLook descriptors achieve a 100% AUC value, and on the SYNTHIA dataset, they achieve a 99% AUC value, demonstrating outstanding performance. The experimental results indicate that introducing global descriptors for initial screening and utilizing local descriptors combining both semantic and appearance information for precise matching can effectively address the issue of location recognition in scenarios with semantic ambiguity or sparsity. This algorithm enhances descriptor performance, making it more accurate and robust in scenes with variations in appearance and viewpoint.
Collapse
Affiliation(s)
| | | | - Pengshuai Hou
- School of Mechanical Engineering and Automation, Northeastern University, Shenyang 110819, China; (J.C.); (W.L.); (Z.Y.); (H.Z.)
| | | | | |
Collapse
|
2
|
Arshad S, Park TH. SVS-VPR: A Semantic Visual and Spatial Information-Based Hierarchical Visual Place Recognition for Autonomous Navigation in Challenging Environmental Conditions. SENSORS (BASEL, SWITZERLAND) 2024; 24:906. [PMID: 38339624 PMCID: PMC10857550 DOI: 10.3390/s24030906] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Revised: 01/24/2024] [Accepted: 01/26/2024] [Indexed: 02/12/2024]
Abstract
Robust visual place recognition (VPR) enables mobile robots to identify previously visited locations. For this purpose, the extracted visual information and place matching method plays a significant role. In this paper, we critically review the existing VPR methods and group them into three major categories based on visual information used, i.e., handcrafted features, deep features, and semantics. Focusing the benefits of convolutional neural networks (CNNs) and semantics, and limitations of existing research, we propose a robust appearance-based place recognition method, termed SVS-VPR, which is implemented as a hierarchical model consisting of two major components: global scene-based and local feature-based matching. The global scene semantics are extracted and compared with pre-visited images to filter the match candidates while reducing the search space and computational cost. The local feature-based matching involves the extraction of robust local features from CNN possessing invariant properties against environmental conditions and a place matching method utilizing semantic, visual, and spatial information. SVS-VPR is evaluated on publicly available benchmark datasets using true positive detection rate, recall at 100% precision, and area under the curve. Experimental findings demonstrate that SVS-VPR surpasses several state-of-the-art deep learning-based methods, boosting robustness against significant changes in viewpoint and appearance while maintaining efficient matching time performance.
Collapse
Affiliation(s)
- Saba Arshad
- Industrial Artificial Intelligence Research Center, Chungbuk National University, Cheongju 28644, Republic of Korea;
| | - Tae-Hyoung Park
- Department of Intelligent Systems and Robotics, Chungbuk National University, Cheongju 28644, Republic of Korea
| |
Collapse
|
3
|
Gao G, Xiong Z, Zhao Y, Zhang L. Landmark Topology Descriptor-Based Place Recognition and Localization under Large View-Point Changes. SENSORS (BASEL, SWITZERLAND) 2023; 23:9775. [PMID: 38139621 PMCID: PMC10747459 DOI: 10.3390/s23249775] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Revised: 11/28/2023] [Accepted: 12/08/2023] [Indexed: 12/24/2023]
Abstract
Accurate localization between cameras is a prerequisite for a vision-based heterogeneous robot systems task. The core issue is how to accurately perform place recognition from different view-points. Traditional appearance-based methods have a high probability of failure in place recognition and localization under large view-point changes. In recent years, semantic graph matching-based place recognition methods have been proposed to solve the above problem. However, these methods rely on high-precision semantic segmentation results and have a high time complexity in node extraction or graph matching. In addition, methods only utilize the semantic labels of the landmarks themselves to construct graphs and descriptors, making such approaches fail in some challenging scenarios (e.g., scene repetition). In this paper, we propose a graph-matching method based on a novel landmark topology descriptor, which is robust to view-point changes. According to the experiment on real-world data, our algorithm can run in real-time and is approximately four times and three times faster than state-of-the-art algorithms in the graph extraction and matching phases, respectively. In terms of place recognition performance, our algorithm achieves the best place recognition precision at a recall of 0-70% compared with classic appearance-based algorithms and an advanced graph-based algorithm in the scene of significant view-point changes. In terms of positioning accuracy, compared to the traditional appearance-based DBoW2 and NetVLAD algorithms, our method outperforms by 95%, on average, in terms of the mean translation error and 95% in terms of the mean RMSE. Compared to the state-of-the-art SHM algorithm, our method outperforms by 30%, on average, in terms of the mean translation error and 29% in terms of the mean RMSE. In addition, our method outperforms the current state-of-the-art algorithm, even in challenging scenarios where the benchmark algorithms fail.
Collapse
Affiliation(s)
| | - Zhi Xiong
- College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China; (G.G.); (Y.Z.); (L.Z.)
| | | | | |
Collapse
|
4
|
A Review of Collaborative Air-Ground Robots Research. J INTELL ROBOT SYST 2022. [DOI: 10.1007/s10846-022-01756-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/31/2022]
|
5
|
Yue H, Miao J, Chen W, Wang W, Guo F, Li Z. Automatic vocabulary and graph verification for accurate loop closure detection. J FIELD ROBOT 2022. [DOI: 10.1002/rob.22088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Affiliation(s)
- Haosong Yue
- School of Automation Science and Electrical Engineering Beihang University Beijing 100191 China
| | - Jinyu Miao
- School of Automation Science and Electrical Engineering Beihang University Beijing 100191 China
| | - Weihai Chen
- School of Automation Science and Electrical Engineering Beihang University Beijing 100191 China
| | - Wei Wang
- Department of Computer Science University of Oxford Oxford UK
| | - Fanghong Guo
- Department of Automation Zhejiang University of Technology Hangzhou China
| | - Zhengguo Li
- Robotics Department Institute for Infocomm Research Singapore Singapore
| |
Collapse
|
6
|
Real-Time Artificial Intelligence Based Visual Simultaneous Localization and Mapping in Dynamic Environments – a Review. J INTELL ROBOT SYST 2022. [DOI: 10.1007/s10846-022-01643-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
|
7
|
Bhutta MUM, Sun Y, Lau D, Liu M. Why-So-Deep: Towards Boosting Previously Trained Models for Visual Place Recognition. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3142741] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
8
|
Wang X, Christie M, Marchand E. Binary Graph Descriptor for Robust Relocalization on Heterogeneous Data. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3142854] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
9
|
Deng Z, Zhang Y, Wu Y, Ge Z, Hu X, Sun W. Object-Plane Co-Represented and Graph Propagation-Based Semantic Descriptor for Relocalization. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3191056] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Zhiqiang Deng
- College of Information Science and Engineering, Northeastern University, Shenyang, China
| | - Yunzhou Zhang
- College of Information Science and Engineering, Northeastern University, Shenyang, China
| | - Yanmin Wu
- School of Electronic and Computer Engineering, Peking University, Shenzhen, China
| | - Zhexue Ge
- School of Intelligent Science and Technology, National University of Defense Technology, Hunan, Hunan
| | - Xinggang Hu
- College of Information Science and Engineering, Northeastern University, Shenyang, China
| | - Wenkai Sun
- College of Information Science and Engineering, Northeastern University, Shenyang, China
| |
Collapse
|
10
|
Keetha NV, Milford M, Garg S. A Hierarchical Dual Model of Environment- and Place-Specific Utility for Visual Place Recognition. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3096751] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
11
|
Guo X, Hu J, Chen J, Deng F, Lam TL. Semantic Histogram Based Graph Matching for Real-Time Multi-Robot Global Localization in Large Scale Environment. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3058935] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
12
|
SVG-Loop: Semantic–Visual–Geometric Information-Based Loop Closure Detection. REMOTE SENSING 2021. [DOI: 10.3390/rs13173520] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Loop closure detection is an important component of visual simultaneous localization and mapping (SLAM). However, most existing loop closure detection methods are vulnerable to complex environments and use limited information from images. As higher-level image information and multi-information fusion can improve the robustness of place recognition, a semantic–visual–geometric information-based loop closure detection algorithm (SVG-Loop) is proposed in this paper. In detail, to reduce the interference of dynamic features, a semantic bag-of-words model was firstly constructed by connecting visual features with semantic labels. Secondly, in order to improve detection robustness in different scenes, a semantic landmark vector model was designed by encoding the geometric relationship of the semantic graph. Finally, semantic, visual, and geometric information was integrated by fuse calculation of the two modules. Compared with art-of-the-state methods, experiments on the TUM RBG-D dataset, KITTI odometry dataset, and practical environment show that SVG-Loop has advantages in complex environments with varying light, changeable weather, and dynamic interference.
Collapse
|
13
|
Miller ID, Cowley A, Konkimalla R, Shivakumar SS, Nguyen T, Smith T, Taylor CJ, Kumar V. Any Way You Look at It: Semantic Crossview Localization and Mapping With LiDAR. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3061332] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
14
|
Nardari GV, Cohen A, Chen SW, Liu X, Arcot V, Romero RAF, Kumar V. Place Recognition in Forests With Urquhart Tessellations. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2020.3039217] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
15
|
Robust Loop Closure Detection Integrating Visual–Spatial–Semantic Information via Topological Graphs and CNN Features. REMOTE SENSING 2020. [DOI: 10.3390/rs12233890] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Loop closure detection is a key module for visual simultaneous localization and mapping (SLAM). Most previous methods for this module have not made full use of the information provided by images, i.e., they have only used the visual appearance or have only considered the spatial relationships of landmarks; the visual, spatial and semantic information have not been fully integrated. In this paper, a robust loop closure detection approach integrating visual–spatial–semantic information is proposed by employing topological graphs and convolutional neural network (CNN) features. Firstly, to reduce mismatches under different viewpoints, semantic topological graphs are introduced to encode the spatial relationships of landmarks, and random walk descriptors are employed to characterize the topological graphs for graph matching. Secondly, dynamic landmarks are eliminated by using semantic information, and distinctive landmarks are selected for loop closure detection, thus alleviating the impact of dynamic scenes. Finally, to ease the effect of appearance changes, the appearance-invariant descriptor of the landmark region is extracted by a pre-trained CNN without the specially designed manual features. The proposed approach weakens the influence of viewpoint changes and dynamic scenes, and extensive experiments conducted on open datasets and a mobile robot demonstrated that the proposed method has more satisfactory performance compared to state-of-the-art methods.
Collapse
|
16
|
Uygur I, Miyagusuku R, Pathak S, Moro A, Yamashita A, Asama H. Robust and Efficient Indoor Localization Using Sparse Semantic Information from a Spherical Camera. SENSORS 2020; 20:s20154128. [PMID: 32722263 PMCID: PMC7435920 DOI: 10.3390/s20154128] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/10/2020] [Revised: 07/18/2020] [Accepted: 07/21/2020] [Indexed: 11/16/2022]
Abstract
Self-localization enables a system to navigate and interact with its environment. In this study, we propose a novel sparse semantic self-localization approach for robust and efficient indoor localization. “Sparse semantic” refers to the detection of sparsely distributed objects such as doors and windows. We use sparse semantic information to self-localize on a human-readable 2D annotated map in the sensor model. Thus, compared to previous works using point clouds or other dense and large data structures, our work uses a small amount of sparse semantic information, which efficiently reduces uncertainty in real-time localization. Unlike complex 3D constructions, the annotated map required by our method can be easily prepared by marking the approximate centers of the annotated objects on a 2D map. Our approach is robust to the partial obstruction of views and geometrical errors on the map. The localization is performed using low-cost lightweight sensors, an inertial measurement unit and a spherical camera. We conducted experiments to show the feasibility and robustness of our approach.
Collapse
Affiliation(s)
- Irem Uygur
- Department of Precision Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan; (S.P.); (A.M.); (A.Y.); (H.A.)
- Correspondence:
| | - Renato Miyagusuku
- Department of Mechanical and Intelligent Engineering, Utsunomiya University, Utsunomiya 321-8585, Tochigi, Japan;
| | - Sarthak Pathak
- Department of Precision Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan; (S.P.); (A.M.); (A.Y.); (H.A.)
| | - Alessandro Moro
- Department of Precision Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan; (S.P.); (A.M.); (A.Y.); (H.A.)
| | - Atsushi Yamashita
- Department of Precision Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan; (S.P.); (A.M.); (A.Y.); (H.A.)
| | - Hajime Asama
- Department of Precision Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan; (S.P.); (A.M.); (A.Y.); (H.A.)
| |
Collapse
|
17
|
Xia L, Cui J, Shen R, Xu X, Gao Y, Li X. A survey of image semantics-based visual simultaneous localization and mapping: Application-oriented solutions to autonomous navigation of mobile robots. INT J ADV ROBOT SYST 2020. [DOI: 10.1177/1729881420919185] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
As one of the typical application-oriented solutions to robot autonomous navigation, visual simultaneous localization and mapping is essentially restricted to simplex environmental understanding based on geometric features of images. By contrast, the semantic simultaneous localization and mapping that is characterized by high-level environmental perception has apparently opened the door to apply image semantics to efficiently estimate poses, detect loop closures, build 3D maps, and so on. This article presents a detailed review of recent advances in semantic simultaneous localization and mapping, which mainly covers the treatments in terms of perception, robustness, and accuracy. Specifically, the concept of “semantic extractor” and the framework of “modern visual simultaneous localization and mapping” are initially presented. As the challenges associated with perception, robustness, and accuracy are being stated, we further discuss some open problems from a macroscopic view and attempt to find answers. We argue that multiscaled map representation, object simultaneous localization and mapping system, and deep neural network-based simultaneous localization and mapping pipeline design could be effective solutions to image semantics-fused visual simultaneous localization and mapping.
Collapse
Affiliation(s)
- Linlin Xia
- School of Automation Engineering, Northeast Electric Power University, Jilin, China
| | - Jiashuo Cui
- School of Automation Engineering, Northeast Electric Power University, Jilin, China
| | - Ran Shen
- School of Automation Engineering, Northeast Electric Power University, Jilin, China
| | - Xun Xu
- Institute for Superconducting and Electronic Materials, University of Wollongong, Wollongong, Australia
| | - Yiping Gao
- School of Automation Engineering, Northeast Electric Power University, Jilin, China
| | - Xinying Li
- School of Automation Engineering, Northeast Electric Power University, Jilin, China
| |
Collapse
|
18
|
|
19
|
Abstract
Multi-robot mapping and environment modeling have several advantages that makeit an attractive alternative to the mapping with a single robot: faster exploration, higherfault tolerance, richer data due to different sensors being used by different systems. However,the environment modeling with several robotic systems operating in the same area causes problemsof higher-order—acquired knowledge fusion and synchronization over time, revealing the sameenvironment properties using different sensors with different technical specifications. While theexisting robot map and environment model merging techniques allow merging certain homogeneousmaps, the possibility to use sensors of different physical nature and different mapping algorithms islimited. The resulting maps from robots with different specifications are heterogeneous, and eventhough some research on how to merge fundamentally different maps exists, it is limited to specificapplications. This research reviews the state of the art in homogeneous and heterogeneous mapmerging and illustrates the main research challenges in the area. Six factors are identified thatinfluence the outcome of map merging: (1) robotic platform hardware configurations, (2) maprepresentation types, (3) mapping algorithms, (4) shared information between robots, (5) relativepositioning information, (6) resulting global maps.
Collapse
|
20
|
Garg S, Suenderhauf N, Milford M. Semantic–geometric visual place recognition: a new perspective for reconciling opposing views. Int J Rob Res 2019. [DOI: 10.1177/0278364919839761] [Citation(s) in RCA: 28] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Affiliation(s)
- Sourav Garg
- Australian Centre for Robotic Vision, Queensland University of Technology, Brisbane, Australia
| | - Niko Suenderhauf
- Australian Centre for Robotic Vision, Queensland University of Technology, Brisbane, Australia
| | - Michael Milford
- Australian Centre for Robotic Vision, Queensland University of Technology, Brisbane, Australia
| |
Collapse
|
21
|
Guo J, Borges PVK, Park C, Gawel A. Local Descriptor for Robust Place Recognition Using LiDAR Intensity. IEEE Robot Autom Lett 2019. [DOI: 10.1109/lra.2019.2893887] [Citation(s) in RCA: 45] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
22
|
Bernreiter L, Gawel AR, Sommer H, Nieto J, Siegwart R, Cadena Lerma C. Multiple Hypothesis Semantic Mapping for Robust Data Association. IEEE Robot Autom Lett 2019. [DOI: 10.1109/lra.2019.2925756] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|