1
|
Toft C, Maddern W, Torii A, Hammarstrand L, Stenborg E, Safari D, Okutomi M, Pollefeys M, Sivic J, Pajdla T, Kahl F, Sattler T. Long-Term Visual Localization Revisited. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:2074-2088. [PMID: 33074802 DOI: 10.1109/tpami.2020.3032010] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Visual localization enables autonomous vehicles to navigate in their surroundings and augmented reality applications to link virtual to real worlds. Practical visual localization approaches need to be robust to a wide variety of viewing conditions, including day-night changes, as well as weather and seasonal variations, while providing highly accurate six degree-of-freedom (6DOF) camera pose estimates. In this paper, we extend three publicly available datasets containing images captured under a wide variety of viewing conditions, but lacking camera pose information, with ground truth pose information, making evaluation of the impact of various factors on 6DOF camera pose estimation accuracy possible. We also discuss the performance of state-of-the-art localization approaches on these datasets. Additionally, we release around half of the poses for all conditions, and keep the remaining half private as a test set, in the hopes that this will stimulate research on long-term visual localization, learned local image features, and related research areas. Our datasets are available at visuallocalization.net, where we are also hosting a benchmarking server for automatic evaluation of results on the test set. The presented state-of-the-art results are to a large degree based on submissions to our server.
Collapse
|
2
|
Xu M, Fischer T, Sunderhauf N, Milford M. Probabilistic Appearance-Invariant Topometric Localization With New Place Awareness. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3096745] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
3
|
Jaenal A, Moreno FA, Gonzalez-Jimenez J. Appearance-Based Sequential Robot Localization Using a Patchwise Approximation of a Descriptor Manifold. SENSORS (BASEL, SWITZERLAND) 2021; 21:2483. [PMID: 33918493 PMCID: PMC8038242 DOI: 10.3390/s21072483] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Revised: 03/26/2021] [Accepted: 03/29/2021] [Indexed: 11/16/2022]
Abstract
This paper addresses appearance-based robot localization in 2D with a sparse, lightweight map of the environment composed of descriptor-pose image pairs. Based on previous research in the field, we assume that image descriptors are samples of a low-dimensional Descriptor Manifold that is locally articulated by the camera pose. We propose a piecewise approximation of the geometry of such Descriptor Manifold through a tessellation of so-called Patches of Smooth Appearance Change (PSACs), which defines our appearance map. Upon this map, the presented robot localization method applies both a Gaussian Process Particle Filter (GPPF) to perform camera tracking and a Place Recognition (PR) technique for relocalization within the most likely PSACs according to the observed descriptor. A specific Gaussian Process (GP) is trained for each PSAC to regress a Gaussian distribution over the descriptor for any particle pose lying within that PSAC. The evaluation of the observed descriptor in this distribution gives us a likelihood, which is used as the weight for the particle. Besides, we model the impact of appearance variations on image descriptors as a white noise distribution within the GP formulation, ensuring adequate operation under lighting and scene appearance changes with respect to the conditions in which the map was constructed. A series of experiments with both real and synthetic images show that our method outperforms state-of-the-art appearance-based localization methods in terms of robustness and accuracy, with median errors below 0.3 m and 6°.
Collapse
Affiliation(s)
| | - Francisco-Angel Moreno
- Machine Perception and Intelligent Robotics Group (MAPIR), Department of System Engineering and Automation Biomedical Research Institute of Malaga (IBIMA), University of Malaga, 29071 Málaga, Spain; (A.J.); (J.G.-J.)
| | | |
Collapse
|
4
|
Xu M, Snderhauf N, Milford M. Probabilistic Visual Place Recognition for Hierarchical Localization. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2020.3040134] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
5
|
Molloy TL, Fischer T, Milford M, Nair GN. Intelligent Reference Curation for Visual Place Recognition Via Bayesian Selective Fusion. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2020.3047791] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
6
|
Wang Z, Peng Z, Guan Y, Wu L. Two-Stage vSLAM Loop Closure Detection Based on Sequence Node Matching and Semi-Semantic Autoencoder. J INTELL ROBOT SYST 2021. [DOI: 10.1007/s10846-020-01302-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
|
7
|
Learning second-order statistics for place recognition based on robust covariance estimation of CNN features. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2020.02.001] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
8
|
Zaffar M, Ehsan S, Milford M, McDonald-Maier K. CoHOG: A Light-Weight, Compute-Efficient, and Training-Free Visual Place Recognition Technique for Changing Environments. IEEE Robot Autom Lett 2020. [DOI: 10.1109/lra.2020.2969917] [Citation(s) in RCA: 31] [Impact Index Per Article: 7.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
9
|
Yu F, Shang J, Hu Y, Milford M. NeuroSLAM: a brain-inspired SLAM system for 3D environments. BIOLOGICAL CYBERNETICS 2019; 113:515-545. [PMID: 31571007 DOI: 10.1007/s00422-019-00806-9] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/29/2019] [Accepted: 09/14/2019] [Indexed: 06/10/2023]
Abstract
Roboticists have long drawn inspiration from nature to develop navigation and simultaneous localization and mapping (SLAM) systems such as RatSLAM. Animals such as birds and bats possess superlative navigation capabilities, robustly navigating over large, three-dimensional environments, leveraging an internal neural representation of space combined with external sensory cues and self-motion cues. This paper presents a novel neuro-inspired 4DoF (degrees of freedom) SLAM system named NeuroSLAM, based upon computational models of 3D grid cells and multilayered head direction cells, integrated with a vision system that provides external visual cues and self-motion cues. NeuroSLAM's neural network activity drives the creation of a multilayered graphical experience map in a real time, enabling relocalization and loop closure through sequences of familiar local visual cues. A multilayered experience map relaxation algorithm is used to correct cumulative errors in path integration after loop closure. Using both synthetic and real-world datasets comprising complex, multilayered indoor and outdoor environments, we demonstrate NeuroSLAM consistently producing topologically correct three-dimensional maps.
Collapse
Affiliation(s)
- Fangwen Yu
- Faculty of Information Engineering, China University of Geosciences and National Engineering Research Center for Geographic Information System, Wuhan, 430074, China
- Science and Engineering Faculty, Queensland University of Technology and Australian Centre for Robotic Vision, Brisbane, QLD, 4000, Australia
| | - Jianga Shang
- Faculty of Information Engineering, China University of Geosciences and National Engineering Research Center for Geographic Information System, Wuhan, 430074, China.
| | - Youjian Hu
- Faculty of Information Engineering, China University of Geosciences and National Engineering Research Center for Geographic Information System, Wuhan, 430074, China
| | - Michael Milford
- Science and Engineering Faculty, Queensland University of Technology and Australian Centre for Robotic Vision, Brisbane, QLD, 4000, Australia
| |
Collapse
|
10
|
Affiliation(s)
| | - Shaojie Shen
- Department of Electronic and Computer EngineeringThe Hong Kong University of Science and TechnologyHong Kong China
| |
Collapse
|
11
|
Loop Closure Detection Based on Multi-Scale Deep Feature Fusion. APPLIED SCIENCES-BASEL 2019. [DOI: 10.3390/app9061120] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Loop closure detection plays a very important role in the mobile robot navigation field. It is useful in achieving accurate navigation in complex environments and reducing the cumulative error of the robot’s pose estimation. The current mainstream methods are based on the visual bag of word model, but traditional image features are sensitive to illumination changes. This paper proposes a loop closure detection algorithm based on multi-scale deep feature fusion, which uses a Convolutional Neural Network (CNN) to extract more advanced and more abstract features. In order to deal with the different sizes of input images and enrich receptive fields of the feature extractor, this paper uses the spatial pyramid pooling (SPP) of multi-scale to fuse the features. In addition, considering the different contributions of each feature to loop closure detection, the paper defines the distinguishability weight of features and uses it in similarity measurement. It reduces the probability of false positives in loop closure detection. The experimental results show that the loop closure detection algorithm based on multi-scale deep feature fusion has higher precision and recall rates and is more robust to illumination changes than the mainstream methods.
Collapse
|
12
|
Santos MM, Zaffari GB, Ribeiro POCS, Drews-Jr PLJ, Botelho SSC. Underwater place recognition using forward-looking sonar images: A topological approach. J FIELD ROBOT 2018. [DOI: 10.1002/rob.21822] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Matheus M. Santos
- Center of Computer Science - C3, NAUTEC - Automation and Robotics Intelligent Group; Universidade Federal do Rio Grande - FURG; Rio Grande Brazil
| | - Guilherme B. Zaffari
- Center of Computer Science - C3, NAUTEC - Automation and Robotics Intelligent Group; Universidade Federal do Rio Grande - FURG; Rio Grande Brazil
| | - Pedro O. C. S. Ribeiro
- Center of Computer Science - C3, NAUTEC - Automation and Robotics Intelligent Group; Universidade Federal do Rio Grande - FURG; Rio Grande Brazil
| | - Paulo L. J. Drews-Jr
- Center of Computer Science - C3, NAUTEC - Automation and Robotics Intelligent Group; Universidade Federal do Rio Grande - FURG; Rio Grande Brazil
| | - Silvia S. C. Botelho
- Center of Computer Science - C3, NAUTEC - Automation and Robotics Intelligent Group; Universidade Federal do Rio Grande - FURG; Rio Grande Brazil
| |
Collapse
|
13
|
|
14
|
|
15
|
Lowry S, Sunderhauf N, Newman P, Leonard JJ, Cox D, Corke P, Milford MJ. Visual Place Recognition: A Survey. IEEE T ROBOT 2016. [DOI: 10.1109/tro.2015.2496823] [Citation(s) in RCA: 531] [Impact Index Per Article: 66.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
16
|
Majdik AL, Verda D, Albers-Schoenberg Y, Scaramuzza D. Air-ground Matching: Appearance-based GPS-denied Urban Localization of Micro Aerial Vehicles. J FIELD ROBOT 2015. [DOI: 10.1002/rob.21585] [Citation(s) in RCA: 42] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Affiliation(s)
- András L. Majdik
- Department of Informatics; University of Zurich; Zurich Switzerland
| | - Damiano Verda
- Italian National Council of Research; CNR-IEIIT Genova Italy
| | | | | |
Collapse
|
17
|
Abstract
This paper deals with the task of appearance-based mapping and place recognition. Previously, the scope of a location generally varied between either using discrete poses or loosely defined sequences of poses, facing problems related to perceptual aliasing and path invariance respectively. Here, we present a unified framework for defining, modelling and recognizing places in a way which is directly related to the underlying structure of features in the environment. A covisibility map of the environment is incrementally maintained over time, where visual landmarks represent nodes in a graph which are connected if seen together. When queried, relevant places are retrieved as clusters from this map, and a novel probabilistic observation model is used to evaluate place recognition. Place retrieval is able to adapt to a given query and also inherently cope with trajectory variations, due to the use of the landmark covisibility structure. In addition, the chosen generative model is developed in a way which is robust to observation errors, mapping errors, perceptual aliasing, and parameter sensitivity. Validation is provided through a variety of tests using real-world datasets, which compare the behaviour of the proposed approach to other representative state-of-the-art methods (namely FAB-MAP and SeqSLAM).
Collapse
Affiliation(s)
- Elena S. Stumm
- CNRS, LAAS, Toulouse, France
- Université de Toulouse, UPS, LAAS, Toulouse, France
| | - Christopher Mei
- CNRS, LAAS, Toulouse, France
- Université de Toulouse, LAAS, Toulouse, France
| | - Simon Lacroix
- CNRS, LAAS, Toulouse, France
- Université de Toulouse, LAAS, Toulouse, France
| |
Collapse
|
18
|
Siagian C, Chang CK, Itti L. Autonomous Mobile Robot Localization and Navigation Using a Hierarchical Map Representation Primarily Guided by Vision. J FIELD ROBOT 2014. [DOI: 10.1002/rob.21505] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Christian Siagian
- Department of Computer Science; University of Southern California; Los Angeles California 90089
| | - Chin Kai Chang
- Department of Computer Science; University of Southern California; Los Angeles California 90089
| | - Laurent Itti
- Department of Computer Science; University of Southern California; Los Angeles California 90089
| |
Collapse
|
19
|
Jacobson A, Chen Z, Milford M. Autonomous Multisensor Calibration and Closed-loop Fusion for SLAM. J FIELD ROBOT 2014. [DOI: 10.1002/rob.21500] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Affiliation(s)
- Adam Jacobson
- School of Electrical Engineering and Computer Science; Queensland University of Technology; Brisbane Queensland Australia 4000
| | - Zetao Chen
- School of Electrical Engineering and Computer Science; Queensland University of Technology; Brisbane Queensland Australia 4000
| | - Michael Milford
- School of Electrical Engineering and Computer Science; Queensland University of Technology; Brisbane Queensland Australia 4000
| |
Collapse
|
20
|
Martinez-Gomez J, Fernandez-Caballero A, Garcia-Varea I, Rodriguez L, Romero-Gonzalez C. A Taxonomy of Vision Systems for Ground Mobile Robots. INT J ADV ROBOT SYST 2014. [DOI: 10.5772/58900] [Citation(s) in RCA: 24] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022] Open
Abstract
This paper introduces a taxonomy of vision systems for ground mobile robots. In the last five years, a significant number of relevant papers have contributed to this subject. Firstly, a thorough review of the papers is proposed to discuss and classify both past and the most current approaches in the field. As a result, a global picture of the state of the art of the last five years is obtained. Moreover, the study of the articles is used to put forward a comprehensive taxonomy based on the most up-to-date research in ground mobile robotics. In this sense, the paper aims at being especially helpful to both budding and experienced researchers in the areas of vision systems and mobile ground robots. The taxonomy described is devised from a novel perspective, namely in order to respond to the main questions posed when designing robotic vision systems: why?, what for?, what with?, how?, and where? The answers are derived from the most relevant techniques described in the recent literature, leading in a natural way to a series of classifications that are discussed and contextualized. The article offers a global picture of the state of the art in the area and discovers some promising research lines.
Collapse
Affiliation(s)
- Jesus Martinez-Gomez
- Universidad de Castilla-La Mancha, Departamento de Sistemas Informaticos, Albacete, Spain
| | | | - Ismael Garcia-Varea
- Universidad de Castilla-La Mancha, Departamento de Sistemas Informaticos, Albacete, Spain
| | - Luis Rodriguez
- Universidad de Castilla-La Mancha, Departamento de Sistemas Informaticos, Albacete, Spain
| | | |
Collapse
|
21
|
Paul R, Newman P. Self-help: Seeking out perplexing images for ever improving topological mapping. Int J Rob Res 2013. [DOI: 10.1177/0278364913509859] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
In this work, we present a novel approach that allows a robot to improve its own navigation performance through introspection and then targeted data retrieval. It is a step in the direction of life-long learning and adaptation and is motivated by the desire to build robots that have plastic competencies which are not baked in. They should react to and benefit from use. We consider a particular instantiation of this problem in the context of place recognition. Based on a topic-based probabilistic representation for images, we use a measure of perplexity to evaluate how well a working set of background images explain the robot’s online view of the world. Offline, the robot then searches an external resource to seek out additional background images that bolster its ability to localize in its environment when used next. In this way the robot adapts and improves performance through use. We demonstrate this approach using data collected from a mobile robot operating in outdoor workspaces.
Collapse
Affiliation(s)
- Rohan Paul
- Oxford University Mobile Robotics Research Group, UK
| | - Paul Newman
- Oxford University Mobile Robotics Research Group, UK
| |
Collapse
|
22
|
Abstract
In this paper we use the algorithm SeqSLAM to address the question, how little and what quality of visual information is needed to localize along a familiar route? We conduct a comprehensive investigation of place recognition performance on seven datasets while varying image resolution (primarily 1 to 512 pixel images), pixel bit depth, field of view, motion blur, image compression and matching sequence length. Results confirm that place recognition using single images or short image sequences is poor, but improves to match or exceed current benchmarks as the matching sequence length increases. We then present place recognition results from two experiments where low-quality imagery is directly caused by sensor limitations; in one, place recognition is achieved along an unlit mountain road by using noisy, long-exposure blurred images, and in the other, two single pixel light sensors are used to localize in an indoor environment. We also show failure modes caused by pose variance and sequence aliasing, and discuss ways in which they may be overcome. By showing how place recognition along a route is feasible even with severely degraded image sequences, we hope to provoke a re-examination of how we develop and test future localization and mapping systems.
Collapse
Affiliation(s)
- Michael Milford
- Queensland University of Technology, Brisbane, Queensland, Australia
| |
Collapse
|
23
|
|
24
|
McManus C, Furgale P, Stenning B, Barfoot TD. Lighting‐invariant Visual Teach and Repeat Using Appearance‐based Lidar. J FIELD ROBOT 2012. [DOI: 10.1002/rob.21444] [Citation(s) in RCA: 32] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Affiliation(s)
| | - Paul Furgale
- Autonomous Systems LabETH Zürich Zürich Switzerland
| | - Braden Stenning
- Autonomous Space Robotics LabUniversity of Toronto Institute for Aerospace Studies Toronto Canada
| | - Timothy D. Barfoot
- Autonomous Space Robotics LabUniversity of Toronto Institute for Aerospace Studies Toronto Canada
| |
Collapse
|