RGB-D Point Cloud Registration Based on Salient Object Detection

We propose a robust algorithm for aligning rigid, noisy, and partially overlapping red green blue-depth (RGB-D) point clouds. To address the problems of data degradation and uneven distribution, we offer three strategies to increase the robustness of the iterative closest point (ICP) algorithm. First, we introduce a salient object detection (SOD) method to extract a set of points with significant structural variation in the foreground, which can avoid the unbalanced proportion of foreground and background point sets leading to the local registration. Second, registration algorithms that rely only on structural information for alignment cannot establish the correct correspondences when faced with the point set with no significant change in structure. Therefore, a bidirectional color distance (BCD) is designed to build precise correspondence with bidirectional search and color guidance. Third, the maximum correntropy criterion (MCC) and trimmed strategy are introduced into our algorithm to handle with noise and outliers. We experimentally validate that our algorithm is more robust than previous algorithms on simulated and real-world scene data in most scenarios and achieve a satisfying 3-D reconstruction of indoor scenes.

LiDAR Odometry and Mapping Based on Semantic Information for Outdoor Environment

Simultaneous Localization and Mapping (SLAM) in an unknown environment is a crucial part for intelligent mobile robots to achieve high-level navigation and interaction tasks. As one of the typical LiDAR-based SLAM algorithms, the Lidar Odometry and Mapping in Real-time (LOAM) algorithm has shown impressive results. However, LOAM only uses low-level geometric features without considering semantic information. Moreover, the lack of a dynamic object removal strategy limits the algorithm to obtain higher accuracy. To this end, this paper extends the LOAM pipeline by integrating semantic information into the original framework. Specifically, we first propose a two-step dynamic objects filtering strategy. Point-wise semantic labels are then used to improve feature extraction and searching for corresponding points. We evaluate the performance of the proposed method in many challenging scenarios, including highway, country and urban from the KITTI dataset. The results demonstrate that the proposed SLAM system outperforms the state-of-the-art SLAM methods in terms of accuracy and robustness.

Large-Scale LiDAR SLAM with Factor Graph Optimization on High-Level Geometric Features

Although visual SLAM (simultaneous localization and mapping) methods obtain very accurate results using optimization of residual errors defined with respect to the matching features, the SLAM systems based on 3-D laser (LiDAR) data commonly employ variants of the iterative closest points algorithm and raw point clouds as the map representation. However, it is possible to extract from point clouds features that are more spatially extended and more meaningful than points: line segments and/or planar patches. In particular, such features provide a natural way to represent human-made environments, such as urban and mixed indoor/outdoor scenes. In this paper, we perform an analysis of the advantages of a LiDAR-based SLAM that employs high-level geometric features in large-scale urban environments. We present a new approach to the LiDAR SLAM that uses planar patches and line segments for map representation and employs factor graph optimization typical to state-of-the-art visual SLAM for the final map and trajectory optimization. The new map structure and matching of features make it possible to implement in our system an efficient loop closure method, which exploits learned descriptors for place recognition and factor graph for optimization. With these improvements, the overall software structure is based on the proven LOAM concept to ensure real-time operation. A series of experiments were performed to compare the proposed solution to the open-source LOAM, considering different approaches to loop closure computation. The results are compared using standard metrics of trajectory accuracy, focusing on the final quality of the estimated trajectory and the consistency of the environment map. With some well-discussed reservations, our results demonstrate the gains due to using the high-level features in the full-optimization approach in the large-scale LiDAR SLAM.

The UMA-SAR Dataset: Multimodal data collection from a ground vehicle during outdoor disaster response training exercises

This article presents a collection of multimodal raw data captured from a manned all-terrain vehicle in the course of two realistic outdoor search and rescue (SAR) exercises for actual emergency responders conducted in Málaga (Spain) in 2018 and 2019: the UMA-SAR dataset. The sensor suite, applicable to unmanned ground vehicles (UGVs), consisted of overlapping visible light (RGB) and thermal infrared (TIR) forward-looking monocular cameras, a Velodyne HDL-32 three-dimensional (3D) lidar, as well as an inertial measurement unit (IMU) and two global positioning system (GPS) receivers as ground truth. Our mission was to collect a wide range of data from the SAR domain, including persons, vehicles, debris, and SAR activity on unstructured terrain. In particular, four data sequences were collected following closed-loop routes during the exercises, with a total path length of 5.2 km and a total time of 77 min. In addition, we provide three more sequences of the empty site for comparison purposes (an extra 4.9 km and 46 min). Furthermore, the data is offered both in human-readable format and as rosbag files, and two specific software tools are provided for extracting and adapting this dataset to the users’ preference. The review of previously published disaster robotics repositories indicates that this dataset can contribute to fill a gap regarding visual and thermal datasets and can serve as a research tool for cross-cutting areas such as multispectral image fusion, machine learning for scene understanding, person and object detection, and localization and mapping in unstructured environments. The full dataset is publicly available at: www.uma.es/robotics-and-mechatronics/sar-datasets .

Polylidar3D-Fast Polygon Extraction from 3D Data

Flat surfaces captured by 3D point clouds are often used for localization, mapping, and modeling. Dense point cloud processing has high computation and memory costs making low-dimensional representations of flat surfaces such as polygons desirable. We present Polylidar3D, a non-convex polygon extraction algorithm which takes as input unorganized 3D point clouds (e.g., LiDAR data), organized point clouds (e.g., range images), or user-provided meshes. Non-convex polygons represent flat surfaces in an environment with interior cutouts representing obstacles or holes. The Polylidar3D front-end transforms input data into a half-edge triangular mesh. This representation provides a common level of abstraction for subsequent back-end processing. The Polylidar3D back-end is composed of four core algorithms: mesh smoothing, dominant plane normal estimation, planar segment extraction, and finally polygon extraction. Polylidar3D is shown to be quite fast, making use of CPU multi-threading and GPU acceleration when available. We demonstrate Polylidar3D's versatility and speed with real-world datasets including aerial LiDAR point clouds for rooftop mapping, autonomous driving LiDAR point clouds for road surface detection, and RGBD cameras for indoor floor/wall detection. We also evaluate Polylidar3D on a challenging planar segmentation benchmark dataset. Results consistently show excellent speed and accuracy.

Fast and Automatic Registration of Terrestrial Point Clouds Using 2D Line Features

Point cloud registration, as the first step for the use of point cloud data, has attracted increasing attention. In order to obtain the entire point cloud of a scene, the registration of point clouds from multiple views is necessary. In this paper, we propose an automatic method for the coarse registration of point clouds. The 2D lines are first extracted from the two point clouds being matched. Then, the line correspondences are established and the 2D transformation is calculated. Finally, a method is developed to calculate the displacement along the z-axis. With the 2D transformation and displacement, the 3D transformation can be easily achieved. Thus, the two point clouds are aligned together. The experimental results well demonstrate that our method can obtain high-precision registration results and is computationally very efficient. In the experimental results obtained by our method, the biggest rotation error is 0.5219o, and the biggest horizontal and vertical errors are 0.2319 m and 0.0119 m, respectively. The largest total computation time is only 713.4647 s.

Global Registration of Terrestrial Laser Scanner Point Clouds Using Plane-to-Plane Correspondences

Registration of point clouds is a central problem in many mapping and monitoring applications, such as outdoor and indoor mapping, high-speed railway track inspection, heritage documentation, building information modeling, and others. However, ensuring the global consistency of the registration is still a challenging task when there are multiple point clouds because the different scans should be transformed into a common coordinate frame. The aim of this paper is the registration of multiple terrestrial laser scanner point clouds. We present a plane-based matching algorithm to find plane-to-plane correspondences using a new parametrization based on complex numbers. The multiplication of complex numbers is based on analysis of the quadrants to avoid the ambiguity in the calculation of the rotation angle formed between normal vectors of adjacent planes. As a matching step may contain several matrix operations, our strategy is applied to reduce the number of mathematical operations. We also design a novel method for global refinement of terrestrial laser scanner data based on plane-to-plane correspondences. The rotation parameters are globally refined using operations of quaternion multiplication, while the translation parameters are refined using the parameters of planes. The global refinement is done non-iteratively. The experimental results show that the proposed plane-based matching algorithm efficiently finds plane correspondences in partial overlapping scans providing approximate values for the global registration, and indicate that an accuracy better than 8 cm can be achieved by using our global fine plane-to-plane registration method.

A Novel Method for Plane Extraction from Low-Resolution Inhomogeneous Point Clouds and its Application to a Customized Low-Cost Mobile Mapping System

Over the last decade, increasing demands for building interior mapping have brought the challenge of effectively and efficiently acquiring geometric information. Most mobile mapping methods rely on the integration of Simultaneous Localization And Mapping (SLAM) and costly Inertial Measurement Units (IMUs). Meanwhile, the methods also suffer misalignment errors caused by the low-resolution inhomogeneous point clouds captured using multi-line Mobile Laser Scanners (MLSs). While point-based alignments between such point clouds are affected by the highly dynamic moving patterns of the platform, plane-based methods are limited by the poor quality of the planes extracted, which reduce the methods’ robustness, reliability, and applicability. To alleviate these issues, we proposed and developed a method for plane extraction from low-resolution inhomogeneous point clouds. Based on the definition of virtual scanlines and the Enhanced Line Simplification (ELS) algorithm, the method extracts feature points, generates line segments, forms patches, and merges multi-direction fractions to form planes. The proposed method reduces the over-segmentation fractions caused by measurement noise and scanline curvature. A dedicated plane-to-plane point cloud alignment workflow based on the proposed plane extraction method was created to demonstrate the method’s application. The implementation of the coarse-to-fine procedure and the shortest-path initialization strategy eliminates the necessity of IMUs in mobile mapping. A mobile mapping prototype was designed to test the performance of the proposed methods. The results show that the proposed workflow and hardware system achieves centimeter-level accuracy, which suggests that it can be applied to mobile mapping and sensor fusion.

Heterogeneous Map Merging: State of the Art

Multi-robot mapping and environment modeling have several advantages that makeit an attractive alternative to the mapping with a single robot: faster exploration, higherfault tolerance, richer data due to different sensors being used by different systems. However,the environment modeling with several robotic systems operating in the same area causes problemsof higher-order—acquired knowledge fusion and synchronization over time, revealing the sameenvironment properties using different sensors with different technical specifications. While theexisting robot map and environment model merging techniques allow merging certain homogeneousmaps, the possibility to use sensors of different physical nature and different mapping algorithms islimited. The resulting maps from robots with different specifications are heterogeneous, and eventhough some research on how to merge fundamentally different maps exists, it is limited to specificapplications. This research reviews the state of the art in homogeneous and heterogeneous mapmerging and illustrates the main research challenges in the area. Six factors are identified thatinfluence the outcome of map merging: (1) robotic platform hardware configurations, (2) maprepresentation types, (3) mapping algorithms, (4) shared information between robots, (5) relativepositioning information, (6) resulting global maps.

The Accuracy Comparison of Three Simultaneous Localization and Mapping (SLAM)-Based Indoor Mapping Technologies

The growing interest and the market for indoor Location Based Service (LBS) have been drivers for a huge demand for building data and reconstructing and updating of indoor maps in recent years. The traditional static surveying and mapping methods can’t meet the requirements for accuracy, efficiency and productivity in a complicated indoor environment. Utilizing a Simultaneous Localization and Mapping (SLAM)-based mapping system with ranging and/or camera sensors providing point cloud data for the maps is an auspicious alternative to solve such challenges. There are various kinds of implementations with different sensors, for instance LiDAR, depth cameras, event cameras, etc. Due to the different budgets, the hardware investments and the accuracy requirements of indoor maps are diverse. However, limited studies on evaluation of these mapping systems are available to offer a guideline of appropriate hardware selection. In this paper we try to characterize them and provide some extensive references for SLAM or mapping system selection for different applications. Two different indoor scenes (a L shaped corridor and an open style library) were selected to review and compare three different mapping systems, namely: (1) a commercial Matterport system equipped with depth cameras; (2) SLAMMER: a high accuracy small footprint LiDAR with a fusion of hector-slam and graph-slam approaches; and (3) NAVIS: a low-cost large footprint LiDAR with Improved Maximum Likelihood Estimation (IMLE) algorithm developed by the Finnish Geospatial Research Institute (FGI). Firstly, an L shaped corridor (2nd floor of FGI) with approximately 80 m length was selected as the testing field for Matterport testing. Due to the lack of quantitative evaluation of Matterport indoor mapping performance, we attempted to characterize the pros and cons of the system by carrying out six field tests with different settings. The results showed that the mapping trajectory would influence the final mapping results and therefore, there was optimal Matterport configuration for better indoor mapping results. Secondly, a medium-size indoor environment (the FGI open library) was selected for evaluation of the mapping accuracy of these three indoor mapping technologies: SLAMMER, NAVIS and Matterport. Indoor referenced maps were collected with a small footprint Terrestrial Laser Scanner (TLS) and using spherical registration targets. The 2D indoor maps generated by these three mapping technologies were assessed by comparing them with the reference 2D map for accuracy evaluation; two feature selection methods were also utilized for the evaluation: interactive selection and minimum bounding rectangles (MBRs) selection. The mapping RMS errors of SLAMMER, NAVIS and Matterport were 2.0 cm, 3.9 cm and 4.4 cm, respectively, for the interactively selected features, and the corresponding values using MBR features were 1.7 cm, 3.2 cm and 4.7 cm. The corresponding detection rates for the feature points were 100%, 98.9%, 92.3% for the interactive selected features and 100%, 97.3% and 94.7% for the automated processing. The results indicated that the accuracy of all the evaluated systems could generate indoor map at centimeter-level, but also variation of the density and quality of collected point clouds determined the applicability of a system into a specific LBS.

Spectral 6DOF registration of noisy 3D range data with partial overlap

We present Spectral Registration with Multilayer Resampling (SRMR) as a 6 Degrees Of Freedom (DOF) registration method for noisy 3D data with partial overlap. The algorithm is based on decoupling 3D rotation from 3D translation by a corresponding resampling process of the spectral magnitude of a 3D Fast Fourier Transform (FFT) calculation on discretized 3D range data. The registration of all 6DOF is then subsequently carried out with spectral registrations using Phase Only Matched Filtering (POMF). There are two main aspects for the fast and robust registration of Euler angles from spherical information in SRMR. First of all, there is the permanent use of phase matching. Second, based on the FFT on a discrete Cartesian grid, not only one spherical layer but also a complete stack of layers are processed in one step. Experiments are presented with challenging datasets with respect to interference and overlap. The results include the fast and robust registration of artificially transformed data for ground-truth comparison, scans from the Stanford Bunny dataset, high end 3D laser range finder (LRF) scans of a city center, and range data from a low-cost actuated LRF in a disaster response scenario.

A Fast and Robust Feature-Based Scan-Matching Method in 3D SLAM and the Effect of Sampling Strategies

Simultaneous localization and mapping (SLAM) plays an important role in fully autonomous systems when a GNSS (global navigation satellite system) is not available. Studies in both 2D indoor and 3D outdoor SLAM are based on the appearance of environments and utilize scan-matching methods to find rigid body transformation parameters between two consecutive scans. In this study, a fast and robust scan-matching method based on feature extraction is introduced. Since the method is based on the matching of certain geometric structures, like plane segments, the outliers and noise in the point cloud are considerably eliminated. Therefore, the proposed scan-matching algorithm is more robust than conventional methods. Besides, the registration time and the number of iterations are significantly reduced, since the number of matching points is efficiently decreased. As a scan-matching framework, an improved version of the normal distribution transform (NDT) is used. The probability density functions (PDFs) of the reference scan are generated as in the traditional NDT, and the feature extraction - based on stochastic plane detection - is applied to the only input scan. By using experimental dataset belongs to an outdoor environment like a university campus, we obtained satisfactory performance results. Moreover, the feature extraction part of the algorithm is considered as a special sampling strategy for scan-matching and compared to other sampling strategies, such as random sampling and grid-based sampling, the latter of which is first used in the NDT. Thus, this study also shows the effect of the subsampling on the performance of the NDT.

Simultaneous localization and mapping with multimodal probability distributions

Simultaneous Localization and Mapping (SLAM) has focused on noisy but unique data associations resulting in linear Gaussian uncertainty models. However, a unique decision is often not possible using only local information, giving rise to ambiguities that have to be resolved globally during optimization. To solve this problem, the pose graph data structure is extended here by multimodal constraints modeled by mixtures of Gaussians (MoG). Furthermore, optimization methods for this novel formulation are introduced, namely (a) robust iteratively reweighted least squares, and (b) Prefilter Stochastic Gradient Descent (SGD) where a preprocessing step determines globally consistent modes before applying SGD. In addition, a variant of the Prefilter method (b) is introduced in form of (c) Prefilter Levenberg–Marquardt. The methods are compared with traditional state-of-the-art optimization methods including (d) Stochastic Gradient Descent and (e) Levenberg–Marquardt as well as (f) Particle filter SLAM and with (g) an optimal exhaustive algorithm. Experiments show that ambiguities significantly impact state-of-the-art methods, and that the novel Prefilter methods (b) and (c) perform best. This is further substantiated with experiments using real-world data. To this end, a method to generate MoG constraints from a plane-based registration algorithm is introduced and used for 3D SLAM under ambiguities.