Template-Guided Hierarchical Multi-View Registration Framework of Unordered Bridge Terrestrial Laser Scanning Data

The registration of bridge point cloud data (PCD) is an important preprocessing step for tasks such as bridge modeling, deformation detection, and bridge health monitoring. However, most existing research on bridge PCD registration only focused on pairwise registration, and payed insufficient attention to multi-view registration. In addition, to recover the overlaps of unordered multiple scans and obtain the merging order, extensive pairwise matching and the creation of a fully connected graph of all scans are often required, resulting in low efficiency. To address these issues, this paper proposes a marker-free template-guided method to align multiple unordered bridge PCD to a global coordinate system. Firstly, by aligning each scan to a given registration template, the overlaps between all the scans are recovered. Secondly, a fully connected graph is created based on the overlaps and scanning locations, and then a graph-partition algorithm is utilized to construct the scan-blocks. Then, the coarse-to-fine registration is performed within each scan-block, and the transformation matrix of coarse registration is obtained using an intelligent optimization algorithm. Finally, global block-to-block registration is performed to align all scans to a unified coordinate reference system. We tested our framework on different bridge point cloud datasets, including a suspension bridge and a continuous rigid frame bridge, to evaluate its accuracy. Experimental results demonstrate that our method has high accuracy.

From SLAM to Situational Awareness: Challenges and Survey

The capability of a mobile robot to efficiently and safely perform complex missions is limited by its knowledge of the environment, namely the situation. Advanced reasoning, decision-making, and execution skills enable an intelligent agent to act autonomously in unknown environments. Situational Awareness (SA) is a fundamental capability of humans that has been deeply studied in various fields, such as psychology, military, aerospace, and education. Nevertheless, it has yet to be considered in robotics, which has focused on single compartmentalized concepts such as sensing, spatial perception, sensor fusion, state estimation, and Simultaneous Localization and Mapping (SLAM). Hence, the present research aims to connect the broad multidisciplinary existing knowledge to pave the way for a complete SA system for mobile robotics that we deem paramount for autonomy. To this aim, we define the principal components to structure a robotic SA and their area of competence. Accordingly, this paper investigates each aspect of SA, surveying the state-of-the-art robotics algorithms that cover them, and discusses their current limitations. Remarkably, essential aspects of SA are still immature since the current algorithmic development restricts their performance to only specific environments. Nevertheless, Artificial Intelligence (AI), particularly Deep Learning (DL), has brought new methods to bridge the gap that maintains these fields apart from the deployment to real-world scenarios. Furthermore, an opportunity has been discovered to interconnect the vastly fragmented space of robotic comprehension algorithms through the mechanism of Situational Graph (S-Graph), a generalization of the well-known scene graph. Therefore, we finally shape our vision for the future of robotic situational awareness by discussing interesting recent research directions.

A Review on Visual-SLAM: Advancements from Geometric Modelling to Learning-Based Semantic Scene Understanding Using Multi-Modal Sensor Fusion

Simultaneous Localisation and Mapping (SLAM) is one of the fundamental problems in autonomous mobile robots where a robot needs to reconstruct a previously unseen environment while simultaneously localising itself with respect to the map. In particular, Visual-SLAM uses various sensors from the mobile robot for collecting and sensing a representation of the map. Traditionally, geometric model-based techniques were used to tackle the SLAM problem, which tends to be error-prone under challenging environments. Recent advancements in computer vision, such as deep learning techniques, have provided a data-driven approach to tackle the Visual-SLAM problem. This review summarises recent advancements in the Visual-SLAM domain using various learning-based methods. We begin by providing a concise overview of the geometric model-based approaches, followed by technical reviews on the current paradigms in SLAM. Then, we present the various learning-based approaches to collecting sensory inputs from mobile robots and performing scene understanding. The current paradigms in deep-learning-based semantic understanding are discussed and placed under the context of Visual-SLAM. Finally, we discuss challenges and further opportunities in the direction of learning-based approaches in Visual-SLAM.

The Deep Convolutional Neural Network Role in the Autonomous Navigation of Mobile Robots (SROBO)

The ability to navigate unstructured environments is an essential task for intelligent systems. Autonomous navigation by ground vehicles requires developing an internal representation of space, trained by recognizable landmarks, robust visual processing, computer vision and image processing. A mobile robot needs a platform enabling it to operate in an environment autonomously, recognize the objects, and avoid obstacles in its path. In this study, an open-source ground robot called SROBO was designed to accurately identify its position and navigate certain areas using a deep convolutional neural network and transfer learning. The framework uses an RGB-D MYNTEYE camera, a 2D laser scanner and inertial measurement units (IMU) operating through an embedded system capable of deep learning. The real-time decision-making process and experiments were conducted while the onboard signal processing and image capturing system enabled continuous information analysis. State-of-the-art Real-Time Graph-Based SLAM (RTAB-Map) was adopted to create a map of indoor environments while benefiting from deep convolutional neural network (Deep-CNN) capability. Enforcing Deep-CNN improved the performance quality of the RTAB-Map SLAM. The proposed setting equipped the robot with more insight into its surroundings. The robustness of the SROBO increased by 35% using the proposed system compared to the conventional RTAB-Map SLAM.

Real-Time Lidar Odometry and Mapping with Loop Closure

Real-time performance and global consistency are extremely important in Simultaneous Localization and Mapping (SLAM) problems. Classic lidar-based SLAM systems often consist of front-end odometry and back-end pose optimization. However, due to expensive computation, it is often difficult to achieve loop-closure detection without compromising the real-time performance of the odometry. We propose a SLAM system where scan-to-submap-based local lidar odometry and global pose optimization based on submap construction as well as loop-closure detection are designed as separated from each other. In our work, extracted edge and surface feature points are inserted into two consecutive feature submaps and added to the pose graph prepared for loop-closure detection and global pose optimization. In addition, a submap is added to the pose graph for global data association when it is marked as in a finished state. In particular, a method to filter out false loops is proposed to accelerate the construction of constraints in the pose graph. The proposed method is evaluated on public datasets and achieves competitive performance with pose estimation frequency over 15 Hz in local lidar odometry and low drift in global consistency.

Multi-LiDAR Mapping for Scene Segmentation in Indoor Environments for Mobile Robots

Nowadays, most mobile robot applications use two-dimensional LiDAR for indoor mapping, navigation, and low-level scene segmentation. However, single data type maps are not enough in a six degree of freedom world. Multi-LiDAR sensor fusion increments the capability of robots to map on different levels the surrounding environment. It exploits the benefits of several data types, counteracting the cons of each of the sensors. This research introduces several techniques to achieve mapping and navigation through indoor environments. First, a scan matching algorithm based on ICP with distance threshold association counter is used as a multi-objective-like fitness function. Then, with Harmony Search, results are optimized without any previous initial guess or odometry. A global map is then built during SLAM, reducing the accumulated error and demonstrating better results than solo odometry LiDAR matching. As a novelty, both algorithms are implemented in 2D and 3D mapping, overlapping the resulting maps to fuse geometrical information at different heights. Finally, a room segmentation procedure is proposed by analyzing this information, avoiding occlusions that appear in 2D maps, and proving the benefits by implementing a door recognition system. Experiments are conducted in both simulated and real scenarios, proving the performance of the proposed algorithms.

Exploration-Based SLAM (e-SLAM) for the Indoor Mobile Robot Using Lidar

This paper attempts to uncover one possible method for the IMR (indoor mobile robot) to perform indoor exploration associated with SLAM (simultaneous localization and mapping) using LiDAR. Specifically, the IMR is required to construct a map when it has landed on an unexplored floor of a building. We had implemented the e-SLAM (exploration-based SLAM) using the coordinate transformation and the navigation prediction techniques to achieve that purpose in the engineering school building which consists of many 100-m² labs, corridors, elevator waiting space and the lobby. We first derive the LiDAR mesh for the orthogonal walls and filter out the static furniture and dynamic humans in the same space as the IMR. Then, we define the LiDAR pose frame including the translation and rotation from the orthogonal walls. According to the MSC (most significant corner) obtained from the intersection of the orthogonal walls, we calculate the displacement of the IMR. The orientation of the IMR is calculated from the alignment of orthogonal walls in the consecutive LiDAR pose frames, which is also assisted by the LQE (linear quadratic estimation) method. All the computation can be done in a single processor machine in real-time. The e-SLAM technique leads to a potential for the in-house service robot to start operation without having pre-scan LiDAR maps, which can save the installation time of the service robot. In this study, we use only the LiDAR and compared our result with the IMU to verify the consistency between the two navigation sensors in the experiments. The scenario of the experiment consists of rooms, corridors, elevators, and the lobby, which is common to most office buildings.

Local and Global Path Planning for Autonomous Mobile Robots Using Hierarchized Maps

We are currently facing a “labor crisis,” particularly in the field of logistics, because of reductions in the labor force. Therefore, industries must make their logistics more efficient by utilizing autonomous mobile robotics technologies. This paper proposes a hierarchized map concept that makes unmanned delivery tasks which use multiple autonomous robots more efficiently. Using our proposed concept, an autonomous mobile robot can move automatically on a more efficient path than using current methods. In addition, the management platform for autonomous robots can be used to prevent accidents such as collisions or deadlocks between autonomous robots.

Image based Localization under large perspective difference between Sfm and SLAM using split sim(3) optimization

Image based Localization (IbL) uses both Structure from Motion (SfM) and Simultaneous Localization and Mapping (SLAM) data for accurate pose estimation. However, under conditions where there is a large perspective difference between the SfM images and SLAM keyframes, the SfM-SLAM co-visibility graph becomes sparse. As a result, the scale drift can increase especially when using monocular SLAM as part of the IbL framework. The drift rarely gets corrected at loop closure due to its large magnitude. We propose a split affine transformation approach that uses SfM-SLAM information along with Sim(3) optimization to minimize the scale drift. Experiments are performed using an image dataset collected in a campus environment with different trajectories, showing the improvement in scale drift correction with the proposed method. The SLAM data was collected close to plainly textured structures like buildings while SfM images were captured from a larger distance from the building facade which leads to a challenging navigation scenario in the context of IbL. Localizing mobile platforms moving close to buildings is an example of such a case. The paper positively impacts the widespread use of small autonomous robotic platforms, which is to perform an accurate outdoor localization under urban conditions using only a monocular camera.

Low-Bandwidth and Compute-Bound RGB-D Planar Semantic SLAM

Visual simultaneous location and mapping (SLAM) using RGB-D cameras has been a necessary capability for intelligent mobile robots. However, when using point-cloud map representations as most RGB-D SLAM systems do, limitations in onboard compute resources, and especially communication bandwidth can significantly limit the quantity of data processed and shared. This article proposes techniques that help address these challenges by mapping point clouds to parametric models in order to reduce computation and bandwidth load on agents. This contribution is coupled with a convolutional neural network (CNN) that extracts semantic information. Semantics provide guidance in object modeling which can reduce the geometric complexity of the environment. Pairing a parametric model with a semantic label allows agents to share the knowledge of the world with much less complexity, opening a door for multi-agent systems to perform complex tasking, and human–robot cooperation. This article takes the first step towards a generalized parametric model by limiting the geometric primitives to a planar surface and providing semantic labels when appropriate. Two novel compression algorithms for depth data and a method to independently fit planes to RGB-D data are provided, so that plane data can be used for real-time odometry estimation and mapping. Additionally, we extend maps with semantic information predicted from sparse geometries (planes) by a CNN. In experiments, the advantages of our approach in terms of computational and bandwidth resources savings are demonstrated and compared with other state-of-the-art SLAM systems.

Localization and Mapping for Robots in Agriculture and Forestry: A Survey

Research and development of autonomous mobile robotic solutions that can perform several active agricultural tasks (pruning, harvesting, mowing) have been growing. Robots are now used for a variety of tasks such as planting, harvesting, environmental monitoring, supply of water and nutrients, and others. To do so, robots need to be able to perform online localization and, if desired, mapping. The most used approach for localization in agricultural applications is based in standalone Global Navigation Satellite System-based systems. However, in many agricultural and forest environments, satellite signals are unavailable or inaccurate, which leads to the need of advanced solutions independent from these signals. Approaches like simultaneous localization and mapping and visual odometry are the most promising solutions to increase localization reliability and availability. This work leads to the main conclusion that, few methods can achieve simultaneously the desired goals of scalability, availability, and accuracy, due to the challenges imposed by these harsh environments. In the near future, novel contributions to this field are expected that will help one to achieve the desired goals, with the development of more advanced techniques, based on 3D localization, and semantic and topological mapping. In this context, this work proposes an analysis of the current state-of-the-art of localization and mapping approaches in agriculture and forest environments. Additionally, an overview about the available datasets to develop and test these approaches is performed. Finally, a critical analysis of this research field is done, with the characterization of the literature using a variety of metrics.

Robot Localization in Water Pipes Using Acoustic Signals and Pose Graph Optimization

One of the most fundamental tasks for robots inspecting water distribution pipes is localization, which allows for autonomous navigation, for faults to be communicated, and for interventions to be instigated. Pose-graph optimization using spatially varying information is used to enable localization within a feature-sparse length of pipe. We present a novel method for improving estimation of a robot’s trajectory using the measured acoustic field, which is applicable to other measurements such as magnetic field sensing. Experimental results show that the use of acoustic information in pose-graph optimization reduces errors by 39% compared to the use of typical pose-graph optimization using landmark features only. High location accuracy is essential to efficiently and effectively target investment to maximise the use of our aging pipe infrastructure.

A brain-inspired compact cognitive mapping system

In many simultaneous localization and mapping (SLAM) systems, the map of the environment grows over time as the robot explores the environment. The ever-growing map prevents long-term mapping, especially in large-scale environments. In this paper, we develop a compact cognitive mapping approach inspired by neurobiological experiments. Mimicking the firing activities of neighborhood cells, neighborhood fields determined by movement information, i.e. translation and rotation, are modeled to describe one of the distinct segments of the explored environment. The vertices with low neighborhood field activities are avoided to be added into the cognitive map. The optimization of the cognitive map is formulated as a robust non-linear least squares problem constrained by the transitions between vertices, and is numerically solved efficiently. According to the cognitive decision-making of place familiarity, loop closure edges are clustered depending on time intervals, and then batch global optimization of the cognitive map is performed to satisfy the combined constraint of the whole cluster. After the loop closure process, scene integration is performed, in which revisited vertices are removed subsequently to further reduce the size of the cognitive map. The compact cognitive mapping approach is tested on a monocular visual SLAM system in a naturalistic maze for a biomimetic animated robot. Our results demonstrate that the proposed method largely restricts the growth of the size of the cognitive map over time, and meanwhile, the compact cognitive map correctly represents the overall layout of the environment. The compact cognitive mapping method is well suitable for the representation of large-scale environments to achieve long-term robot navigation.

Exactly sparse Gaussian variational inference with application to derivative-free batch nonlinear state estimation

We present a Gaussian variational inference (GVI) technique that can be applied to large-scale nonlinear batch state estimation problems. The main contribution is to show how to fit both the mean and (inverse) covariance of a Gaussian to the posterior efficiently, by exploiting factorization of the joint likelihood of the state and data, as is common in practical problems. This is different than maximum a posteriori (MAP) estimation, which seeks the point estimate for the state that maximizes the posterior (i.e., the mode). The proposed exactly sparse Gaussian variational inference (ESGVI) technique stores the inverse covariance matrix, which is typically very sparse (e.g., block-tridiagonal for classic state estimation). We show that the only blocks of the (dense) covariance matrix that are required during the calculations correspond to the non-zero blocks of the inverse covariance matrix, and further show how to calculate these blocks efficiently in the general GVI problem. ESGVI operates iteratively, and while we can use analytical derivatives at each iteration, Gaussian cubature can be substituted, thereby producing an efficient derivative-free batch formulation. ESGVI simplifies to precisely the Rauch–Tung–Striebel (RTS) smoother in the batch linear estimation case, but goes beyond the ‘extended’ RTS smoother in the nonlinear case because it finds the best-fit Gaussian (mean and covariance), not the MAP point estimate. We demonstrate the technique on controlled simulation problems and a batch nonlinear simultaneous localization and mapping problem with an experimental dataset.

Least Squares Optimization: From Theory to Practice

Nowadays, Nonlinear Least-Squares embodies the foundation of many Robotics and Computer Vision systems. The research community deeply investigated this topic in the last few years, and this resulted in the development of several open-source solvers to approach constantly increasing classes of problems. In this work, we propose a unified methodology to design and develop efficient Least-Squares Optimization algorithms, focusing on the structures and patterns of each specific domain. Furthermore, we present a novel open-source optimization system that addresses problems transparently with a different structure and designed to be easy to extend. The system is written in modern C++ and runs efficiently on embedded systemsWe validated our approach by conducting comparative experiments on several problems using standard datasets. The results show that our system achieves state-of-the-art performances in all tested scenarios.

A survey of image semantics-based visual simultaneous localization and mapping: Application-oriented solutions to autonomous navigation of mobile robots

As one of the typical application-oriented solutions to robot autonomous navigation, visual simultaneous localization and mapping is essentially restricted to simplex environmental understanding based on geometric features of images. By contrast, the semantic simultaneous localization and mapping that is characterized by high-level environmental perception has apparently opened the door to apply image semantics to efficiently estimate poses, detect loop closures, build 3D maps, and so on. This article presents a detailed review of recent advances in semantic simultaneous localization and mapping, which mainly covers the treatments in terms of perception, robustness, and accuracy. Specifically, the concept of “semantic extractor” and the framework of “modern visual simultaneous localization and mapping” are initially presented. As the challenges associated with perception, robustness, and accuracy are being stated, we further discuss some open problems from a macroscopic view and attempt to find answers. We argue that multiscaled map representation, object simultaneous localization and mapping system, and deep neural network-based simultaneous localization and mapping pipeline design could be effective solutions to image semantics-fused visual simultaneous localization and mapping.

A Review of Visual-LiDAR Fusion based Simultaneous Localization and Mapping

Autonomous navigation requires both a precise and robust mapping and localization solution. In this context, Simultaneous Localization and Mapping (SLAM) is a very well-suited solution. SLAM is used for many applications including mobile robotics, self-driving cars, unmanned aerial vehicles, or autonomous underwater vehicles. In these domains, both visual and visual-IMU SLAM are well studied, and improvements are regularly proposed in the literature. However, LiDAR-SLAM techniques seem to be relatively the same as ten or twenty years ago. Moreover, few research works focus on vision-LiDAR approaches, whereas such a fusion would have many advantages. Indeed, hybridized solutions offer improvements in the performance of SLAM, especially with respect to aggressive motion, lack of light, or lack of visual features. This study provides a comprehensive survey on visual-LiDAR SLAM. After a summary of the basic idea of SLAM and its implementation, we give a complete review of the state-of-the-art of SLAM research, focusing on solutions using vision, LiDAR, and a sensor fusion of both modalities.

Global Registration of Terrestrial Laser Scanner Point Clouds Using Plane-to-Plane Correspondences

Registration of point clouds is a central problem in many mapping and monitoring applications, such as outdoor and indoor mapping, high-speed railway track inspection, heritage documentation, building information modeling, and others. However, ensuring the global consistency of the registration is still a challenging task when there are multiple point clouds because the different scans should be transformed into a common coordinate frame. The aim of this paper is the registration of multiple terrestrial laser scanner point clouds. We present a plane-based matching algorithm to find plane-to-plane correspondences using a new parametrization based on complex numbers. The multiplication of complex numbers is based on analysis of the quadrants to avoid the ambiguity in the calculation of the rotation angle formed between normal vectors of adjacent planes. As a matching step may contain several matrix operations, our strategy is applied to reduce the number of mathematical operations. We also design a novel method for global refinement of terrestrial laser scanner data based on plane-to-plane correspondences. The rotation parameters are globally refined using operations of quaternion multiplication, while the translation parameters are refined using the parameters of planes. The global refinement is done non-iteratively. The experimental results show that the proposed plane-based matching algorithm efficiently finds plane correspondences in partial overlapping scans providing approximate values for the global registration, and indicate that an accuracy better than 8 cm can be achieved by using our global fine plane-to-plane registration method.

Multi-Objective Optimization of Loop Closure Detection Parameters for Indoor 2D Simultaneous Localization and Mapping

Aiming at addressing the issues related to the tuning of loop closure detection parameters for indoor 2D graph-based simultaneous localization and mapping (SLAM), this article proposes a multi-objective optimization method for these parameters. The proposed method unifies the Karto SLAM algorithm, an efficient evaluation approach for map quality with three quantitative metrics, and a multi-objective optimization algorithm. More particularly, the evaluation metrics, i.e., the proportion of occupied grids, the number of corners and the amount of enclosed areas, can reflect the errors such as overlaps, blurring and misalignment when mapping nested loops, even in the absence of ground truth. The proposed method has been implemented and validated by testing on four datasets and two real-world environments. For all these tests, the map quality can be improved using the proposed method. Only loop closure detection parameters have been considered in this article, but the proposed evaluation metrics and optimization method have potential applications in the automatic tuning of other SLAM parameters to improve the map quality.

Confidence-based dynamic optimization model for biomedical image mosaicking

Biomedical image mosaicking is a trending topic. It consists of computing a single large image from multiple observations and becomes a challenging task when said observations barely overlap or are subject to illumination changes, poor resolution, blur, or either highly textured or predominantly homogeneous content. Because such challenges are common in biomedical images, classical keypoint/feature-based methods perform poorly. In this paper, we propose a new framework based on pairwise template matching coupled with a constrained, confidence-driven global optimization strategy to solve the issue of microscopic biomedical image mosaicking. First we provide experimental results that show significant improvement on a subjective level. Then we describe a new validation strategy for objective assessment, which shows improvement as well.

Learning the Cost Function for Foothold Selection in a Quadruped Robot

This paper is focused on designing a cost function of selecting a foothold for a physical quadruped robot walking on rough terrain. The quadruped robot is modeled with Denavit–Hartenberg (DH) parameters, and then a default foothold is defined based on the model. Time of Flight (TOF) camera is used to perceive terrain information and construct a 2.5D elevation map, on which the terrain features are detected. The cost function is defined as the weighted sum of several elements including terrain features and some features on the relative pose between the default foothold and other candidates. It is nearly impossible to hand-code the weight vector of the function, so the weights are learned using Supporting Vector Machine (SVM) techniques, and the training data set is generated from the 2.5D elevation map of a real terrain under the guidance of experts. Four candidate footholds around the default foothold are randomly sampled, and the expert gives the order of such four candidates by rotating and scaling the view for seeing clearly. Lastly, the learned cost function is used to select a suitable foothold and drive the quadruped robot to walk autonomously across the rough terrain with wooden steps. Comparing to the approach with the original standard static gait, the proposed cost function shows better performance.

Have I Seen This Place Before? A Fast and Robust Loop Detection and Correction Method for 3D Lidar SLAM

In this paper, we present a complete loop detection and correction system developed for data originating from lidar scanners. Regarding detection, we propose a combination of a global point cloud matcher with a novel registration algorithm to determine loop candidates in a highly effective way. The registration method can deal with point clouds that are largely deviating in orientation while improving the efficiency over existing techniques. In addition, we accelerated the computation of the global point cloud matcher by a factor of 2–4, exploiting the GPU to its maximum. Experiments demonstrated that our combined approach more reliably detects loops in lidar data compared to other point cloud matchers as it leads to better precision–recall trade-offs: for nearly 100% recall, we gain up to 7% in precision. Finally, we present a novel loop correction algorithm that leads to an improvement by a factor of 2 on the average and median pose error, while at the same time only requires a handful of seconds to complete.