Fast and Fault-Tolerant Passive Hyperbolic Localization Using Sensor Consensus

The accuracy of passive hyperbolic localization applications using Time Difference of Arrival (TDOA) measurements can be severely compromised in non-line-of-sight (NLOS) situations. Consensus functions have been successfully used to provide robust and accurate location estimates in such challenging situations. In this paper, a fast branch-and-bound computational method for finding the global maximum of consensus functions is proposed and the global convergence property of the algorithm is mathematically proven. The performance of the method is illustrated by simulation experiments and real measurements.

Exploiting Graph and Geodesic Distance Constraint for Deep Learning-Based Visual Odometry

Visual odometry is the task of estimating the trajectory of the moving agents from consecutive images. It is a hot research topic both in robotic and computer vision communities and facilitates many applications, such as autonomous driving and virtual reality. The conventional odometry methods predict the trajectory by utilizing the multiple view geometry between consecutive overlapping images. However, these methods need to be carefully designed and fine-tuned to work well in different environments. Deep learning has been explored to alleviate the challenge by directly predicting the relative pose from the paired images. Deep learning-based methods usually focus on the consecutive images that are feasible to propagate the error over time. In this paper, graph loss and geodesic rotation loss are proposed to enhance deep learning-based visual odometry methods based on graph constraints and geodesic distance, respectively. The graph loss not only considers the relative pose loss of consecutive images, but also the relative pose of non-consecutive images. The relative pose of non-consecutive images is not directly predicted but computed from the relative pose of consecutive ones. The geodesic rotation loss is constructed by the geodesic distance and the model regresses a Lie algebra so(3) (3D vector). This allows a robust and stable convergence. To increase the efficiency, a random strategy is adopted to select the edges of the graph instead of using all of the edges. This strategy provides additional regularization for training the networks. Extensive experiments are conducted on visual odometry benchmarks, and the obtained results demonstrate that the proposed method has comparable performance to other supervised learning-based methods, as well as monocular camera-based methods. The source code and the weight are made publicly available.

A Robust Localization System Fusion Vision-CNN Relocalization and Progressive Scan Matching for Indoor Mobile Robots

Map-based, high-precision dynamic pose tracking and rapid relocalization in the case of unknown poses are very important for indoor navigation robots. This paper aims to propose a robust and high-precision indoor robot positioning algorithm that combines vision and laser sensor information. This algorithm mainly includes two parts: initialization and real-time pose tracking. The initialization component is mainly to solve the problem of the uncertainty of a robot’s initial pose and loss of pose tracking. First, the laser information is added to the posenetLSTM neural network that only considers image information as a geometric constraint, and the loss function is redesigned thereby improving global positioning accuracy. Second, on the basis of visual rough positioning, the branch and bound method is used to quickly search the high-precision pose of the robot. In the real-time tracking component, small-scale correlation sampling is performed on the high-resolution environment grid map, and the robot’s pose is dynamically tracked in real time. When the score of the tracking pose is lower than a certain threshold, the method of nonlinear graph optimization is used to perform the pose optimization. In order to prove the robustness, high precision, and real-time performance of the algorithm, this article first builds a simulation environment in Gazebo for evaluation, and then verifies the relevant performance of the algorithm through the Mir robot platform. Both simulations and experiments show that the introduction of laser information into the neural network can greatly improve the accuracy of vision relocalization and the system can quickly perform high-precision repositioning when the camera is not severely blocked. At the same time, compared with the pose tracking performance of the adaptive Monte Carlo localization (AMCL) algorithm, the proposed algorithm has also improved in accuracy and in real-time performance.

Smartphone Sensor Based Indoor Positioning: Current Status, Opportunities, and Future Challenges

The last two decades have witnessed a rich variety of indoor positioning and localization research. Starting with Microsoft Research pioneering the fingerprint approach based RADAR, MIT’s Cricket, and then moving towards beacon-based localization are few among many others. In parallel, researchers looked into other appealing and promising technologies like radio frequency identification, ultra-wideband, infrared, and visible light-based systems. However, the proliferation of smartphones over the past few years revolutionized and reshaped indoor localization towards new horizons. The deployment of MEMS sensors in modern smartphones have initiated new opportunities and challenges for the industry and academia alike. Additionally, the demands and potential of location-based services compelled the researchers to look into more robust, accurate, smartphone deployable, and context-aware location sensing. This study presents a comprehensive review of the approaches that make use of data from one or more sensors to estimate the user’s indoor location. By analyzing the approaches leveraged on smartphone sensors, it discusses the associated challenges of such approaches and points out the areas that need considerable research to overcome their limitations.

DeepLocate: Smartphone Based Indoor Localization with a Deep Neural Network Ensemble Classifier

A quickly growing location-based services area has led to increased demand for indoor positioning and localization. Undoubtedly, Wi-Fi fingerprint-based localization is one of the promising indoor localization techniques, yet the variation of received signal strength is a major problem for accurate localization. Magnetic field-based localization has emerged as a new player and proved a potential indoor localization technology. However, one of its major limitations is degradation in localization accuracy when various smartphones are used. The localization performance is different from various smartphones even with the same localization technique. This research leverages the use of a deep neural network-based ensemble classifier to perform indoor localization with heterogeneous devices. The chief aim is to devise an approach that can achieve a similar localization accuracy using various smartphones. Features extracted from magnetic data of Galaxy S8 are fed into neural networks (NNs) for training. The experiments are performed with Galaxy S8, LG G6, LG G7, and Galaxy A8 smartphones to investigate the impact of device dependence on localization accuracy. Results demonstrate that NNs can play a significant role in mitigating the impact of device heterogeneity and increasing indoor localization accuracy. The proposed approach is able to achieve a localization accuracy of 2.64 m at 50% on four different devices. The mean error is 2.23 m, 2.52 m, 2.59 m, and 2.78 m for Galaxy S8, LG G6, LG G7, and Galaxy A8, respectively. Experiments on a publicly available magnetic dataset of Sony Xperia M2 using the proposed approach show a mean error of 2.84 m with a standard deviation of 2.24 m, while the error at 50% is 2.33 m. Furthermore, the impact of devices on various attitudes on the localization accuracy is investigated.

Indoor Positioning on Disparate Commercial Smartphones Using Wi-Fi Access Points Coverage Area

The applications of location-based services require precise location information of a user both indoors and outdoors. Global positioning system’s reduced accuracy for indoor environments necessitated the initiation of Indoor Positioning Systems (IPSs). However, the development of an IPS which can determine the user’s position with heterogeneous smartphones in the same fashion is a challenging problem. The performance of Wi-Fi fingerprinting-based IPSs is degraded by many factors including shadowing, absorption, and interference caused by obstacles, human mobility, and body loss. Moreover, the use of various smartphones and different orientations of the very same smartphone can limit its positioning accuracy as well. As Wi-Fi fingerprinting is based on Received Signal Strength (RSS) vector, it is prone to dynamic intrinsic limitations of radio propagation, including changes over time, and far away locations having similar RSS vector. This article presents a Wi-Fi fingerprinting approach that exploits Wi-Fi Access Points (APs) coverage area and does not utilize the RSS vector. Using the concepts of APs coverage area uniqueness and coverage area overlap, the proposed approach calculates the user’s current position with the help of APs’ intersection area. The experimental results demonstrate that the device dependency can be mitigated by making the fingerprinting database with the proposed approach. The experiments performed at a public place proves that positioning accuracy can also be increased because the proposed approach performs well in dynamic environments with human mobility. The impact of human body loss is studied as well.

A Robust Indoor Localization System Integrating Visual Localization Aided by CNN-Based Image Retrieval with Monte Carlo Localization

This paper proposes a novel multi-sensor-based indoor global localization system integrating visual localization aided by CNN-based image retrieval with a probabilistic localization approach. The global localization system consists of three parts: coarse place recognition, fine localization and re-localization from kidnapping. Coarse place recognition exploits a monocular camera to realize the initial localization based on image retrieval, in which off-the-shelf features extracted from a pre-trained Convolutional Neural Network (CNN) are adopted to determine the candidate locations of the robot. In the fine localization, a laser range finder is equipped to estimate the accurate pose of a mobile robot by means of an adaptive Monte Carlo localization, in which the candidate locations obtained by image retrieval are considered as seeds for initial random sampling. Additionally, to address the problem of robot kidnapping, we present a closed-loop localization mechanism to monitor the state of the robot in real time and make adaptive adjustments when the robot is kidnapped. The closed-loop mechanism effectively exploits the correlation of image sequences to realize the re-localization based on Long-Short Term Memory (LSTM) network. Extensive experiments were conducted and the results indicate that the proposed method not only exhibits great improvement on accuracy and speed, but also can recover from localization failures compared to two conventional localization methods.

Intention Estimation Using Set of Reference Trajectories as Behaviour Model

Autonomous robotic systems operating in the vicinity of other agents, such as humans, manually driven vehicles and other robots, can model the behaviour and estimate intentions of the other agents to enhance efficiency of their operation, while preserving safety. We propose a data-driven approach to model the behaviour of other agents, which is based on a set of trajectories navigated by other agents. Then, to evaluate the proposed behaviour modelling approach, we propose and compare two methods for agent intention estimation based on: (i) particle filtering; and (ii) decision trees. The proposed methods were validated using three datasets that consist of real-world bicycle and car trajectories in two different scenarios, at a roundabout and at a t-junction with a pedestrian crossing. The results validate the utility of the data-driven behaviour model, and show that decision-tree based intention estimation works better on a binary-class problem, whereas the particle-filter based technique performs better on a multi-class problem, such as the roundabout, where the method yielded an average gain of 14.88 m for correct intention estimation locations compared to the decision-tree based method.

Fast loop-closure detection using visual-word-vectors from image sequences

In this paper, a novel pipeline for loop-closure detection is proposed. We base our work on a bag of binary feature words and we produce a description vector capable of characterizing a physical scene as a whole. Instead of relying on single camera measurements, the robot’s trajectory is dynamically segmented into image sequences according to its content. The visual word occurrences from each sequence are then combined to create sequence-visual-word-vectors and provide additional information to the matching functionality. In this way, scenes with considerable visual differences are firstly discarded, while the respective image-to-image associations are provided subsequently. With the purpose of further enhancing the system’s performance, a novel temporal consistency filter (trained offline) is also introduced to advance matches that persist over time. Evaluation results prove that the presented method compares favorably with other state-of-the-art techniques, while our algorithm is tested on a tablet device, verifying the computational efficiency of the approach.

Filtering with State-Observation Examples via Kernel Monte Carlo Filter

This letter addresses the problem of filtering with a state-space model. Standard approaches for filtering assume that a probabilistic model for observations (i.e., the observation model) is given explicitly or at least parametrically. We consider a setting where this assumption is not satisfied; we assume that the knowledge of the observation model is provided only by examples of state-observation pairs. This setting is important and appears when state variables are defined as quantities that are very different from the observations. We propose kernel Monte Carlo filter, a novel filtering method that is focused on this setting. Our approach is based on the framework of kernel mean embeddings, which enables nonparametric posterior inference using the state-observation examples. The proposed method represents state distributions as weighted samples, propagates these samples by sampling, estimates the state posteriors by kernel Bayes' rule, and resamples by kernel herding. In particular, the sampling and resampling procedures are novel in being expressed using kernel mean embeddings, so we theoretically analyze their behaviors. We reveal the following properties, which are similar to those of corresponding procedures in particle methods: the performance of sampling can degrade if the effective sample size of a weighted sample is small, and resampling improves the sampling performance by increasing the effective sample size. We first demonstrate these theoretical findings by synthetic experiments. Then we show the effectiveness of the proposed filter by artificial and real data experiments, which include vision-based mobile robot localization.

Probabilistic Multi-Sensor Fusion Based Indoor Positioning System on a Mobile Device

Nowadays, smart mobile devices include more and more sensors on board, such as motion sensors (accelerometer, gyroscope, magnetometer), wireless signal strength indicators (WiFi, Bluetooth, Zigbee), and visual sensors (LiDAR, camera). People have developed various indoor positioning techniques based on these sensors. In this paper, the probabilistic fusion of multiple sensors is investigated in a hidden Markov model (HMM) framework for mobile-device user-positioning. We propose a graph structure to store the model constructed by multiple sensors during the offline training phase, and a multimodal particle filter to seamlessly fuse the information during the online tracking phase. Based on our algorithm, we develop an indoor positioning system on the iOS platform. The experiments carried out in a typical indoor environment have shown promising results for our proposed algorithm and system design.

Localization from semantic observations via the matrix permanent

Most approaches to robot localization rely on low-level geometric features such as points, lines, and planes. In this paper, we use object recognition to obtain semantic information from the robot’s sensors and consider the task of localizing the robot within a prior map of landmarks, which are annotated with semantic labels. As object recognition algorithms miss detections and produce false alarms, correct data association between the detections and the landmarks on the map is central to the semantic localization problem. Instead of the traditional vector-based representation, we propose a sensor model, which encodes the semantic observations via random finite sets and enables a unified treatment of missed detections, false alarms, and data association. Our second contribution is to reduce the problem of computing the likelihood of a set-valued observation to the problem of computing a matrix permanent. It is this crucial transformation that allows us to solve the semantic localization problem with a polynomial-time approximation to the set-based Bayes filter. Finally, we address the active semantic localization problem, in which the observer’s trajectory is planned in order to improve the accuracy and efficiency of the localization process. The performance of our approach is demonstrated in simulation and in real environments using deformable-part-model-based object detectors. Robust global localization from semantic observations is demonstrated for a mobile robot, for the Project Tango phone, and on the KITTI visual odometry dataset. Comparisons are made with the traditional lidar-based geometric Monte Carlo localization.

FEATURE GROUP MATCHING: A NOVEL METHOD TO FILTER OUT INCORRECT LOCAL FEATURE MATCHINGS

The importance of finding correct correspondences between two images is the major aspect in problems such as appearance-based robot localization and content-based image retrieval. Local feature matching has become a commonly used method to compare images, despite being highly probable that at least some of the matchings/correspondences it detects are incorrect. In this paper, we describe a novel approach to local feature matching, named Feature Group Matching (FGM), to select stable features and obtain a more reliable similarity value between two images. The proposed technique is demonstrated to be translational, rotational and scaling invariant. Experimental evaluation was performed on large and heterogeneous datasets of images using SIFT and SURF, the actual state-of-the-art feature extractors. Results show that FGM avoids almost 95% of incorrect matchings, reduces the visual aliasing (number of images considered similar) and increases both robotic localization and image retrieval accuracy on the average of 13%.

Tracking Control of Mobile Robots Localized via Chained Fusion of Discrete and Continuous Epipolar Geometry, IMU and Odometry

This paper presents a novel navigation and control system for autonomous mobile robots that includes path planning, localization, and control. A unique vision-based pose and velocity estimation scheme utilizing both the continuous and discrete forms of the Euclidean homography matrix is fused with inertial and optical encoder measurements to estimate the pose, orientation, and velocity of the robot and ensure accurate localization and control signals. A depth estimation system is integrated in order to overcome the loss of scale inherent in vision-based estimation. A path following control system is introduced that is capable of guiding the robot along a designated curve. Stability analysis is provided for the control system and experimental results are presented that prove the combined localization and control system performs with high accuracy.

Bubble space and place representation in topological maps

This paper presents bubble space based representation of “places” (nodes) in topological maps. Bubble space simultaneously provides for detailed (bubble surfaces) and holistic (bubble descriptors) representation of places. It is based on bubble memory where visual feature values and their local S2-metric relations from robot’s viewpoint are simultaneously encoded on a deformable spherical surface. Bubble surfaces extend bubble memory to accommodate varying robot pose and multiple features. They are transformed into bubble descriptors that are rotationally invariant with respect to heading changes while being computable in an incremental manner as each new set of visual observations is made. We use bubble descriptors for place learning and recognition with support vector machines in both indoor and outdoor environments and provide analysis results on recognition, recall and precision rates and time performance including a comparative study with the state-of-the-art descriptors.

Efficient Topological Localization Using Global and Local Feature Matching

We present an efficient vision-based global topological localization approach in which different image features are used in a coarse-to-fine matching framework. Orientation Adjacency Coherence Histogram (OACH), a novel image feature, is proposed to improve the coarse localization. The coarse localization results are taken as inputs for the fine localization which is carried out by matching Harris-Laplace interest points characterized by the SIFT descriptor. The computation of OACHs and interest points is efficient due to the fact that these features are computed in an integrated process. The matching of local features is improved by using approximate nearest neighbor searching technique. We have implemented and tested the localization system in real environments. The experimental results demonstrate that our approach is efficient and reliable in both indoor and outdoor environments. This work has also been compared with previous works. The comparison results show that our approach has better performance with higher correct ratio and lower computational complexity.

Precise Localization of Mobile Robots via Odometry and Wireless Sensor Network

Precise localization of mobile robots in uncertain environments is a fundamental and crucial issue in robotics. In this paper, to deal with the unbounded accumulated errors of dead reckoning (DR)-based localization, wireless sensor network (WSN)-based localization is applied to calibrate the uncertainty of odometry using a Kalman filter (KF). In addition, to further aid in obtaining precise positions and reduce uncertainty, a novel backward dead reckoning (BDR) localization approach is proposed. The experimental results demonstrate the success and reliability of the proposed method.

The uspIC: performing scan matching localization using an imaging sonar

This paper presents a novel approach to localize an underwater mobile robot based on scan matching using a Mechanically Scanned Imaging Sonar (MSIS). When used to perform scan matching, this sensor presents some problems such as significant uncertainty in the measurements or large scan times, which lead to a motion induced distortion. This paper presents the uspIC, which deals with these problems by adopting a probabilistic scan matching strategy and by defining a method to strongly alleviate the motion induced distortion. Experimental results evaluating our approach and comparing it to previously existing methods are provided.

A dynamic localization algorithm for a high-speed mobile robot using indoor GPS

This paper proposes a dynamic localization algorithm for a variable speed mobile robot. In order to localize a mobile robot with active beacon sensors, a relatively long time is needed, since the distance to the beacon is measured using the time-of-flight of the ultrasonic signal. The measurement time does not cause a high error rate when the mobile robot moves slowly. However, with an increase of the mobile robot's speed, the localization error becomes too high to use for accurate mobile robot navigation. Therefore, in this research into high-speed mobile robot operations, instead of using two active beacons for localization, an active beacon and the predicted path are utilized to localize the mobile robot. This new approach resolves the high-localization error caused by the speed of mobile robot. The performance of this method was verified by comparing it to the conventional method through real-world experiments.

High-accuracy global localization filter for three-dimensional environments

The localization problem in mobile robotics can be defined as the search of the robot's coordinates in a known environment. If there is no information about the initial location, we are talking about global localization. In this work, we have developed an algorithm that solves this problem in a three-dimensional (3D) environment using evolutionary computation concepts. The method has been called RELF-3D and has many features that make it very robust and reliable: thresholding and discarding mechanisms, different cost functions, effective convergence criteria, and so on. The resulting global localization module has been tested in numerous experiments and the most important improvement obtained is the accuracy of the method, allowing its application in manipulation tasks.

Localization of a high-speed mobile robot using global features

A new localization algorithm is proposed for a fast moving mobile robot, which utilizes only one beacon and the global features of the differential-driving mobile robot. It takes a relatively long time to localize a mobile robot with active beacon sensors, since the distance to the beacon is measured based on the traveling time of the ultrasonic signal. When the mobile robot is moving slowly, the measurement time does not yield a high error. At a higher speed, however, the localization error becomes too large for the mobile robot to be located accurately. Therefore, in high-speed mobile robot operations, instead of using two or more active beacons, only one active beacon and the global features of the mobile robot are used to localize the mobile robot. The two global features are the radius and center of the rotational motion for the differential-driving mobile robot, which generally describe the motion of, and are used for the trace prediction of, a mobile robot. In high-speed operations, the localizer finds the intersection point of this predicted trace and the circle, which is centered at the beacon whose radius is the distance between the mobile robot and the beacon. This new approach overcomes the large localization error caused by the high speed of the mobile robot. The performance of the new localization algorithm is verified through experiments with a high-speed mobile robot.

Online and Incremental Appearance-based SLAM in Highly Dynamic Environments

In this paper we present a novel method for online and incremental appearance-based localization and mapping in a highly dynamic environment. Using position-invariant robust features (PIRFs), the method can achieve a high rate of recall with 100% precision. It can handle both strong perceptual aliasing and dynamic changes of places efficiently. Its performance also extends beyond conventional images; it is applicable to omnidirectional images for which the major portions of scenes are similar for most places. The proposed PIRF-based Navigation method named PIRF-Nav is evaluated by testing it on two standard datasets in a similar manner as in FAB-MAP and on an additional omnidirectional image dataset that we collected. This extra dataset was collected on 2 days with different specific events, i.e. an open-campus event, to present challenges related to illumination variance and strong dynamic changes, and to test assessment of dynamic scene changes. Results show that PIRF-Nav outperforms FAB-MAP; PIRF-Nav at precision-1 yields a recall rate about twice as high (approximately 80% increase) than that of FAB-MAP. Its computation time is sufficiently short for real-time applications. The method is fully incremental, and requires no offline process for dictionary creation. Additional testing using combined datasets proves that PIRF-Nav can function over the long term and can solve the kidnapped robot problem.

Distance and cable length measurement system

A simple, economic and successful design for distance and cable length detection is presented. The measurement system is based on the continuous repetition of a pulse that endlessly travels along the distance to be detected. There is a pulse repeater at both ends of the distance or cable to be measured. The endless repetition of the pulse generates a frequency that varies almost inversely with the distance to be measured. The resolution and distance or cable length range could be adjusted by varying the repetition time delay introduced at both ends and the measurement time. With this design a distance can be measured with centimeter resolution using electronic system with microsecond resolution, simplifying classical time of flight designs which require electronics with picosecond resolution. This design was also applied to position measurement.

Navigating, Recognizing and Describing Urban Spaces With Vision and Lasers

In this paper we describe a body of work aimed at extending the reach of mobile navigation and mapping. We describe how running topological and metric mapping and pose estimation processes concurrently, using vision and laser ranging, has produced a full six-degree-of-freedom outdoor navigation system. It is capable of producing intricate three-dimensional maps over many kilometers and in real time. We consider issues concerning the intrinsic quality of the built maps and describe our progress towards adding semantic labels to maps via scene de-construction and labeling. We show how our choices of representation, inference methods and use of both topological and metric techniques naturally allow us to fuse maps built from multiple sessions with no need for manual frame alignment or data association.

Particle filtering for multiple object tracking in dynamic fluorescence microscopy images: application to microtubule growth analysis

Quantitative analysis of dynamic processes in living cells by means of fluorescence microscopy imaging requires tracking of hundreds of bright spots in noisy image sequences. Deterministic approaches, which use object detection prior to tracking, perform poorly in the case of noisy image data. We propose an improved, completely automatic tracker, built within a Bayesian probabilistic framework. It better exploits spatiotemporal information and prior knowledge than common approaches, yielding more robust tracking also in cases of photobleaching and object interaction. The tracking method was evaluated using simulated but realistic image sequences, for which ground truth was available. The results of these experiments show that the method is more accurate and robust than popular tracking methods. In addition, validation experiments were conducted with real fluorescence microscopy image data acquired for microtubule growth analysis. These demonstrate that the method yields results that are in good agreement with manual tracking performed by expert cell biologists. Our findings suggest that the method may replace laborious manual procedures.

FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance

This paper describes a probabilistic approach to the problem of recognizing places based on their appearance. The system we present is not limited to localization, but can determine that a new observation comes from a previously unseen place, and so augment its map. Effectively this is a SLAM system in the space of appearance. Our probabilistic approach allows us to explicitly account for perceptual aliasing in the environment—identical but indistinctive observations receive a low probability of having come from the same place. We achieve this by learning a generative model of place appearance. By partitioning the learning problem into two parts, new place models can be learned online from only a single observation of a place. The algorithm complexity is linear in the number of places in the map, and is particularly suitable for online loop closure detection in mobile robotics.

Qualitative visual environment retrieval

A system for retrieval of an unstructured environment under static and dynamic scenarios is proposed. The use of cylindrical mosaics or omnidirectional images is exploited for providing a rich description about the surrounding environment spanning 360 degrees. The environment description is based on defining the attributes of the nodes of a graph derived from the angular partitions of the captured images. Content-based image retrieval for each of these partitions is performed on an exemplar image database to annotate the nodes of the graph. The complete environment description is recovered by collating the retrieval results over all the partitions based on a simple voting scheme. This offers a qualitative description of the location in a totally natural and unstructured surrounding. The experiments yield quite promising results.