1
|
Tenzin S, Rassau A, Chai D. Application of Event Cameras and Neuromorphic Computing to VSLAM: A Survey. Biomimetics (Basel) 2024; 9:444. [PMID: 39056885 DOI: 10.3390/biomimetics9070444] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2024] [Revised: 07/15/2024] [Accepted: 07/18/2024] [Indexed: 07/28/2024] Open
Abstract
Simultaneous Localization and Mapping (SLAM) is a crucial function for most autonomous systems, allowing them to both navigate through and create maps of unfamiliar surroundings. Traditional Visual SLAM, also commonly known as VSLAM, relies on frame-based cameras and structured processing pipelines, which face challenges in dynamic or low-light environments. However, recent advancements in event camera technology and neuromorphic processing offer promising opportunities to overcome these limitations. Event cameras inspired by biological vision systems capture the scenes asynchronously, consuming minimal power but with higher temporal resolution. Neuromorphic processors, which are designed to mimic the parallel processing capabilities of the human brain, offer efficient computation for real-time data processing of event-based data streams. This paper provides a comprehensive overview of recent research efforts in integrating event cameras and neuromorphic processors into VSLAM systems. It discusses the principles behind event cameras and neuromorphic processors, highlighting their advantages over traditional sensing and processing methods. Furthermore, an in-depth survey was conducted on state-of-the-art approaches in event-based SLAM, including feature extraction, motion estimation, and map reconstruction techniques. Additionally, the integration of event cameras with neuromorphic processors, focusing on their synergistic benefits in terms of energy efficiency, robustness, and real-time performance, was explored. The paper also discusses the challenges and open research questions in this emerging field, such as sensor calibration, data fusion, and algorithmic development. Finally, the potential applications and future directions for event-based SLAM systems are outlined, ranging from robotics and autonomous vehicles to augmented reality.
Collapse
Affiliation(s)
- Sangay Tenzin
- School of Engineering, Edith Cowan University, Perth, WA 6027, Australia
| | - Alexander Rassau
- School of Engineering, Edith Cowan University, Perth, WA 6027, Australia
| | - Douglas Chai
- School of Engineering, Edith Cowan University, Perth, WA 6027, Australia
| |
Collapse
|
2
|
Yue S, Wang Z, Zhang X. DSOMF: A Dynamic Environment Simultaneous Localization and Mapping Technique Based on Machine Learning. SENSORS (BASEL, SWITZERLAND) 2024; 24:3063. [PMID: 38793916 PMCID: PMC11125829 DOI: 10.3390/s24103063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/12/2024] [Revised: 05/07/2024] [Accepted: 05/10/2024] [Indexed: 05/26/2024]
Abstract
To address the challenges of reduced localization accuracy and incomplete map construction demonstrated using classical semantic simultaneous localization and mapping (SLAM) algorithms in dynamic environments, this study introduces a dynamic scene SLAM technique that builds upon direct sparse odometry (DSO) and incorporates instance segmentation and video completion algorithms. While prioritizing the algorithm's real-time performance, we leverage the rapid matching capabilities of Direct Sparse Odometry (DSO) to link identical dynamic objects in consecutive frames. This association is achieved through merging semantic and geometric data, thereby enhancing the matching accuracy during image tracking through the inclusion of semantic probability. Furthermore, we incorporate a loop closure module based on video inpainting algorithms into our mapping thread. This allows our algorithm to rely on the completed static background for loop closure detection, further enhancing the localization accuracy of our algorithm. The efficacy of this approach is validated using the TUM and KITTI public datasets and the unmanned platform experiment. Experimental results show that, in various dynamic scenes, our method achieves an improvement exceeding 85% in terms of localization accuracy compared with the DSO system.
Collapse
Affiliation(s)
| | - Zhengjie Wang
- School of Electromechanical Engineering, Beijing Institute of Technology, Beijing 100081, China; (S.Y.); (X.Z.)
| | | |
Collapse
|
3
|
Zhuang L, Zhong X, Xu L, Tian C, Yu W. Visual SLAM for Unmanned Aerial Vehicles: Localization and Perception. SENSORS (BASEL, SWITZERLAND) 2024; 24:2980. [PMID: 38793834 PMCID: PMC11126069 DOI: 10.3390/s24102980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Revised: 05/01/2024] [Accepted: 05/04/2024] [Indexed: 05/26/2024]
Abstract
Localization and perception play an important role as the basis of autonomous Unmanned Aerial Vehicle (UAV) applications, providing the internal state of movements and the external understanding of environments. Simultaneous Localization And Mapping (SLAM), one of the critical techniques for localization and perception, is facing technical upgrading, due to the development of embedded hardware, multi-sensor technology, and artificial intelligence. This survey aims at the development of visual SLAM and the basis of UAV applications. The solutions to critical problems for visual SLAM are shown by reviewing state-of-the-art and newly presented algorithms, providing the research progression and direction in three essential aspects: real-time performance, texture-less environments, and dynamic environments. Visual-inertial fusion and learning-based enhancement are discussed for UAV localization and perception to illustrate their role in UAV applications. Subsequently, the trend of UAV localization and perception is shown. The algorithm components, camera configuration, and data processing methods are also introduced to give comprehensive preliminaries. In this paper, we provide coverage of visual SLAM and its related technologies over the past decade, with a specific focus on their applications in autonomous UAV applications. We summarize the current research, reveal potential problems, and outline future trends from academic and engineering perspectives.
Collapse
Affiliation(s)
- Licong Zhuang
- Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Yutang Street, Guangming District, Shenzhen 518132, China; (L.Z.); (X.Z.); (C.T.)
| | - Xiaorong Zhong
- Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Yutang Street, Guangming District, Shenzhen 518132, China; (L.Z.); (X.Z.); (C.T.)
| | - Linjie Xu
- The College of Civil and Transportation Engineering, Shenzhen University, 3688 Nanhai Avenue, Nanshan District, Shenzhen 518060, China;
| | - Chunbao Tian
- Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Yutang Street, Guangming District, Shenzhen 518132, China; (L.Z.); (X.Z.); (C.T.)
| | - Wenshuai Yu
- The College of Civil and Transportation Engineering, Shenzhen University, 3688 Nanhai Avenue, Nanshan District, Shenzhen 518060, China;
| |
Collapse
|
4
|
Lahemer ESF, Rad A. An Audio-Based SLAM for Indoor Environments: A Robotic Mixed Reality Presentation. SENSORS (BASEL, SWITZERLAND) 2024; 24:2796. [PMID: 38732904 PMCID: PMC11086165 DOI: 10.3390/s24092796] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/07/2024] [Revised: 04/21/2024] [Accepted: 04/25/2024] [Indexed: 05/13/2024]
Abstract
In this paper, we present a novel approach referred to as the audio-based virtual landmark-based HoloSLAM. This innovative method leverages a single sound source and microphone arrays to estimate the voice-printed speaker's direction. The system allows an autonomous robot equipped with a single microphone array to navigate within indoor environments, interact with specific sound sources, and simultaneously determine its own location while mapping the environment. The proposed method does not require multiple audio sources in the environment nor sensor fusion to extract pertinent information and make accurate sound source estimations. Furthermore, the approach incorporates Robotic Mixed Reality using Microsoft HoloLens to superimpose landmarks, effectively mitigating the audio landmark-related issues of conventional audio-based landmark SLAM, particularly in situations where audio landmarks cannot be discerned, are limited in number, or are completely missing. The paper also evaluates an active speaker detection method, demonstrating its ability to achieve high accuracy in scenarios where audio data are the sole input. Real-time experiments validate the effectiveness of this method, emphasizing its precision and comprehensive mapping capabilities. The results of these experiments showcase the accuracy and efficiency of the proposed system, surpassing the constraints associated with traditional audio-based SLAM techniques, ultimately leading to a more detailed and precise mapping of the robot's surroundings.
Collapse
Affiliation(s)
- Elfituri S. F. Lahemer
- Autonomous and Intelligent Systems Laboratory, School of Mechatronic Systems Engineering, Simon Fraser University, Surrey, BC V3T 0A3, Canada;
| | | |
Collapse
|
5
|
Al-Tawil B, Hempel T, Abdelrahman A, Al-Hamadi A. A review of visual SLAM for robotics: evolution, properties, and future applications. Front Robot AI 2024; 11:1347985. [PMID: 38686339 PMCID: PMC11056647 DOI: 10.3389/frobt.2024.1347985] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Accepted: 02/20/2024] [Indexed: 05/02/2024] Open
Abstract
Visual simultaneous localization and mapping (V-SLAM) plays a crucial role in the field of robotic systems, especially for interactive and collaborative mobile robots. The growing reliance on robotics has increased complexity in task execution in real-world applications. Consequently, several types of V-SLAM methods have been revealed to facilitate and streamline the functions of robots. This work aims to showcase the latest V-SLAM methodologies, offering clear selection criteria for researchers and developers to choose the right approach for their robotic applications. It chronologically presents the evolution of SLAM methods, highlighting key principles and providing comparative analyses between them. The paper focuses on the integration of the robotic ecosystem with a robot operating system (ROS) as Middleware, explores essential V-SLAM benchmark datasets, and presents demonstrative figures for each method's workflow.
Collapse
Affiliation(s)
- Basheer Al-Tawil
- Institute for Information Technology and Communications, Otto-von-Guericke-University, Magdeburg, Germany
| | | | | | | |
Collapse
|
6
|
Morueta-Holme N, Iversen LL, Corcoran D, Rahbek C, Normand S. Unlocking ground-based imagery for habitat mapping. Trends Ecol Evol 2024; 39:349-358. [PMID: 38087707 DOI: 10.1016/j.tree.2023.11.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Revised: 11/06/2023] [Accepted: 11/14/2023] [Indexed: 04/05/2024]
Abstract
Fine-grained environmental data across large extents are needed to resolve the processes that impact species communities from local to global scales. Ground-based images (GBIs) have the potential to capture habitat complexity at biologically relevant spatial and temporal resolutions. Moving beyond existing applications of GBIs for species identification and monitoring ecological change from repeat photography, we describe promising approaches to habitat mapping, leveraging multimodal data and computer vision. We illustrate empirically how GBIs can be applied to predict distributions of species at fine scales along Street View routes, or to automatically classify and quantify habitat features. Further, we outline future research avenues using GBIs that can bring a leap forward in analyses for ecology and conservation with this underused resource.
Collapse
Affiliation(s)
- N Morueta-Holme
- Center for Macroecology, Evolution and Climate, Globe Institute, University of Copenhagen, Copenhagen, Denmark.
| | - L L Iversen
- Department of Biology, McGill University, Montréal, Québec, H3A 1B1, Canada
| | - D Corcoran
- Section for Ecoinformatics & Biodiversity, Department of Biology, Aarhus University, Aarhus, Denmark; Center for Sustainable Landscapes under Global Change, Department of Biology, Aarhus University, Aarhus, Denmark
| | - C Rahbek
- Center for Macroecology, Evolution and Climate, Globe Institute, University of Copenhagen, Copenhagen, Denmark; Center for Global Mountain Biodiversity, Globe Institute, University of Copenhagen, Copenhagen, Denmark; Institute of Ecology, Peking University, Beijing, China; Danish Institute for Advanced Study, University of Southern Denmark, Odense, Denmark
| | - S Normand
- Section for Ecoinformatics & Biodiversity, Department of Biology, Aarhus University, Aarhus, Denmark; Center for Sustainable Landscapes under Global Change, Department of Biology, Aarhus University, Aarhus, Denmark; Center for Landscape Research in Sustainable Agricultural Futures, Department of Biology, Aarhus University, Aarhus, Denmark
| |
Collapse
|
7
|
Zheng F, Zhou L, Lin W, Liu J, Sun L. LRPL-VIO: A Lightweight and Robust Visual-Inertial Odometry with Point and Line Features. SENSORS (BASEL, SWITZERLAND) 2024; 24:1322. [PMID: 38400480 PMCID: PMC10892506 DOI: 10.3390/s24041322] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Revised: 01/26/2024] [Accepted: 02/15/2024] [Indexed: 02/25/2024]
Abstract
Visual-inertial odometry (VIO) algorithms, fusing various features such as points and lines, are able to improve their performance in challenging scenes while the running time severely increases. In this paper, we propose a novel lightweight point-line visual-inertial odometry algorithm to solve this problem, called LRPL-VIO. Firstly, a fast line matching method is proposed based on the assumption that the photometric values of endpoints and midpoints are invariant between consecutive frames, which greatly reduces the time consumption of the front end. Then, an efficient filter-based state estimation framework is designed to finish information fusion (point, line, and inertial). Fresh measurements of line features with good tracking quality are selected for state estimation using a unique feature selection scheme, which improves the efficiency of the proposed algorithm. Finally, validation experiments are conducted on public datasets and in real-world tests to evaluate the performance of LRPL-VIO and the results show that we outperform other state-of-the-art algorithms especially in terms of speed and robustness.
Collapse
Affiliation(s)
- Feixiang Zheng
- College of Artificial Intelligence, Nankai University, Tianjin 300350, China; (F.Z.); (L.Z.); (J.L.)
| | - Lu Zhou
- College of Artificial Intelligence, Nankai University, Tianjin 300350, China; (F.Z.); (L.Z.); (J.L.)
| | - Wanbiao Lin
- Shenzhen Research Institute, Nankai University, Shenzhen 518081, China;
| | - Jingyang Liu
- College of Artificial Intelligence, Nankai University, Tianjin 300350, China; (F.Z.); (L.Z.); (J.L.)
| | - Lei Sun
- College of Artificial Intelligence, Nankai University, Tianjin 300350, China; (F.Z.); (L.Z.); (J.L.)
| |
Collapse
|
8
|
Liu Y, Wang S, Xie Y, Xiong T, Wu M. A Review of Sensing Technologies for Indoor Autonomous Mobile Robots. SENSORS (BASEL, SWITZERLAND) 2024; 24:1222. [PMID: 38400380 PMCID: PMC10893033 DOI: 10.3390/s24041222] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Revised: 02/04/2024] [Accepted: 02/12/2024] [Indexed: 02/25/2024]
Abstract
As a fundamental issue in robotics academia and industry, indoor autonomous mobile robots (AMRs) have been extensively studied. For AMRs, it is crucial to obtain information about their working environment and themselves, which can be realized through sensors and the extraction of corresponding information from the measurements of these sensors. The application of sensing technologies can enable mobile robots to perform localization, mapping, target or obstacle recognition, and motion tasks, etc. This paper reviews sensing technologies for autonomous mobile robots in indoor scenes. The benefits and potential problems of using a single sensor in application are analyzed and compared, and the basic principles and popular algorithms used in processing these sensor data are introduced. In addition, some mainstream technologies of multi-sensor fusion are introduced. Finally, this paper discusses the future development trends in the sensing technology for autonomous mobile robots in indoor scenes, as well as the challenges in the practical application environments.
Collapse
Affiliation(s)
| | | | - Yuanlong Xie
- School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China; (Y.L.); (S.W.); (T.X.); (M.W.)
| | | | | |
Collapse
|
9
|
Brata KC, Funabiki N, Panduman YYF, Fajrianti ED. An Enhancement of Outdoor Location-Based Augmented Reality Anchor Precision through VSLAM and Google Street View. SENSORS (BASEL, SWITZERLAND) 2024; 24:1161. [PMID: 38400319 PMCID: PMC10893312 DOI: 10.3390/s24041161] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/22/2023] [Revised: 01/12/2024] [Accepted: 02/07/2024] [Indexed: 02/25/2024]
Abstract
Outdoor Location-Based Augmented Reality (LAR) applications require precise positioning for seamless integrations of virtual content into immersive experiences. However, common solutions in outdoor LAR applications rely on traditional smartphone sensor fusion methods, such as the Global Positioning System (GPS) and compasses, which often lack the accuracy needed for precise AR content alignments. In this paper, we introduce an innovative approach to enhance LAR anchor precision in outdoor environments. We leveraged Visual Simultaneous Localization and Mapping (VSLAM) technology, in combination with innovative cloud-based methodologies, and harnessed the extensive visual reference database of Google Street View (GSV), to address the accuracy limitation problems. For the evaluation, 10 Point of Interest (POI) locations were used as anchor point coordinates in the experiments. We compared the accuracies between our approach and the common sensor fusion LAR solution comprehensively involving accuracy benchmarking and running load performance testing. The results demonstrate substantial enhancements in overall positioning accuracies compared to conventional GPS-based approaches for aligning AR anchor content in the real world.
Collapse
Affiliation(s)
- Komang Candra Brata
- Graduate School of Natural Science and Technology, Okayama University, Okayama 700-8530, Japan; (Y.Y.F.P.); (E.D.F.)
- Department of Informatics Engineering, Universitas Brawijaya, Malang 65145, Indonesia
| | - Nobuo Funabiki
- Graduate School of Natural Science and Technology, Okayama University, Okayama 700-8530, Japan; (Y.Y.F.P.); (E.D.F.)
| | | | - Evianita Dewi Fajrianti
- Graduate School of Natural Science and Technology, Okayama University, Okayama 700-8530, Japan; (Y.Y.F.P.); (E.D.F.)
| |
Collapse
|
10
|
Moon S, Lee M. Analyzing the Impact of Objects in an Image on Location Estimation Accuracy in Visual Localization. SENSORS (BASEL, SWITZERLAND) 2024; 24:816. [PMID: 38339532 PMCID: PMC10857014 DOI: 10.3390/s24030816] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Revised: 01/22/2024] [Accepted: 01/24/2024] [Indexed: 02/12/2024]
Abstract
Visual localization refers to the process of determining an observer's pose by analyzing the spatial relationships between a query image and a pre-existing set of images. In this procedure, matched visual features between images are identified and utilized for pose estimation; consequently, the accuracy of the estimation heavily relies on the precision of feature matching. Incorrect feature matchings, such as those between different objects and/or different points within an object in an image, should thus be avoided. In this paper, our initial evaluation focused on gauging the reliability of each object class within image datasets concerning pose estimation accuracy. This assessment revealed the building class to be reliable, while humans exhibited unreliability across diverse locations. The subsequent study delved deeper into the degradation of pose estimation accuracy by artificially increasing the proportion of the unreliable object-humans. The findings revealed a noteworthy decline started when the average proportion of the humans in the images exceeded 20%. We discuss the results and implications for dataset construction for visual localization.
Collapse
Affiliation(s)
- Sungho Moon
- Department of Information Convergence Engineering, Pusan National University, Busan 46241, Republic of Korea;
| | - Myungho Lee
- School of Computer Science and Engineering, Pusan National University, Busan 46241, Republic of Korea
| |
Collapse
|
11
|
Wang Y, Zhang Y, Hu L, Wang W, Ge G, Tan S. A Semantic Topology Graph to Detect Re-Localization and Loop Closure of the Visual Simultaneous Localization and Mapping System in a Dynamic Environment. SENSORS (BASEL, SWITZERLAND) 2023; 23:8445. [PMID: 37896538 PMCID: PMC10611121 DOI: 10.3390/s23208445] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Revised: 10/02/2023] [Accepted: 10/10/2023] [Indexed: 10/29/2023]
Abstract
Simultaneous localization and mapping (SLAM) plays a crucial role in the field of intelligent mobile robots. However, the traditional Visual SLAM (VSLAM) framework is based on strong assumptions about static environments, which are not applicable to dynamic real-world environments. The correctness of re-localization and recall of loop closure detection are both lower when the mobile robot loses frames in a dynamic environment. Thus, in this paper, the re-localization and loop closure detection method with a semantic topology graph based on ORB-SLAM2 is proposed. First, we use YOLOv5 for object detection and label the recognized dynamic and static objects. Secondly, the topology graph is constructed using the position information of static objects in space. Then, we propose a weight expression for the topology graph to calculate the similarity of topology in different keyframes. Finally, the re-localization and loop closure detection are determined based on the value of topology similarity. Experiments on public datasets show that the semantic topology graph is effective in improving the correct rate of re-localization and the accuracy of loop closure detection in a dynamic environment.
Collapse
Affiliation(s)
- Yang Wang
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| | - Yi Zhang
- Advanced Manufacturing and Automatization Engineering Laboratory, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
| | - Lihe Hu
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| | - Wei Wang
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| | - Gengyu Ge
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| | - Shuyi Tan
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| |
Collapse
|
12
|
Kim J, Kim C, Yoon S, Choi T, Sull S. RBF-Based Camera Model Based on a Ray Constraint to Compensate for Refraction Error. SENSORS (BASEL, SWITZERLAND) 2023; 23:8430. [PMID: 37896523 PMCID: PMC10610825 DOI: 10.3390/s23208430] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/16/2023] [Revised: 10/01/2023] [Accepted: 10/10/2023] [Indexed: 10/29/2023]
Abstract
A camera equipped with a transparent shield can be modeled using the pinhole camera model and residual error vectors defined by the difference between the estimated ray from the pinhole camera model and the actual three-dimensional (3D) point. To calculate the residual error vectors, we employ sparse calibration data consisting of 3D points and their corresponding 2D points on the image. However, the observation noise and sparsity of the 3D calibration points pose challenges in determining the residual error vectors. To address this, we first fit Gaussian Process Regression (GPR) operating robustly against data noise to the observed residual error vectors from the sparse calibration data to obtain dense residual error vectors. Subsequently, to improve performance in unobserved areas due to data sparsity, we use an additional constraint; the 3D points on the estimated ray should be projected to one 2D image point, called the ray constraint. Finally, we optimize the radial basis function (RBF)-based regression model to reduce the residual error vector differences with GPR at the predetermined dense set of 3D points while reflecting the ray constraint. The proposed RBF-based camera model reduces the error of the estimated rays by 6% on average and the reprojection error by 26% on average.
Collapse
Affiliation(s)
| | | | | | | | - Sanghoon Sull
- School of Electrical Engineering, Korea University, Seoul 02841, Republic of Korea; (J.K.); (C.K.); (S.Y.); (T.C.)
| |
Collapse
|
13
|
Wang S, Su D, Li M, Jiang Y, Zhang L, Yan H, Hu N, Tan Y. LFSD: a VSLAM dataset with plant detection and tracking in lettuce farm. FRONTIERS IN PLANT SCIENCE 2023; 14:1175743. [PMID: 37705704 PMCID: PMC10497103 DOI: 10.3389/fpls.2023.1175743] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 08/01/2023] [Indexed: 09/15/2023]
Affiliation(s)
- Shuo Wang
- College of Engineering, China Agricultural University, Beijing, China
| | - Daobilige Su
- College of Engineering, China Agricultural University, Beijing, China
| | - Maofeng Li
- Beijing Zhong Nong LV Tong Agriculture Development LTD, Beijing, China
| | - Yiyu Jiang
- College of Engineering, China Agricultural University, Beijing, China
| | - Lina Zhang
- College of Engineering, China Agricultural University, Beijing, China
| | - Hao Yan
- College of Engineering, China Agricultural University, Beijing, China
| | - Nan Hu
- College of Engineering, China Agricultural University, Beijing, China
| | - Yu Tan
- College of Engineering, China Agricultural University, Beijing, China
| |
Collapse
|
14
|
Bavle H, Sanchez-Lopez JL, Cimarelli C, Tourani A, Voos H. From SLAM to Situational Awareness: Challenges and Survey. SENSORS (BASEL, SWITZERLAND) 2023; 23:4849. [PMID: 37430762 DOI: 10.3390/s23104849] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Revised: 04/27/2023] [Accepted: 05/13/2023] [Indexed: 07/12/2023]
Abstract
The capability of a mobile robot to efficiently and safely perform complex missions is limited by its knowledge of the environment, namely the situation. Advanced reasoning, decision-making, and execution skills enable an intelligent agent to act autonomously in unknown environments. Situational Awareness (SA) is a fundamental capability of humans that has been deeply studied in various fields, such as psychology, military, aerospace, and education. Nevertheless, it has yet to be considered in robotics, which has focused on single compartmentalized concepts such as sensing, spatial perception, sensor fusion, state estimation, and Simultaneous Localization and Mapping (SLAM). Hence, the present research aims to connect the broad multidisciplinary existing knowledge to pave the way for a complete SA system for mobile robotics that we deem paramount for autonomy. To this aim, we define the principal components to structure a robotic SA and their area of competence. Accordingly, this paper investigates each aspect of SA, surveying the state-of-the-art robotics algorithms that cover them, and discusses their current limitations. Remarkably, essential aspects of SA are still immature since the current algorithmic development restricts their performance to only specific environments. Nevertheless, Artificial Intelligence (AI), particularly Deep Learning (DL), has brought new methods to bridge the gap that maintains these fields apart from the deployment to real-world scenarios. Furthermore, an opportunity has been discovered to interconnect the vastly fragmented space of robotic comprehension algorithms through the mechanism of Situational Graph (S-Graph), a generalization of the well-known scene graph. Therefore, we finally shape our vision for the future of robotic situational awareness by discussing interesting recent research directions.
Collapse
Affiliation(s)
- Hriday Bavle
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
| | - Jose Luis Sanchez-Lopez
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
| | - Claudio Cimarelli
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
| | - Ali Tourani
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
| | - Holger Voos
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
- Department of Engineering, Faculty of Science, Technology, and Medicine (FSTM), University of Luxembourg, 1359 Luxembourg, Luxembourg
| |
Collapse
|