1
|
Siva S, Zhang H. Robot perceptual adaptation to environment changes for long-term human teammate following. Int J Rob Res 2022. [DOI: 10.1177/0278364919896625] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Perception is one of the several fundamental abilities required by robots, and it also poses significant challenges, especially in real-world field applications. Long-term autonomy introduces additional difficulties to robot perception, including short- and long-term changes of the robot operation environment (e.g., lighting changes). In this article, we propose an innovative human-inspired approach named robot perceptual adaptation (ROPA) that is able to calibrate perception according to the environment context, which enables perceptual adaptation in response to environmental variations. ROPA jointly performs feature learning, sensor fusion, and perception calibration under a unified regularized optimization framework. We also implement a new algorithm to solve the formulated optimization problem, which has a theoretical guarantee to converge to the optimal solution. In addition, we collect a large-scale dataset from physical robots in the field, called perceptual adaptation to environment changes (PEAC), with the aim to benchmark methods for robot adaptation to short-term and long-term, and fast and gradual lighting changes for human detection based upon different feature modalities extracted from color and depth sensors. Utilizing the PEAC dataset, we conduct extensive experiments in the application of human recognition and following in various scenarios to evaluate ROPA. Experimental results have validated that the ROPA approach obtains promising performance in terms of accuracy and efficiency, and effectively adapts robot perception to address short-term and long-term lighting changes in human detection and following applications.
Collapse
Affiliation(s)
- Sriram Siva
- Human-Centered Robotics Lab, Colorado School of Mines, Golden, CO, USA
| | - Hao Zhang
- Human-Centered Robotics Lab, Colorado School of Mines, Golden, CO, USA
| |
Collapse
|
2
|
Zhang G, Moyes H, Chen Y. Smart three-dimensional processing of unconstrained cave scans using small unmanned aerial systems and red, green, and blue-depth cameras. INT J ADV ROBOT SYST 2022. [DOI: 10.1177/17298814211017728] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022] Open
Abstract
This article focuses on a novel three-dimensional reconstruction system that maps large archeological caves using data collected by a small unmanned aircraft system with red, green, and blue-depth cameras. Cave sites often contain the best-preserved material in the archeological record. Yet few sites are fully mapped. Large caves environment usually contains complex geometric structures and objects, which must be scanned with long overlapped camera trajectories for better coverage. Due to the error in camera tracking of such scanning, reconstruction results often contain flaws and mismatches. To solve this problem, we propose a framework for surface loop closure, where loops are detected with a compute unified device architecture accelerated point cloud registration algorithm. After a loop is detected, a novel surface loop filtering method is proposed for robust loop optimization. This loop filtering method is robust to different scan patterns and can cope with tracking failure recovery so that there is more flexibility for unmanned aerial vehicles to fly and record data. We run experiments on public data sets and our cave data set for analysis and robustness tests. Experiments show that our system produces improved results on baseline methods.
Collapse
Affiliation(s)
- Guoxiang Zhang
- Mechatronics, Embedded Systems and Automation (MESA) Lab, University of California, Merced, CA, USA
| | - Holley Moyes
- School of Social Sciences, Humanities, and Arts, University of California, Merced, CA, USA
| | - YangQuan Chen
- Mechatronics, Embedded Systems and Automation (MESA) Lab, University of California, Merced, CA, USA
| |
Collapse
|
3
|
Yadav R, Kala R. Fusion of visual odometry and place recognition for SLAM in extreme conditions. APPL INTELL 2022. [DOI: 10.1007/s10489-021-03050-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
4
|
Manzoor S, Joo SH, Kim EJ, Bae SH, In GG, Pyo JW, Kuc TY. 3D Recognition Based on Sensor Modalities for Robotic Systems: A Survey. SENSORS (BASEL, SWITZERLAND) 2021; 21:7120. [PMID: 34770429 PMCID: PMC8587961 DOI: 10.3390/s21217120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/26/2021] [Revised: 10/17/2021] [Accepted: 10/20/2021] [Indexed: 11/16/2022]
Abstract
3D visual recognition is a prerequisite for most autonomous robotic systems operating in the real world. It empowers robots to perform a variety of tasks, such as tracking, understanding the environment, and human-robot interaction. Autonomous robots equipped with 3D recognition capability can better perform their social roles through supportive task assistance in professional jobs and effective domestic services. For active assistance, social robots must recognize their surroundings, including objects and places to perform the task more efficiently. This article first highlights the value-centric role of social robots in society by presenting recently developed robots and describes their main features. Instigated by the recognition capability of social robots, we present the analysis of data representation methods based on sensor modalities for 3D object and place recognition using deep learning models. In this direction, we delineate the research gaps that need to be addressed, summarize 3D recognition datasets, and present performance comparisons. Finally, a discussion of future research directions concludes the article. This survey is intended to show how recent developments in 3D visual recognition based on sensor modalities using deep-learning-based approaches can lay the groundwork to inspire further research and serves as a guide to those who are interested in vision-based robotics applications.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Tae-Yong Kuc
- Department of Electrical and Computer Engineering, College of Information and Communication Engineering, Sungkyunkwan University, Suwon 16419, Korea; (S.M.); (S.-H.J.); (E.-J.K.); (S.-H.B.); (G.-G.I.); (J.-W.P.)
| |
Collapse
|
5
|
Chen S, Wu J, Lu Q, Wang Y, Lin Z. Cross-scene loop-closure detection with continual learning for visual simultaneous localization and mapping. INT J ADV ROBOT SYST 2021. [DOI: 10.1177/17298814211050560] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
Abstract
Humans maintain good memory and recognition capability of previous environments when they are learning about new ones. Thus humans are able to continually learn and increase their experience. It is also obvious importance for autonomous mobile robot. The simultaneous localization and mapping system plays an important role in localization and navigation of robot. The loop-closure detection method is an indispensable part of the relocation and map construction, which is critical to correct mappoint errors of simultaneous localization and mapping. Existing visual loop-closure detection methods based on deep learning are not capable of continual learning in terms of cross-scene environment, which bring a great limitation to the application scope. In this article, we propose a novel end-to-end loop-closure detection method based on continual learning, which can effectively suppress the decline of the memory capability of simultaneous localization and mapping system by introducing firstly the orthogonal projection operator into the loop-closure detection to overcome the catastrophic forgetting problem of mobile robot in large-scale and multi-scene environments. Based on the three scenes from public data sets, the experimental results show that the proposed method has a strong capability of continual learning in the cross-scene environment where existing state-of-the-art methods fail.
Collapse
Affiliation(s)
- Shilang Chen
- School of Mechatronic Engineering and Automation, Foshan University, Foshan, China
| | - Junjun Wu
- School of Mechatronic Engineering and Automation, Foshan University, Foshan, China
| | - Qinghua Lu
- School of Mechatronic Engineering and Automation, Foshan University, Foshan, China
| | - Yanran Wang
- Department of Computer Science, Jinan University, Guangzhou, China
| | - Zeqin Lin
- School of Mechatronic Engineering and Automation, Foshan University, Foshan, China
| |
Collapse
|
6
|
Lin S, Wang J, Xu M, Zhao H, Chen Z. Topology Aware Object-Level Semantic Mapping Towards More Robust Loop Closure. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3097242] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
7
|
Reily B, Gao P, Han F, Wang H, Zhang H. Real-time recognition of team behaviors by multisensory graph-embedded robot learning. Int J Rob Res 2021. [DOI: 10.1177/02783649211043155] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Awareness of team behaviors (e.g., individual activities and team intents) plays a critical role in human–robot teaming. Autonomous robots need to be aware of the overall intent of the team they are collaborating with in order to effectively aid their human peers or augment the team’s capabilities. Team intents encode the goal of the team, which cannot be simply identified from a collection of individual activities. Instead, teammate relationships must also be encoded for team intent recognition. In this article, we introduce a novel representation learning approach to recognizing team intent awareness in real-time, based upon both individual human activities and the relationship between human peers in the team. Our approach formulates the task of robot learning for team intent recognition as a joint regularized optimization problem, which encodes individual activities as latent variables and represents teammate relationships through graph embedding. In addition, we design a new algorithm to efficiently solve the formulated regularized optimization problem, which possesses a theoretical guarantee to converge to the optimal solution. To evaluate our approach’s performance on team intent recognition, we test our approach on a public benchmark group activity dataset and a multisensory underground search and rescue team behavior dataset newly collected from robots in an underground environment, as well as perform a proof-of-concept case study on a physical robot. The experimental results have demonstrated both the superior accuracy of our proposed approach and its suitability for real-time applications on mobile robots.
Collapse
Affiliation(s)
- Brian Reily
- Human-Centered Robotics Lab, Colorado School of Mines, Golden, CO, USA
| | - Peng Gao
- Human-Centered Robotics Lab, Colorado School of Mines, Golden, CO, USA
| | - Fei Han
- Human-Centered Robotics Lab, Colorado School of Mines, Golden, CO, USA
| | - Hua Wang
- Human-Centered Robotics Lab, Colorado School of Mines, Golden, CO, USA
| | - Hao Zhang
- Human-Centered Robotics Lab, Colorado School of Mines, Golden, CO, USA
| |
Collapse
|
8
|
Learning second-order statistics for place recognition based on robust covariance estimation of CNN features. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2020.02.001] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
9
|
Kazmi SMAM, Mertsching B. Detecting the Expectancy of a Place Using Nearby Context for Appearance-Based Mapping. IEEE T ROBOT 2019. [DOI: 10.1109/tro.2019.2926475] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
10
|
Multi-Process Fusion: Visual Place Recognition Using Multiple Image Processing Methods. IEEE Robot Autom Lett 2019. [DOI: 10.1109/lra.2019.2898427] [Citation(s) in RCA: 30] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
11
|
Han F, Beleidy SE, Wang H, Ye C, Zhang H. Learning of Holism-Landmark Graph Embedding for Place Recognition in Long-Term Autonomy. IEEE Robot Autom Lett 2018. [DOI: 10.1109/lra.2018.2856274] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
12
|
Sequence-based sparse optimization methods for long-term loop closure detection in visual SLAM. Auton Robots 2018. [DOI: 10.1007/s10514-018-9736-3] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
|
13
|
Real-Time Visual Place Recognition Based on Analyzing Distribution of Multi-scale CNN Landmarks. J INTELL ROBOT SYST 2018. [DOI: 10.1007/s10846-018-0804-x] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
|