1
|
Lee J, Bjelonic M, Reske A, Wellhausen L, Miki T, Hutter M. Learning robust autonomous navigation and locomotion for wheeled-legged robots. Sci Robot 2024; 9:eadi9641. [PMID: 38657088 DOI: 10.1126/scirobotics.adi9641] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Accepted: 03/27/2024] [Indexed: 04/26/2024]
Abstract
Autonomous wheeled-legged robots have the potential to transform logistics systems, improving operational efficiency and adaptability in urban environments. Navigating urban environments, however, poses unique challenges for robots, necessitating innovative solutions for locomotion and navigation. These challenges include the need for adaptive locomotion across varied terrains and the ability to navigate efficiently around complex dynamic obstacles. This work introduces a fully integrated system comprising adaptive locomotion control, mobility-aware local navigation planning, and large-scale path planning within the city. Using model-free reinforcement learning (RL) techniques and privileged learning, we developed a versatile locomotion controller. This controller achieves efficient and robust locomotion over various rough terrains, facilitated by smooth transitions between walking and driving modes. It is tightly integrated with a learned navigation controller through a hierarchical RL framework, enabling effective navigation through challenging terrain and various obstacles at high speed. Our controllers are integrated into a large-scale urban navigation system and validated by autonomous, kilometer-scale navigation missions conducted in Zurich, Switzerland, and Seville, Spain. These missions demonstrate the system's robustness and adaptability, underscoring the importance of integrated control systems in achieving seamless navigation in complex environments. Our findings support the feasibility of wheeled-legged robots and hierarchical RL for autonomous navigation, with implications for last-mile delivery and beyond.
Collapse
Affiliation(s)
- Joonho Lee
- Robotic Systems Lab, ETH Zurich, Zurich, Switzerland
| | | | | | | | - Takahiro Miki
- Robotic Systems Lab, ETH Zurich, Zurich, Switzerland
| | - Marco Hutter
- Robotic Systems Lab, ETH Zurich, Zurich, Switzerland
| |
Collapse
|
2
|
Chirayil Nandakumar S, Mitchell D, Erden MS, Flynn D, Lim T. Anomaly Detection Methods in Autonomous Robotic Missions. SENSORS (BASEL, SWITZERLAND) 2024; 24:1330. [PMID: 38400491 PMCID: PMC10892279 DOI: 10.3390/s24041330] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/24/2024] [Revised: 02/13/2024] [Accepted: 02/16/2024] [Indexed: 02/25/2024]
Abstract
Since 2015, there has been an increase in articles on anomaly detection in robotic systems, reflecting its growing importance in improving the robustness and reliability of the increasingly utilized autonomous robots. This review paper investigates the literature on the detection of anomalies in Autonomous Robotic Missions (ARMs). It reveals different perspectives on anomaly and juxtaposition to fault detection. To reach a consensus, we infer a unified understanding of anomalies that encapsulate their various characteristics observed in ARMs and propose a classification of anomalies in terms of spatial, temporal, and spatiotemporal elements based on their fundamental features. Further, the paper discusses the implications of the proposed unified understanding and classification in ARMs and provides future directions. We envisage a study surrounding the specific use of the term anomaly, and methods for their detection could contribute to and accelerate the research and development of a universal anomaly detection system for ARMs.
Collapse
Affiliation(s)
- Shivoh Chirayil Nandakumar
- School of Engineering and Physical Sciences, Heriot Watt University, Edinburgh EH14 4AS, UK; (S.C.N.); (T.L.)
| | - Daniel Mitchell
- James Watt School of Engineering, University of Glasgow, Glasgow G12 8QQ, UK; (D.M.); (D.F.)
| | - Mustafa Suphi Erden
- School of Engineering and Physical Sciences, Heriot Watt University, Edinburgh EH14 4AS, UK; (S.C.N.); (T.L.)
| | - David Flynn
- James Watt School of Engineering, University of Glasgow, Glasgow G12 8QQ, UK; (D.M.); (D.F.)
| | - Theodore Lim
- School of Engineering and Physical Sciences, Heriot Watt University, Edinburgh EH14 4AS, UK; (S.C.N.); (T.L.)
| |
Collapse
|
3
|
Sarkar S, Ganapathysubramanian B, Singh A, Fotouhi F, Kar S, Nagasubramanian K, Chowdhary G, Das SK, Kantor G, Krishnamurthy A, Merchant N, Singh AK. Cyber-agricultural systems for crop breeding and sustainable production. TRENDS IN PLANT SCIENCE 2024; 29:130-149. [PMID: 37648631 DOI: 10.1016/j.tplants.2023.08.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Revised: 07/19/2023] [Accepted: 08/03/2023] [Indexed: 09/01/2023]
Abstract
The cyber-agricultural system (CAS) represents an overarching framework of agriculture that leverages recent advances in ubiquitous sensing, artificial intelligence, smart actuators, and scalable cyberinfrastructure (CI) in both breeding and production agriculture. We discuss the recent progress and perspective of the three fundamental components of CAS - sensing, modeling, and actuation - and the emerging concept of agricultural digital twins (DTs). We also discuss how scalable CI is becoming a key enabler of smart agriculture. In this review we shed light on the significance of CAS in revolutionizing crop breeding and production by enhancing efficiency, productivity, sustainability, and resilience to changing climate. Finally, we identify underexplored and promising future directions for CAS research and development.
Collapse
Affiliation(s)
- Soumik Sarkar
- Department of Mechanical Engineering, Iowa State University, Ames, IA, USA; Department of Computer Science, Iowa State University, Ames, IA, USA.
| | - Baskar Ganapathysubramanian
- Department of Mechanical Engineering, Iowa State University, Ames, IA, USA; Department of Computer Science, Iowa State University, Ames, IA, USA
| | - Arti Singh
- Department of Agronomy, Iowa State University, Ames, IA, USA
| | - Fateme Fotouhi
- Department of Mechanical Engineering, Iowa State University, Ames, IA, USA; Department of Computer Science, Iowa State University, Ames, IA, USA
| | | | | | - Girish Chowdhary
- Department of Agricultural and Biological Engineering and Department of Computer Science, University of Illinois at Urbana Champaign, Champaign, Urbana, IL, USA
| | - Sajal K Das
- Department of Computer Science, Missouri University of Science and Technology, Rolla, MO, USA
| | - George Kantor
- Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA
| | | | - Nirav Merchant
- Data Science Institute, University of Arizona, Tucson, AZ, USA
| | - Asheesh K Singh
- Department of Agronomy, Iowa State University, Ames, IA, USA.
| |
Collapse
|
4
|
Zhang J, Wang Y, Jiang B, He H, Huang S, Wang C, Zhang Y, Han X, Guo D, He G, Ouyang M. Realistic fault detection of li-ion battery via dynamical deep learning. Nat Commun 2023; 14:5940. [PMID: 37741826 PMCID: PMC10517941 DOI: 10.1038/s41467-023-41226-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Accepted: 08/21/2023] [Indexed: 09/25/2023] Open
Abstract
Accurate evaluation of Li-ion battery (LiB) safety conditions can reduce unexpected cell failures, facilitate battery deployment, and promote low-carbon economies. Despite the recent progress in artificial intelligence, anomaly detection methods are not customized for or validated in realistic battery settings due to the complex failure mechanisms and the lack of real-world testing frameworks with large-scale datasets. Here, we develop a realistic deep-learning framework for electric vehicle (EV) LiB anomaly detection. It features a dynamical autoencoder tailored for dynamical systems and configured by social and financial factors. We test our detection algorithm on released datasets comprising over 690,000 LiB charging snippets from 347 EVs. Our model overcomes the limitations of state-of-the-art fault detection models, including deep learning ones. Moreover, it reduces the expected direct EV battery fault and inspection costs. Our work highlights the potential of deep learning in improving LiB safety and the significance of social and financial information in designing deep learning models.
Collapse
Affiliation(s)
- Jingzhao Zhang
- IIIS Tsinghua University, Beijing, China
- Shanghai Qizhi Institute, Shanghai, China
| | - Yanan Wang
- State Key Laboratory of Intelligent Green Vehicle and Mobility, School of Vehicle and Mobility, Tsinghua University, Beijing, China
| | - Benben Jiang
- Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Haowei He
- IIIS Tsinghua University, Beijing, China
| | - Shaobo Huang
- Beijing Circue Energy Technology Co. Ltd., Beijing, China
| | - Chen Wang
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China
| | - Yang Zhang
- Beijing Circue Energy Technology Co. Ltd., Beijing, China
| | - Xuebing Han
- State Key Laboratory of Intelligent Green Vehicle and Mobility, School of Vehicle and Mobility, Tsinghua University, Beijing, China
| | - Dongxu Guo
- State Key Laboratory of Intelligent Green Vehicle and Mobility, School of Vehicle and Mobility, Tsinghua University, Beijing, China
| | - Guannan He
- Department of Industrial Engineering and Management, College of Engineering, Peking University, Beijing, China.
- National Engineering Laboratory for Big Data Analysis and Applications, Peking University, Beijing, China.
- Peking University Changsha Institute for Computing and Digital Economy, Changsha, China.
| | - Minggao Ouyang
- State Key Laboratory of Intelligent Green Vehicle and Mobility, School of Vehicle and Mobility, Tsinghua University, Beijing, China.
| |
Collapse
|
5
|
Mantegazza D, Xhyra A, Gambardella LM, Giusti A, Guzzi J. Hazards&Robots: A dataset for visual anomaly detection in robotics. Data Brief 2023; 48:109264. [PMID: 37383812 PMCID: PMC10294035 DOI: 10.1016/j.dib.2023.109264] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Revised: 05/01/2023] [Accepted: 05/19/2023] [Indexed: 06/30/2023] Open
Abstract
We propose Hazards&Robots, a dataset for Visual Anomaly Detection in Robotics. The dataset is composed of 324,408 RGB frames, and corresponding feature vectors; it contains 145,470 normal frames and 178,938 anomalous ones categorized in 20 different anomaly classes. The dataset can be used to train and test current and novel visual anomaly detection methods such as those based on deep learning vision models. The data is recorded with a DJI Robomaster S1 front facing camera. The ground robot, controlled by a human operator, traverses university corridors. Considered anomalies include presence of humans, unexpected objects on the floor, defects to the robot. Preliminary versions of the dataset are used in [1,3]. This version is available at [12].
Collapse
Affiliation(s)
- Dario Mantegazza
- IDSIA - Dalle Molle Institute for Artificial Intelligence USI-SUPSI, Polo universitario Lugano - Campus Est, Via la Santa 1, CH-6962 Lugano-Viganello
| | - Alind Xhyra
- IDSIA - Dalle Molle Institute for Artificial Intelligence USI-SUPSI, Polo universitario Lugano - Campus Est, Via la Santa 1, CH-6962 Lugano-Viganello
| | - Luca M. Gambardella
- IDSIA - Dalle Molle Institute for Artificial Intelligence USI-SUPSI, Polo universitario Lugano - Campus Est, Via la Santa 1, CH-6962 Lugano-Viganello
| | - Alessandro Giusti
- IDSIA - Dalle Molle Institute for Artificial Intelligence USI-SUPSI, Polo universitario Lugano - Campus Est, Via la Santa 1, CH-6962 Lugano-Viganello
| | - Jérôme Guzzi
- IDSIA - Dalle Molle Institute for Artificial Intelligence USI-SUPSI, Polo universitario Lugano - Campus Est, Via la Santa 1, CH-6962 Lugano-Viganello
| |
Collapse
|
6
|
Levine S, Shah D. Learning robotic navigation from experience: principles, methods and recent results. Philos Trans R Soc Lond B Biol Sci 2023; 378:20210447. [PMID: 36511408 PMCID: PMC9745865 DOI: 10.1098/rstb.2021.0447] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2022] [Accepted: 06/13/2022] [Indexed: 12/15/2022] Open
Abstract
Navigation is one of the most heavily studied problems in robotics and is conventionally approached as a geometric mapping and planning problem. However, real-world navigation presents a complex set of physical challenges that defies simple geometric abstractions. Machine learning offers a promising way to go beyond geometry and conventional planning, allowing for navigational systems that make decisions based on actual prior experience. Such systems can reason about traversability in ways that go beyond geometry, accounting for the physical outcomes of their actions and exploiting patterns in real-world environments. They can also improve as more data is collected, potentially providing a powerful network effect. In this article, we present a general toolkit for experiential learning of robotic navigation skills that unifies several recent approaches, describe the underlying design principles, summarize experimental results from several of our recent papers, and discuss open problems and directions for future work. This article is part of the theme issue 'New approaches to 3D vision'.
Collapse
Affiliation(s)
- Sergey Levine
- Berkeley AI Research (BAIR), UC Berkeley, 2121 Berkeley Way, Berkeley, CA 94704, USA
| | - Dhruv Shah
- Berkeley AI Research (BAIR), UC Berkeley, 2121 Berkeley Way, Berkeley, CA 94704, USA
| |
Collapse
|
7
|
Maziarka L, Smieja M, Sendera M, Struski L, Tabor J, Spurek P. OneFlow: One-Class Flow for Anomaly Detection Based on a Minimal Volume Region. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:8508-8519. [PMID: 34460365 DOI: 10.1109/tpami.2021.3108223] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
We propose OneFlow - a flow-based one-class classifier for anomaly (outlier) detection that finds a minimal volume bounding region. Contrary to density-based methods, OneFlow is constructed in such a way that its result typically does not depend on the structure of outliers. This is caused by the fact that during training the gradient of the cost function is propagated only over the points located near to the decision boundary (behavior similar to the support vectors in SVM). The combination of flow models and a Bernstein quantile estimator allows OneFlow to find a parametric form of bounding region, which can be useful in various applications including describing shapes from 3D point clouds. Experiments show that the proposed model outperforms related methods on real-world anomaly detection problems.
Collapse
|
8
|
Hsu KC, Ren AZ, Nguyen DP, Majumdar A, Fisac JF. Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees. ARTIF INTELL 2022. [DOI: 10.1016/j.artint.2022.103811] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
9
|
Nguyen TK, Nguyen PTT, Nguyen DD, Kuo CH. Effective Free-Driving Region Detection for Mobile Robots by Uncertainty Estimation Using RGB-D Data. SENSORS (BASEL, SWITZERLAND) 2022; 22:4751. [PMID: 35808244 PMCID: PMC9268933 DOI: 10.3390/s22134751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/30/2022] [Revised: 06/17/2022] [Accepted: 06/22/2022] [Indexed: 01/27/2023]
Abstract
Accurate segmentation of drivable areas and road obstacles is critical for autonomous mobile robots to navigate safely in indoor and outdoor environments. With the fast advancement of deep learning, mobile robots may now perform autonomous navigation based on what they learned in the learning phase. On the other hand, existing techniques often have low performance when confronted with complex situations since unfamiliar objects are not included in the training dataset. Additionally, the use of a large amount of labeled data is generally essential for training deep neural networks to achieve good performance, which is time-consuming and labor-intensive. Thus, this paper presents a solution to these issues by proposing a self-supervised learning method for the drivable areas and road anomaly segmentation. First, we propose the Automatic Generating Segmentation Label (AGSL) framework, which is an efficient system automatically generating segmentation labels for drivable areas and road anomalies by finding dissimilarities between the input and resynthesized image and localizing obstacles in the disparity map. Then, we train RGB-D datasets with a semantic segmentation network using self-generated ground truth labels derived from our method (AGSL labels) to get the pre-trained model. The results showed that our AGSL achieved high performance in labeling evaluation, and the pre-trained model also obtains certain confidence in real-time segmentation application on mobile robots.
Collapse
Affiliation(s)
- Toan-Khoa Nguyen
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei 106335, Taiwan; (T.-K.N.); (P.T.-T.N.); (D.-D.N.)
| | - Phuc Thanh-Thien Nguyen
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei 106335, Taiwan; (T.-K.N.); (P.T.-T.N.); (D.-D.N.)
| | - Dai-Dong Nguyen
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei 106335, Taiwan; (T.-K.N.); (P.T.-T.N.); (D.-D.N.)
| | - Chung-Hsien Kuo
- Department of Mechanical Engineering, National Taiwan University, Taipei 106319, Taiwan
| |
Collapse
|
10
|
Ji T, Sivakumar AN, Chowdhary G, Driggs-Campbell K. Proactive Anomaly Detection for Robot Navigation With Multi-Sensor Fusion. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3153989] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
- Tianchen Ji
- Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, Champaign, IL, USA
| | | | - Girish Chowdhary
- Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, Champaign, IL, USA
| | | |
Collapse
|
11
|
Gasparino MV, Sivakumar AN, Liu Y, Velasquez AEB, Higuti VAH, Rogers J, Tran H, Chowdhary G. WayFAST: Navigation With Predictive Traversability in the Field. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3193464] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Affiliation(s)
- Mateus V. Gasparino
- Field Robotics Engineering and Science Hub (FRESH), University of Illinois at Urbana-Champaign (UIUC), IL, USA
| | - Arun N. Sivakumar
- Field Robotics Engineering and Science Hub (FRESH), University of Illinois at Urbana-Champaign (UIUC), IL, USA
| | - Yixiao Liu
- Field Robotics Engineering and Science Hub (FRESH), University of Illinois at Urbana-Champaign (UIUC), IL, USA
| | - Andres E. B. Velasquez
- Field Robotics Engineering and Science Hub (FRESH), University of Illinois at Urbana-Champaign (UIUC), IL, USA
| | | | | | - Huy Tran
- Dept. of Aerospace Engineering, UIUC, IL, USA
| | - Girish Chowdhary
- Field Robotics Engineering and Science Hub (FRESH), University of Illinois at Urbana-Champaign (UIUC), IL, USA
| |
Collapse
|
12
|
Mantegazza D, Giusti A, Gambardella LM, Guzzi J. An Outlier Exposure Approach to Improve Visual Anomaly Detection Performance for Mobile Robots. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3192794] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Affiliation(s)
- Dario Mantegazza
- Dalle Molle Institute for Artificial Intelligence (IDSIA), USI-SUPSI, Lugano, Switzerland
| | - Alessandro Giusti
- Dalle Molle Institute for Artificial Intelligence (IDSIA), USI-SUPSI, Lugano, Switzerland
| | - Luca Maria Gambardella
- Dalle Molle Institute for Artificial Intelligence (IDSIA), USI-SUPSI, Lugano, Switzerland
| | - Jerome Guzzi
- Dalle Molle Institute for Artificial Intelligence (IDSIA), USI-SUPSI, Lugano, Switzerland
| |
Collapse
|
13
|
Cai S, Zhu K, Ban Y, Narumi T. Visual-Tactile Cross-Modal Data Generation Using Residue-Fusion GAN With Feature-Matching and Perceptual Losses. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3095925] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
14
|
|