1
|
Hou J, He C, Li T, Zhang C, Zhou Q. Learning obstacle avoidance and predation in complex reef environments with deep reinforcement learning. BIOINSPIRATION & BIOMIMETICS 2024; 19:056014. [PMID: 39025108 DOI: 10.1088/1748-3190/ad6544] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/25/2024] [Accepted: 07/18/2024] [Indexed: 07/20/2024]
Abstract
The reef ecosystem plays a vital role as a habitat for fish species with limited swimming capabilities, serving not only as a sanctuary and food source but also influencing their behavioral tendencies. Understanding the intricate mechanism through which fish adeptly navigate the moving targets within reef environments within complex water flow, all while evading obstacles and maintaining stable postures, has remained a challenging and prominent subject in the realms of fish behavior, ecology, and biomimetics alike. An integrated simulation framework is used to investigate fish predation problems within intricate environments, combining deep reinforcement learning algorithms (DRL) with high-precision fluid-structure interaction numerical methods-immersed boundary lattice Boltzmann method (lB-LBM). The Soft Actor-Critic (SAC) algorithm is used to improve the intelligent fish's capacity for random exploration, tackling the multi-objective sparse reward challenge inherent in real-world scenarios. Additionally, a reward shaping method tailored to its action purposes has been developed, capable of capturing outcomes and trend characteristics effectively. The convergence and robustness advantages of the method elucidated in this paper are showcased through two case studies: one addressing fish capturing randomly moving targets in hydrostatic flow field, and the other focusing on fish counter-current foraging in reef environments to capture drifting food. A comprehensive analysis was conducted of the influence and significance of various reward types on the decision-making processes of intelligent fish within intricate environments.
Collapse
Affiliation(s)
- Ji Hou
- Southwest Research Institute for Hydraulic and Water Transport Engineering, Chongqing Jiaotong University, Chongqing 402247, People's Republic of China
- The College of River and Ocean Engineering, Chongqing Jiaotong University, Chongqing 400074, People's Republic of China
| | - Changling He
- Southwest Research Institute for Hydraulic and Water Transport Engineering, Chongqing Jiaotong University, Chongqing 402247, People's Republic of China
- The College of River and Ocean Engineering, Chongqing Jiaotong University, Chongqing 400074, People's Republic of China
| | - Tao Li
- Southwest Research Institute for Hydraulic and Water Transport Engineering, Chongqing Jiaotong University, Chongqing 402247, People's Republic of China
- The College of River and Ocean Engineering, Chongqing Jiaotong University, Chongqing 400074, People's Republic of China
| | - Chunze Zhang
- Southwest Research Institute for Hydraulic and Water Transport Engineering, Chongqing Jiaotong University, Chongqing 402247, People's Republic of China
- The College of River and Ocean Engineering, Chongqing Jiaotong University, Chongqing 400074, People's Republic of China
| | - Qin Zhou
- Chongqing Xike Consulting Co., LTD for Water Transport Engineering, Chongqing 402247, People's Republic of China
| |
Collapse
|
2
|
Liu D, Wang J, Mao X, Deng J. Energetic benefits in coordinated circular swimming motion of two swimmers. Phys Rev E 2023; 108:054603. [PMID: 38115522 DOI: 10.1103/physreve.108.054603] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2023] [Accepted: 09/13/2023] [Indexed: 12/21/2023]
Abstract
The coordinated movement of multiple swimmers is a crucial component of fish schools. Fish swimming in different formations, such as tandem, side-by-side, diamond, and phalanx, can achieve significant energetic advantages. However, the energetic benefits of nonstraight swimming behaviors, such as the collective motion of a milling pattern, are not well understood. To fill in this gap, we consider two swimmers in circular tracks, controlled by a PID approach to reach stable configurations. Our study finds that the optimal phase is affected by circumferential effects, and that substantial energy savings can result from both propulsion and turning. We also explore the radial effect in terms of energetic benefits. In a milling pattern, the inner swimmers can easily gain a certain energetic benefit (-8%), while their peers on the outside must be close enough to the inner swimmer with a proper phase to gain the energetic benefit (-14%). When the radial spacing becomes larger or is in an unmatched phase, the swimming of the outer swimmers becomes more laborious (+16%). Our results indicate that swimmers who maintain a matched phase and minimum radial effect obtain the highest energetic benefits (-26%). These findings highlight the energetic benefits of swimmers, even in a milling pattern, where the position difference dominates the extent of benefit.
Collapse
Affiliation(s)
- Danshi Liu
- Department of Mechanics, Zhejiang University, Hangzhou 310027, People's Republic of China
| | - Jiadong Wang
- Department of Mechanics, Zhejiang University, Hangzhou 310027, People's Republic of China
| | - Xuewei Mao
- Department of Mechanics, Zhejiang University, Hangzhou 310027, People's Republic of China
| | - Jian Deng
- Department of Mechanics, Zhejiang University, Hangzhou 310027, People's Republic of China
| |
Collapse
|
3
|
Tong R, Feng Y, Wang J, Wu Z, Tan M, Yu J. A Survey on Reinforcement Learning Methods in Bionic Underwater Robots. Biomimetics (Basel) 2023; 8:biomimetics8020168. [PMID: 37092420 PMCID: PMC10123646 DOI: 10.3390/biomimetics8020168] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2023] [Revised: 04/15/2023] [Accepted: 04/18/2023] [Indexed: 04/25/2023] Open
Abstract
Bionic robots possess inherent advantages for underwater operations, and research on motion control and intelligent decision making has expanded their application scope. In recent years, the application of reinforcement learning algorithms in the field of bionic underwater robots has gained considerable attention, and continues to grow. In this paper, we present a comprehensive survey of the accomplishments of reinforcement learning algorithms in the field of bionic underwater robots. Firstly, we classify existing reinforcement learning methods and introduce control tasks and decision making tasks based on the composition of bionic underwater robots. We further discuss the advantages and challenges of reinforcement learning for bionic robots in underwater environments. Secondly, we review the establishment of existing reinforcement learning algorithms for bionic underwater robots from different task perspectives. Thirdly, we explore the existing training and deployment solutions of reinforcement learning algorithms for bionic underwater robots, focusing on the challenges posed by complex underwater environments and underactuated bionic robots. Finally, the limitations and future development directions of reinforcement learning in the field of bionic underwater robots are discussed. This survey provides a foundation for exploring reinforcement learning control and decision making methods for bionic underwater robots, and provides insights for future research.
Collapse
Affiliation(s)
- Ru Tong
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Yukai Feng
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Jian Wang
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Zhengxing Wu
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Min Tan
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Junzhi Yu
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
- State Key Laboratory for Turbulence and Complex Systems, Department of Advanced Manufacturing and Robotics, College of Engineering, Peking University, Beijing 100871, China
| |
Collapse
|
4
|
Li Y, Xu Y, Wu Z, Ma L, Guo M, Li Z, Li Y. A comprehensive review on fish-inspired robots. INT J ADV ROBOT SYST 2022. [DOI: 10.1177/17298806221103707] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
Abstract
Recently, the increasing interest in underwater exploration motivates the development of aquatic unmanned vehicles. To execute hazardous tasks in an unknown or even hostile environment, researchers have directed on developing biomimetic robots inspired by the extraordinary maneuverability, cruising speed, and propulsion efficiency of fish. Nevertheless, the performance of current prototypes still has gaps compared with that of real fishes. In this review, recent approaches in structure designs, actuators, and sensors are presented. In addition, the theoretical methods for modeling the robotic fishes are consolidated, and the control strategies are offered. Finally, the current challenges are summarized, and possible future directions are deeply discussed. It is expected that the emergence of new engineering and biological technologies will enhance the field of robotic fish for further advancement.
Collapse
Affiliation(s)
- Yi Li
- Key Laboratory of Special Purpose Equipment and Advanced Processing Technology of Ministry of Education, Zhejiang University of Technology, Hangzhou, China
- Zhejiang Provincial Key Laboratory of Special Purpose Equipment and Advanced Processing Technology, Zhejiang University of Technology, Hangzhou, China
| | - Yuteng Xu
- Key Laboratory of Special Purpose Equipment and Advanced Processing Technology of Ministry of Education, Zhejiang University of Technology, Hangzhou, China
- Zhejiang Provincial Key Laboratory of Special Purpose Equipment and Advanced Processing Technology, Zhejiang University of Technology, Hangzhou, China
| | - Zhenguo Wu
- Key Laboratory of Special Purpose Equipment and Advanced Processing Technology of Ministry of Education, Zhejiang University of Technology, Hangzhou, China
- Zhejiang Provincial Key Laboratory of Special Purpose Equipment and Advanced Processing Technology, Zhejiang University of Technology, Hangzhou, China
| | - Lei Ma
- Key Laboratory of Special Purpose Equipment and Advanced Processing Technology of Ministry of Education, Zhejiang University of Technology, Hangzhou, China
- Zhejiang Provincial Key Laboratory of Special Purpose Equipment and Advanced Processing Technology, Zhejiang University of Technology, Hangzhou, China
| | - Mingfei Guo
- Key Laboratory of Special Purpose Equipment and Advanced Processing Technology of Ministry of Education, Zhejiang University of Technology, Hangzhou, China
- Zhejiang Provincial Key Laboratory of Special Purpose Equipment and Advanced Processing Technology, Zhejiang University of Technology, Hangzhou, China
| | - Zhixin Li
- Key Laboratory of Special Purpose Equipment and Advanced Processing Technology of Ministry of Education, Zhejiang University of Technology, Hangzhou, China
- Zhejiang Provincial Key Laboratory of Special Purpose Equipment and Advanced Processing Technology, Zhejiang University of Technology, Hangzhou, China
| | - Yanbiao Li
- Key Laboratory of Special Purpose Equipment and Advanced Processing Technology of Ministry of Education, Zhejiang University of Technology, Hangzhou, China
- Zhejiang Provincial Key Laboratory of Special Purpose Equipment and Advanced Processing Technology, Zhejiang University of Technology, Hangzhou, China
| |
Collapse
|
5
|
Xu WH, Xu GD, Shan L. Real-time parametric estimation of periodic wake-foil interactions using bioinspired pressure sensing and machine learning. BIOINSPIRATION & BIOMIMETICS 2022; 17:026010. [PMID: 34996050 DOI: 10.1088/1748-3190/ac4953] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/09/2021] [Accepted: 01/07/2022] [Indexed: 06/14/2023]
Abstract
Periodic wake-foil interactions occur in the collective swimming of bio-inspired robots. Wake interaction pattern estimation (and control) is crucial to thrust enhancement and propulsive efficiency optimization. In this paper, we study the wake interaction pattern estimation of two flapping foils in tandem configurations. The experiments are conducted at a Reynolds number of 1.41 × 104in a water channel. A modified wake-foil phase parameter Φ, which unifies the influences of inter-foil distanceLx, motion phase difference Δφand wake convection velocityUv, is introduced to describe the wake interaction patterns parametrically. We use a differential pressure sensor on the downstream foil to capture wake interaction characteristics. Data sets at different tandem configurations are collected. The wake-foil phase Φ is used to label the pressure signals. A one-dimensional convolutional neural networks (1D-CNN) model is used to learn an end-to-end mapping between the raw pressure measurements and the wake-foil phase Φ. The trained 1D-CNN model shows accurate estimations (average error 3.5%) on random wake interaction patterns and is fast enough (within 40 ms). Then the trained 1D-CNN model is applied to online thrust enhancement control of a downstream foil swimming in a periodic wake. Synchronous force monitoring and flow visualization demonstrate the effectiveness of the 1D-CNN model. The limitations of the model are discussed. The proposed approach can be applied to the online estimation and control of wake interactions in the collective swimming and flying of biomimetic robots.
Collapse
Affiliation(s)
- Wen-Hua Xu
- College of Shipbuilding Engineering, Harbin Engineering University, Harbin 150001, People's Republic of China
| | - Guo-Dong Xu
- College of Shipbuilding Engineering, Harbin Engineering University, Harbin 150001, People's Republic of China
| | - Lei Shan
- College of Shipbuilding Engineering, Harbin Engineering University, Harbin 150001, People's Republic of China
| |
Collapse
|