1
|
Cai M, Wang Q, Qi Z, Jin D, Wu X, Xu T, Zhang L. Deep Reinforcement Learning Framework-Based Flow Rate Rejection Control of Soft Magnetic Miniature Robots. IEEE TRANSACTIONS ON CYBERNETICS 2023; 53:7699-7711. [PMID: 36070281 DOI: 10.1109/tcyb.2022.3199213] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Soft magnetic miniature robots (SMMRs) have potential biomedical applications due to their flexible size and mobility to access confined environments. However, navigating the robot to a goal site with precise control performance and high repeatability in unstructured environments, especially in flow rate conditions, still remains a challenge. In this study, drawing inspiration from the control requirements of drug delivery and release to the goal lesion site in the presence of dynamic biofluids, we propose a flow rate rejection control strategy based on a deep reinforcement learning (DRL) framework to actuate an SMMR to achieve goal-reaching and hovering in fluidic tubes. To this end, an SMMR is first fabricated, which can be operated by an external magnetic field to realize its desired functionalities. Subsequently, a simulator is constructed based on neural networks to map the relationship between the applied magnetic field and robot locomotion states. With minimal prior knowledge about the environment and dynamics, a gated recurrent unit (GRU)-based DRL algorithm is formulated by considering the designed history state-action and estimated flow rates. In addition, the randomization technique is applied during training to distill the general control policy for the physical SMMR. The results of numerical simulations and experiments are illustrated to demonstrate the robustness and efficacy of the presented control framework. Finally, in-depth analyses and discussions indicate the potentiality of DRL for soft magnetic robots in biomedical applications.
Collapse
|
2
|
Chen F, Chen L, Xu T, Ye H, Liao H, Zhang D. Precise angle estimation of capsule robot in ultrasound using heatmap guided two-stage network. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 240:107605. [PMID: 37390795 DOI: 10.1016/j.cmpb.2023.107605] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 05/15/2023] [Accepted: 05/15/2023] [Indexed: 07/02/2023]
Abstract
PURPOSE A capsule robot can be controlled inside gastrointestinal (GI) tract by an external permanent magnet outside of human body for finishing non-invasive diagnosis and treatment. Locomotion control of capsule robot relies on the precise angle feedback that can be achieved by ultrasound imaging. However, ultrasound-based angle estimation of capsule robot is interfered by gastric wall tissue and the mixture of air, water, and digestive matter existing in the stomach. METHODS To tackle these issues, we introduce a heatmap guided two-stage network to detect the position and estimate the angle of the capsule robot in ultrasound images. Specifically, this network proposes the probability distribution module and skeleton extraction-based angle calculation to obtain accurate capsule robot position and angle estimation. RESULTS Extensive experiments were finished on the ultrasound image dataset of capsule robot within porcine stomach. Empirical results showed that our method obtained small position center error of 0.48 mm and high angle estimation accuracy of 96.32%. CONCLUSION Our method can provide precise angle feedback for locomotion control of capsule robot.
Collapse
Affiliation(s)
- Fang Chen
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China.
| | - Lingyu Chen
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Tianze Xu
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Haoran Ye
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Hongen Liao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, China
| | - Daoqiang Zhang
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| |
Collapse
|
3
|
Barnoy Y, Erin O, Raval S, Pryor W, Mair LO, Weinberg IN, Diaz-Mercado Y, Krieger A, Hager GD. Control of Magnetic Surgical Robots With Model-Based Simulators and Reinforcement Learning. IEEE TRANSACTIONS ON MEDICAL ROBOTICS AND BIONICS 2022; 4:945-956. [PMID: 37600471 PMCID: PMC10438915 DOI: 10.1109/tmrb.2022.3214426] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/22/2023]
Abstract
Magnetically manipulated medical robots are a promising alternative to current robotic platforms, allowing for miniaturization and tetherless actuation. Controlling such systems autonomously may enable safe, accurate operation. However, classical control methods require rigorous models of magnetic fields, robot dynamics, and robot environments, which can be difficult to generate. Model-free reinforcement learning (RL) offers an alternative that can bypass these requirements. We apply RL to a robotic magnetic needle manipulation system. Reinforcement learning algorithms often require long runtimes, making them impractical for many surgical robotics applications, most of which require careful, constant monitoring. Our approach first constructs a model-based simulation (MBS) on guided real-world exploration, learning the dynamics of the environment. After intensive MBS environment training, we transfer the learned behavior from the MBS environment to the real-world. Our MBS method applies RL roughly 200 times faster than doing so in the real world, and achieves a 6 mm root-mean-square (RMS) error for a square reference trajectory. In comparison, pure simulation-based approaches fail to transfer, producing a 31 mm RMS error. These results demonstrate that MBS environments are a good solution for domains where running model-free RL is impractical, especially if an accurate simulation is not available.
Collapse
Affiliation(s)
- Yotam Barnoy
- Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21287 USA
| | - Onder Erin
- Department of Mechanical Engineering, The Johns Hopkins University, Baltimore, MD 21287 USA
| | - Suraj Raval
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742 USA
| | - Will Pryor
- Department of Mechanical Engineering, The Johns Hopkins University, Baltimore, MD 21287 USA
| | - Lamar O. Mair
- Weinberg Medical Physics, Inc., North Bethesda, MD 20852 USA
| | | | - Yancy Diaz-Mercado
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742 USA
| | - Axel Krieger
- Department of Mechanical Engineering, The Johns Hopkins University, Baltimore, MD 21287 USA
| | - Gregory D. Hager
- Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21287 USA
| |
Collapse
|
4
|
Behrens MR, Ruder WC. Smart Magnetic Microrobots Learn to Swim with Deep Reinforcement Learning. ADVANCED INTELLIGENT SYSTEMS (WEINHEIM AN DER BERGSTRASSE, GERMANY) 2022; 4:2200023. [PMID: 38463142 PMCID: PMC10923539 DOI: 10.1002/aisy.202200023] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/24/2022] [Indexed: 03/12/2024]
Abstract
Swimming microrobots are increasingly developed with complex materials and dynamic shapes and are expected to operate in complex environments in which the system dynamics are difficult to model and positional control of the microrobot is not straightforward to achieve. Deep reinforcement learning is a promising method of autonomously developing robust controllers for creating smart microrobots, which can adapt their behavior to operate in uncharacterized environments without the need to model the system dynamics. This article reports the development of a smart helical magnetic hydrogel microrobot that uses the soft actor critic reinforcement learning algorithm to autonomously derive a control policy which allows the microrobot to swim through an uncharacterized biomimetic fluidic environment under control of a time varying magnetic field generated from a three-axis array of electromagnets. The reinforcement learning agent learned successful control policies from both state vector input and raw images, and the control policies learned by the agent recapitulated the behavior of rationally designed controllers based on physical models of helical swimming microrobots. Deep reinforcement learning applied to microrobot control is likely to significantly expand the capabilities of the next generation of microrobots.
Collapse
Affiliation(s)
- Michael R. Behrens
- Department of Bioengineering, University of Pittsburgh; 300 Technology Drive, Pittsburgh, PA 15213, USA
| | - Warren C. Ruder
- Department of Bioengineering, University of Pittsburgh; 300 Technology Drive, Pittsburgh, PA 15213, USA
- Department of Mechanical Engineering, Carnegie Mellon University; 5000 Forbes Ave. Pittsburgh, PA 15213, USA
| |
Collapse
|
5
|
SelfVIO: Self-supervised deep monocular Visual–Inertial Odometry and depth estimation. Neural Netw 2022; 150:119-136. [DOI: 10.1016/j.neunet.2022.03.005] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2022] [Revised: 03/01/2022] [Accepted: 03/03/2022] [Indexed: 01/31/2023]
|
6
|
Abstract
In conventional classification, soft robots feature mechanical compliance as the main distinguishing factor from traditional robots made of rigid materials. Recent advances in functional soft materials have facilitated the emergence of a new class of soft robots capable of tether-free actuation in response to external stimuli such as heat, light, solvent, or electric or magnetic field. Among the various types of stimuli-responsive materials, magnetic soft materials have shown remarkable progress in their design and fabrication, leading to the development of magnetic soft robots with unique advantages and potential for many important applications. However, the field of magnetic soft robots is still in its infancy and requires further advancements in terms of design principles, fabrication methods, control mechanisms, and sensing modalities. Successful future development of magnetic soft robots would require a comprehensive understanding of the fundamental principle of magnetic actuation, as well as the physical properties and behavior of magnetic soft materials. In this review, we discuss recent progress in the design and fabrication, modeling and simulation, and actuation and control of magnetic soft materials and robots. We then give a set of design guidelines for optimal actuation performance of magnetic soft materials. Lastly, we summarize potential biomedical applications of magnetic soft robots and provide our perspectives on next-generation magnetic soft robots.
Collapse
Affiliation(s)
- Yoonho Kim
- Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| | - Xuanhe Zhao
- Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
- Department of Civil and Environmental Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| |
Collapse
|
7
|
Prendergast JM, Formosa GA, Fulton MJ, Heckman CR, Rentschler ME. A Real-Time State Dependent Region Estimator for Autonomous Endoscope Navigation. IEEE T ROBOT 2021. [DOI: 10.1109/tro.2020.3038709] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
8
|
Singh B, Kumar R, Singh VP. Reinforcement learning in robotic applications: a comprehensive survey. Artif Intell Rev 2021. [DOI: 10.1007/s10462-021-09997-9] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|