1
|
Riaziat ND, Erin O, Krieger A, Brown JD. Investigating Haptic Feedback in Vision-Deficient Millirobot Telemanipulation. IEEE Robot Autom Lett 2024; 9:6178-6185. [PMID: 38948904 PMCID: PMC11210683 DOI: 10.1109/lra.2024.3397529] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/02/2024]
Abstract
The evolution of magnetically actuated millirobots gives rise to unique teleoperation challenges due to their non-traditional kinematic and dynamic architectures, as well as their frequent use of suboptimal imaging modalities. Recent investigations into haptic interfaces for millirobots have shown promise but lack the clinically motivated task scenarios necessary to justify future development. In this work, we investigate the utility of haptic feedback on bilateral teleoperation of a magnetically actuated millirobot in visually deficient conditions. We conducted an N=23 user study in an aneurysm coiling inspired procedure, which required participants to navigate the robot through a maze in near total darkness to manipulate beads to a target under simulated fluoroscopy. We hypothesized that users will be better able to complete the telemanipulation task with haptic feedback while reducing excess forces on their surroundings compared to the no feedback conditions. Our results showed an over 40% improvement in participants' bead scoring, a nearly 10% reduction in mean force, and 13% reduction in maximum force with haptic feedback, as well as significant improvements in other metrics. Results highlight that benefits of haptic feedback are retained when haptic feedback is removed. These findings suggest that haptic feedback has the potential to significantly improve millirobot telemanipulation and control in traditionally vision deficient tasks.
Collapse
Affiliation(s)
- Naveed D Riaziat
- Department of Mechanical Engineering, Johns Hopkins University, 3400 North Charles Street, Baltimore, Maryland, USA
| | - Onder Erin
- Department of Mechanical Engineering, Johns Hopkins University, 3400 North Charles Street, Baltimore, Maryland, USA. He is now with Johnson and Johnson, 5490 Great America Parkway, Santa Clara, California, USA
| | - Axel Krieger
- Department of Mechanical Engineering, Johns Hopkins University, 3400 North Charles Street, Baltimore, Maryland, USA
| | - Jeremy D Brown
- Department of Mechanical Engineering, Johns Hopkins University, 3400 North Charles Street, Baltimore, Maryland, USA
| |
Collapse
|
2
|
Barnoy Y, Erin O, Raval S, Pryor W, Mair LO, Weinberg IN, Diaz-Mercado Y, Krieger A, Hager GD. Control of Magnetic Surgical Robots With Model-Based Simulators and Reinforcement Learning. IEEE TRANSACTIONS ON MEDICAL ROBOTICS AND BIONICS 2022; 4:945-956. [PMID: 37600471 PMCID: PMC10438915 DOI: 10.1109/tmrb.2022.3214426] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/22/2023]
Abstract
Magnetically manipulated medical robots are a promising alternative to current robotic platforms, allowing for miniaturization and tetherless actuation. Controlling such systems autonomously may enable safe, accurate operation. However, classical control methods require rigorous models of magnetic fields, robot dynamics, and robot environments, which can be difficult to generate. Model-free reinforcement learning (RL) offers an alternative that can bypass these requirements. We apply RL to a robotic magnetic needle manipulation system. Reinforcement learning algorithms often require long runtimes, making them impractical for many surgical robotics applications, most of which require careful, constant monitoring. Our approach first constructs a model-based simulation (MBS) on guided real-world exploration, learning the dynamics of the environment. After intensive MBS environment training, we transfer the learned behavior from the MBS environment to the real-world. Our MBS method applies RL roughly 200 times faster than doing so in the real world, and achieves a 6 mm root-mean-square (RMS) error for a square reference trajectory. In comparison, pure simulation-based approaches fail to transfer, producing a 31 mm RMS error. These results demonstrate that MBS environments are a good solution for domains where running model-free RL is impractical, especially if an accurate simulation is not available.
Collapse
Affiliation(s)
- Yotam Barnoy
- Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21287 USA
| | - Onder Erin
- Department of Mechanical Engineering, The Johns Hopkins University, Baltimore, MD 21287 USA
| | - Suraj Raval
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742 USA
| | - Will Pryor
- Department of Mechanical Engineering, The Johns Hopkins University, Baltimore, MD 21287 USA
| | - Lamar O. Mair
- Weinberg Medical Physics, Inc., North Bethesda, MD 20852 USA
| | | | - Yancy Diaz-Mercado
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742 USA
| | - Axel Krieger
- Department of Mechanical Engineering, The Johns Hopkins University, Baltimore, MD 21287 USA
| | - Gregory D. Hager
- Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21287 USA
| |
Collapse
|
3
|
Erin O, Raval S, Schwehr TJ, Pryor W, Barnoy Y, Bell A, Liu X, Mair LO, Weinberg IN, Krieger A, Diaz-Mercado Y. Enhanced Accuracy in Magnetic Actuation: Closed-loop Control of a Magnetic Agent with Low-Error Numerical Magnetic Model Estimation. IEEE Robot Autom Lett 2022; 7:9429-9436. [PMID: 36544557 PMCID: PMC9762677 DOI: 10.1109/lra.2022.3191047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/25/2022]
Abstract
Magnetic actuation holds promise for wirelessly controlling small, magnetic surgical tools and may enable the next generation of ultra minimally invasive surgical robotic systems. Precise torque and force exertion are required for safe surgical operations and accurate state control. Dipole field estimation models perform well far from electromagnets but yield large errors near coils. Thus, manipulations near coils suffer from severe (10×) field modeling errors. We experimentally quantify closed-loop magnetic agent control performance by using both a highly erroneous dipole model and a more accurate numerical magnetic model to estimate magnetic forces and torques for any given robot pose in 2D. We compare experimental measurements with estimation errors for the dipole model and our finite element analysis (FEA) based model of fields near coils. With five different paths designed for this study, we demonstrate that FEA-based magnetic field modeling reduces positioning root-mean-square (RMS) errors by 48% to 79% as compared with dipole models. Models demonstrate close agreement for magnetic field direction estimation, showing similar accuracy for orientation control. Such improved magnetic modelling is crucial for systems requiring robust estimates of magnetic forces for positioning agents, particularly in force-sensitive environments like surgical manipulation.
Collapse
Affiliation(s)
- Onder Erin
- Mechanical Engineering Department, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Suraj Raval
- Mechanical Engineering Department, University of Maryland, College Park, MD 20742, USA
| | - Trevor J Schwehr
- Mechanical Engineering Department, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Will Pryor
- Computer Science Department, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Yotam Barnoy
- Computer Science Department, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Adrian Bell
- Mechanical Engineering Department, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Xiaolong Liu
- Mechanical Engineering Department, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Lamar O Mair
- Division of Magnetic Manipulation & Particle Research, Weinberg Medical Physics, Inc., North Bethesda, MD 20852, USA
| | - Irving N Weinberg
- Division of Magnetic Manipulation & Particle Research, Weinberg Medical Physics, Inc., North Bethesda, MD 20852, USA
| | - Axel Krieger
- Mechanical Engineering Department, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Yancy Diaz-Mercado
- Mechanical Engineering Department, University of Maryland, College Park, MD 20742, USA
| |
Collapse
|
4
|
Erin O, Liu X, Ge J, Opfermann J, Barnoy Y, Mair LO, Kang JU, Gensheimer W, Weinberg IN, Diaz-Mercado Y, Krieger A. Overcoming the Force Limitations of Magnetic Robotic Surgery: Magnetic Pulse Actuated Collisions for Tissue-Penetrating-Needle for Tetherless Interventions. ADVANCED INTELLIGENT SYSTEMS (WEINHEIM AN DER BERGSTRASSE, GERMANY) 2022; 4:2200072. [PMID: 35967598 PMCID: PMC9364690 DOI: 10.1002/aisy.202200072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
The field of magnetic robotics aims to obviate physical connections between the actuators and end-effectors. Such tetherless control may enable new ultra-minimally invasive surgical manipulations in clinical settings. While wireless actuation offers advantages in medical applications, the challenge of providing sufficient force to magnetic needles for tissue penetration remains a barrier to practical application. Applying sufficient force for tissue penetration is required for tasks such as biopsy, suturing, cutting, drug delivery, and accessing deep seated regions of complex structures in organs such as the eye. To expand the force landscape for such magnetic surgical tools, an impact-force based suture needle capable of penetrating in vitro and ex vivo samples with 3-DOF planar motion is proposed. Using custom-built 14G and 25G needles, we demonstrate generation of 410 mN penetration force, a 22.7-fold force increase with more than 20 times smaller volume compared to similar magnetically guided needles. With the MPACT-Needle, in vitro suturing of a gauze mesh onto an agar gel is demonstrated. In addition, we have reduced the tip size to 25G, which is a typical needle size for interventions in the eye, to demonstrate ex vivo penetration in a rabbit eye, mimicking procedures such as corneal injections and transscleral drug delivery.
Collapse
Affiliation(s)
- Onder Erin
- Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Xiaolong Liu
- Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Jiawei Ge
- Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Justin Opfermann
- Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Yotam Barnoy
- Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Lamar O Mair
- Weinberg Medical Physics, Inc., North Bethesda, MD 20852, USA
| | - Jin U Kang
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - William Gensheimer
- Department of Opthalmology, Dartmouth-Hitchcock Medical Center, Lebanon, NH 03756, USA
| | | | - Yancy Diaz-Mercado
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742, USA
| | - Axel Krieger
- Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| |
Collapse
|
5
|
Li M, Tang Y, Soon RH, Dong B, Hu W, Sitti M. Miniature coiled artificial muscle for wireless soft medical devices. SCIENCE ADVANCES 2022; 8:eabm5616. [PMID: 35275717 PMCID: PMC8916729 DOI: 10.1126/sciadv.abm5616] [Citation(s) in RCA: 18] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/06/2023]
Abstract
Wireless small-scale soft-bodied devices are capable of precise operation inside confined internal spaces, enabling various minimally invasive medical applications. However, such potential is constrained by the small output force and low work capacity of the current miniature soft actuators. To address this challenge, we report a small-scale soft actuator that harnesses the synergetic interactions between the coiled artificial muscle and radio frequency-magnetic heating. This wirelessly controlled actuator exhibits a large output force (~3.1 N) and high work capacity (3.5 J/g). Combining this actuator with different mechanical designs, its tensile and torsional behaviors can be engineered into different functional devices, such as a suture device, a pair of scissors, a driller, and a clamper. In addition, by assuming a spatially varying magnetization profile, a multilinked coiled muscle can have both magnetic field-induced bending and high contractile force. Such an approach could be used in various future untethered miniature medical devices.
Collapse
Affiliation(s)
- Mingtong Li
- Physical Intelligence Department, Max Planck Institute for Intelligent Systems, 70569 Stuttgart, Germany
- Institute of Functional Nano & Soft Materials (FUNSOM), Jiangsu Key Laboratory for Carbon-Based Functional Materials & Devices, Soochow University, Suzhou, Jiangsu 215123, P. R. China
| | - Yichao Tang
- Physical Intelligence Department, Max Planck Institute for Intelligent Systems, 70569 Stuttgart, Germany
- School of Mechanical Engineering, Tongji University, Shanghai 201804, P. R. China
| | - Ren Hao Soon
- Physical Intelligence Department, Max Planck Institute for Intelligent Systems, 70569 Stuttgart, Germany
- School of Mechanical Engineering, Tongji University, Shanghai 201804, P. R. China
| | - Bin Dong
- Institute of Functional Nano & Soft Materials (FUNSOM), Jiangsu Key Laboratory for Carbon-Based Functional Materials & Devices, Soochow University, Suzhou, Jiangsu 215123, P. R. China
| | - Wenqi Hu
- Physical Intelligence Department, Max Planck Institute for Intelligent Systems, 70569 Stuttgart, Germany
- Corresponding author. (W.H.); (M.S.)
| | - Metin Sitti
- Physical Intelligence Department, Max Planck Institute for Intelligent Systems, 70569 Stuttgart, Germany
- Institute for Biomedical Engineering, ETH Zürich, 8092 Zürich, Switzerland
- School of Medicine and College of Engineering, Koç University, 34450 Istanbul, Turkey
- Corresponding author. (W.H.); (M.S.)
| |
Collapse
|
6
|
Going Hands-Free: MagnetoSuture™ for Untethered Guided Needle Penetration of Human Tissue Ex Vivo. ROBOTICS 2021. [DOI: 10.3390/robotics10040129] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
The application of force in surgical settings is typically accomplished via physical tethers to the surgical tool. While physical tethers are common and critical, some internal surgical procedures may benefit from a tetherless operation of needles, possibly reducing the number of ports in the patient or the amount of tissue damage caused by tools used to manipulate needles. Magnetic field gradients can dynamically apply kinetic forces to magnetizable objects free of such tethers, possibly enabling ultra-minimally invasive robotic surgical procedures. We demonstrate the untethered manipulation of a suture needle in vitro, exemplified by steering through narrow holes, as well as needle penetration through excised rat and human tissues. We present proof of principle manipulations for the fully untethered control of a minimally modified, standard stainless steel surgical suture needle.
Collapse
|
7
|
Pryor W, Barnoy Y, Raval S, Liu X, Mair L, Lerner D, Erin O, Hager GD, Diaz-Mercado Y, Krieger A. Localization and Control of Magnetic Suture Needles in Cluttered Surgical Site with Blood and Tissue. PROCEEDINGS OF THE ... IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS. IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS 2021; 2021:524-531. [PMID: 35223133 DOI: 10.1109/iros51168.2021.9636441] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Real-time visual localization of needles is necessary for various surgical applications, including surgical automation and visual feedback. In this study we investigate localization and autonomous robotic control of needles in the context of our magneto-suturing system. Our system holds the potential for surgical manipulation with the benefit of minimal invasiveness and reduced patient side effects. However, the nonlinear magnetic fields produce unintuitive forces and demand delicate position-based control that exceeds the capabilities of direct human manipulation. This makes automatic needle localization a necessity. Our localization method combines neural network-based segmentation and classical techniques, and we are able to consistently locate our needle with 0.73 mm RMS error in clean environments and 2.72 mm RMS error in challenging environments with blood and occlusion. The average localization RMS error is 2.16 mm for all environments we used in the experiments. We combine this localization method with our closed-loop feedback control system to demonstrate the further applicability of localization to autonomous control. Our needle is able to follow a running suture path in (1) no blood, no tissue; (2) heavy blood, no tissue; (3) no blood, with tissue; and (4) heavy blood, with tissue environments. The tip position tracking error ranges from 2.6 mm to 3.7 mm RMS, opening the door towards autonomous suturing tasks.
Collapse
Affiliation(s)
- Will Pryor
- Department of Computer Science, Johns Hopkins University, Baltimore, MD 21211 USA
| | - Yotam Barnoy
- Department of Computer Science, Johns Hopkins University, Baltimore, MD 21211 USA
| | - Suraj Raval
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742 USA
| | - Xiaolong Liu
- Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21211 USA
| | - Lamar Mair
- Weinberg Medical Physics, Inc., North Bethesda, MD 20852 USA
| | - Daniel Lerner
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742 USA
| | - Onder Erin
- Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21211 USA
| | - Gregory D Hager
- Department of Computer Science, Johns Hopkins University, Baltimore, MD 21211 USA
| | - Yancy Diaz-Mercado
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742 USA
| | - Axel Krieger
- Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21211 USA
| |
Collapse
|
8
|
Lim A, Schonewille A, Forbrigger C, Looi T, Drake J, Diller E. Design and Comparison of Magnetically-Actuated Dexterous Forceps Instruments for Neuroendoscopy. IEEE Trans Biomed Eng 2021; 68:846-856. [PMID: 32746054 DOI: 10.1109/tbme.2020.3007581] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/18/2022]
Abstract
Robot-assisted minimally invasive surgical (MIS) techniques offer improved instrument precision and dexterity, reduced patient trauma and risk, and promise to lessen the skill gap among surgeons. These approaches are common in general surgery, urology, and gynecology. However, MIS techniques remain largely absent for surgical applications within narrow, confined workspaces, such as neuroendoscopy. The limitation stems from a lack of small yet dexterous robotic tools. In this work, we present the first instance of a surgical robot with a direct magnetically-driven end effector capable of being deployed through a standard neuroendoscopic working channel (3.2 mm outer diameter) and operate at the neuroventricular scale. We propose a physical model for the gripping performance of three unique end-effector magnetization profiles and mechanical designs. Rates of blocking force per external magnetic flux density magnitude were 0.309 N/T, 0.880 N/T, and 0.351 N/T for the three designs which matched the physical model's prediction within 14.9% error. The rate of gripper closure per external magnetic flux density had a mean percent error of 11.2% compared to the model. The robot's performance was qualitatively evaluated during a pineal region tumor resection on a tumor analogue in a silicone brain phantom. These results suggest that wireless magnetic actuation may be feasible for dexterously manipulating tissue during minimally invasive neurosurgical procedures.
Collapse
|
9
|
Fan M, Liu X, Jain K, Lerner D, Mair LO, Weinberg IN, Diaz-Mercado Y, Krieger A. Towards Autonomous Control of Magnetic Suture Needles. PROCEEDINGS OF THE ... IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS. IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS 2020; 2020:10.1109/iros45743.2020.9341425. [PMID: 34457374 PMCID: PMC8389736 DOI: 10.1109/iros45743.2020.9341425] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
This paper proposes a magnetic needle steering controller to manipulate mesoscale magnetic suture needles for executing planned suturing motion. This is an initial step towards our research objective: enabling autonomous control of magnetic suture needles for suturing tasks in minimally invasive surgery. To demonstrate the feasibility of accurate motion control, we employ a cardinally-arranged four-coil electromagnetic system setup and control magnetic suture needles in a 2-dimensional environment, i.e., a Petri dish filled with viscous liquid. Different from only using magnetic field gradients to control small magnetic agents under high damping conditions, the dynamics of a magnetic suture needle are investigated and encoded in the controller. Based on mathematical formulations of magnetic force and torque applied on the needle, we develop a kinematically constrained dynamic model that controls the needle to rotate and only translate along its central axis for mimicking the behavior of surgical sutures. A current controller of the electromagnetic system combining with closed-loop control schemes is designed for commanding the magnetic suture needles to achieve desired linear and angular velocities. To evaluate control performance of magnetic suture needles, we conduct experiments including needle rotation control, needle position control by using discretized trajectories, and velocity control by using a time-varying circular trajectory. The experiment results demonstrate our proposed needle steering controller can perform accurate motion control of mesoscale magnetic suture needles.
Collapse
Affiliation(s)
- Matthew Fan
- Department of Mechanical Engineering, University of Maryland, College Park, and the Maryland Robotics Center, College Park, Maryland 20742 USA
| | - Xiaolong Liu
- Department of Mechanical Engineering, University of Maryland, College Park, and the Maryland Robotics Center, College Park, Maryland 20742 USA
| | - Kamakshi Jain
- Department of Mechanical Engineering, University of Maryland, College Park, and the Maryland Robotics Center, College Park, Maryland 20742 USA
| | - Daniel Lerner
- Department of Mechanical Engineering, University of Maryland, College Park, and the Maryland Robotics Center, College Park, Maryland 20742 USA
| | - Lamar O Mair
- Weinberg Medical Physics, Inc., North Bethesda, Maryland 20852 USA
| | | | - Yancy Diaz-Mercado
- Department of Mechanical Engineering, University of Maryland, College Park, and the Maryland Robotics Center, College Park, Maryland 20742 USA
| | - Axel Krieger
- Department of Mechanical Engineering, University of Maryland, College Park, and the Maryland Robotics Center, College Park, Maryland 20742 USA
| |
Collapse
|