1
|
Riaziat ND, Erin O, Krieger A, Brown JD. Investigating Haptic Feedback in Vision-Deficient Millirobot Telemanipulation. IEEE Robot Autom Lett 2024; 9:6178-6185. [PMID: 38948904 PMCID: PMC11210683 DOI: 10.1109/lra.2024.3397529] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/02/2024]
Abstract
The evolution of magnetically actuated millirobots gives rise to unique teleoperation challenges due to their non-traditional kinematic and dynamic architectures, as well as their frequent use of suboptimal imaging modalities. Recent investigations into haptic interfaces for millirobots have shown promise but lack the clinically motivated task scenarios necessary to justify future development. In this work, we investigate the utility of haptic feedback on bilateral teleoperation of a magnetically actuated millirobot in visually deficient conditions. We conducted an N=23 user study in an aneurysm coiling inspired procedure, which required participants to navigate the robot through a maze in near total darkness to manipulate beads to a target under simulated fluoroscopy. We hypothesized that users will be better able to complete the telemanipulation task with haptic feedback while reducing excess forces on their surroundings compared to the no feedback conditions. Our results showed an over 40% improvement in participants' bead scoring, a nearly 10% reduction in mean force, and 13% reduction in maximum force with haptic feedback, as well as significant improvements in other metrics. Results highlight that benefits of haptic feedback are retained when haptic feedback is removed. These findings suggest that haptic feedback has the potential to significantly improve millirobot telemanipulation and control in traditionally vision deficient tasks.
Collapse
Affiliation(s)
- Naveed D Riaziat
- Department of Mechanical Engineering, Johns Hopkins University, 3400 North Charles Street, Baltimore, Maryland, USA
| | - Onder Erin
- Department of Mechanical Engineering, Johns Hopkins University, 3400 North Charles Street, Baltimore, Maryland, USA. He is now with Johnson and Johnson, 5490 Great America Parkway, Santa Clara, California, USA
| | - Axel Krieger
- Department of Mechanical Engineering, Johns Hopkins University, 3400 North Charles Street, Baltimore, Maryland, USA
| | - Jeremy D Brown
- Department of Mechanical Engineering, Johns Hopkins University, 3400 North Charles Street, Baltimore, Maryland, USA
| |
Collapse
|
2
|
Lee J, Zhang X, Park CH, Kim MJ. Real-Time Teleoperation of Magnetic Force-Driven Microrobots With 3D Haptic Force Feedback for Micro-Navigation and Micro-Transportation. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3060708] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
3
|
Schmidt CK, Medina-Sánchez M, Edmondson RJ, Schmidt OG. Engineering microrobots for targeted cancer therapies from a medical perspective. Nat Commun 2020; 11:5618. [PMID: 33154372 PMCID: PMC7645678 DOI: 10.1038/s41467-020-19322-7] [Citation(s) in RCA: 171] [Impact Index Per Article: 42.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2020] [Accepted: 10/09/2020] [Indexed: 02/07/2023] Open
Abstract
Systemic chemotherapy remains the backbone of many cancer treatments. Due to its untargeted nature and the severe side effects it can cause, numerous nanomedicine approaches have been developed to overcome these issues. However, targeted delivery of therapeutics remains challenging. Engineering microrobots is increasingly receiving attention in this regard. Their functionalities, particularly their motility, allow microrobots to penetrate tissues and reach cancers more efficiently. Here, we highlight how different microrobots, ranging from tailor-made motile bacteria and tiny bubble-propelled microengines to hybrid spermbots, can be engineered to integrate sophisticated features optimised for precision-targeting of a wide range of cancers. Towards this, we highlight the importance of integrating clinicians, the public and cancer patients early on in the development of these novel technologies.
Collapse
Affiliation(s)
- Christine K Schmidt
- Manchester Cancer Research Centre, Division of Cancer Sciences, School of Medical Sciences, Faculty of Biology, Medicine and Health, University of Manchester, 555 Wilmslow Road, Manchester, M20 4GJ, UK.
| | - Mariana Medina-Sánchez
- Institute for Integrative Nanosciences, Leibniz IFW Dresden, Helmholtzstraße 20, 01069, Dresden, Germany.
| | - Richard J Edmondson
- Gynaecological Oncology, Division of Cancer Sciences, School of Medical Sciences, Faculty of Biology, Medicine and Health, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
- St. Mary's Hospital, Central Manchester NHS Foundation Trust, Manchester Academic Health Science Centre, Level 5, Research Floor, Oxford Road, Manchester, M13 9WL, UK
| | - Oliver G Schmidt
- Institute for Integrative Nanosciences, Leibniz IFW Dresden, Helmholtzstraße 20, 01069, Dresden, Germany.
| |
Collapse
|
4
|
Al Khatib E, Zhang X, Kim MJ, Hurmuzlu Y. Teleoperation control scheme for magnetically actuated microrobots with haptic guidance. JOURNAL OF MICRO-BIO ROBOTICS 2020. [DOI: 10.1007/s12213-020-00137-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
5
|
Scheggi S, Yoon C, Ghosh A, Gracias DH, Misra S. A GPU-Accelerated Model-Based Tracker for Untethered Submillimeter Grippers. ROBOTICS AND AUTONOMOUS SYSTEMS 2018; 103:111-121. [PMID: 31481825 PMCID: PMC6720123 DOI: 10.1016/j.robot.2017.11.003] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Miniaturized grippers that possess an untethered structure are suitable for a wide range of tasks, ranging from micromanipulation and microassembly to minimally invasive surgical interventions. In order to robustly perform such tasks, it is critical to properly estimate their overall configuration. Previous studies on tracking and control of miniaturized agents estimated mainly their 2D pixel position, mostly using cameras and optical images as a feedback modality. This paper presents a novel solution to the problem of estimating and tracking the 3D position, orientation and configuration of the tips of submillimeter grippers from marker-less visual observations. We consider this as an optimization problem, which is solved using a variant of the Particle Swarm Optimization algorithm. The proposed approach has been implemented in a Graphics Processing Unit (GPU) which allows a user to track the submillimeter agents online. The proposed approach has been evaluated on several image sequences obtained from a camera and on B-mode ultrasound images obtained from an ultrasound probe. The sequences show the grippers moving, rotating, opening/closing and grasping biological material. Qualitative results obtained using both hydrogel (soft) and metallic (hard) grippers with different shapes and sizes ranging from 750 microns to 4 mm (tip to tip), demonstrate the capability of the proposed method to track the agent in all the video sequences. Quantitative results obtained by processing synthetic data reveal a tracking position error of 25 ± 7 μm and orientation error of 1.7 ± 1.3 degrees. We believe that the proposed technique can be applied to different stimuli responsive miniaturized agents, allowing the user to estimate the full configuration of complex agents from visual marker-less observations.
Collapse
Affiliation(s)
- Stefano Scheggi
- Department of Biomechanical Engineering, University of Twente, 7522 NB, The Netherlands
| | - ChangKyu Yoon
- Department of Materials Science and Engineering, The Johns Hopkins University, MD 21218, USA
| | - Arijit Ghosh
- Department of Materials Science and Engineering, The Johns Hopkins University, MD 21218, USA
| | - David H Gracias
- Department of Materials Science and Engineering, The Johns Hopkins University, MD 21218, USA
- Department of Chemical and Biomolecular Engineering, The Johns Hopkins University, MD 21218, USA
| | - Sarthak Misra
- Department of Biomechanical Engineering, University of Twente, 7522 NB, The Netherlands
- Department of Biomedical Engineering, University of Groningen and University Medical Center Groningen, 9713 GZ, The Netherlands
| |
Collapse
|
6
|
Haptic-Based Manipulation Scheme of Magnetic Nanoparticles in a Multi-Branch Blood Vessel for Targeted Drug Delivery. MICROMACHINES 2018; 9:mi9010014. [PMID: 30393293 PMCID: PMC6187296 DOI: 10.3390/mi9010014] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/14/2017] [Revised: 12/25/2017] [Accepted: 12/28/2017] [Indexed: 12/18/2022]
Abstract
Magnetic drug targeting is a promising technique that can deliver drugs to the diseased region, while keeping the drug away from healthy parts of body. Introducing a human in the control loop of a targeted drug delivery system and using inherent bilateralism of a haptic device at the same time can considerably improve the performance of targeted drug delivery systems. In this paper, we suggest a novel intelligent haptic guidance scheme for steering a number of magnetic nanoparticles (MNPs) using forbidden region virtual fixtures and a haptic rendering scheme with multi particles. Forbidden region virtual fixtures are a general class of guidance modes implemented in software, which help a human-machine collaborative system accomplish a specific task by constraining a movement into limited regions. To examine the effectiveness of our proposed scheme, we implemented a magnetic guided drug delivery system in a virtual environment using a physics-based model of targeted drug delivery including a multi-branch blood vessel and realistic blood dynamics. We performed user studies with different guidance modes: unguided, semi virtual fixture and full virtual fixture modes. We found out that the efficiency of targeting was significantly improved using the forbidden region virtual fixture and the proposed haptic rendering of MNPs. We can expect that using intelligent haptic feedback in real targeted drug delivery systems can improve the targeting efficiency of MNPs in multi-branch vessels.
Collapse
|
7
|
Pacchierotti C, Ongaro F, van den Brink F, Yoon C, Prattichizzo D, Gracias DH, Misra S. Steering and control of miniaturized untethered soft magnetic grippers with haptic assistance. IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING : A PUBLICATION OF THE IEEE ROBOTICS AND AUTOMATION SOCIETY 2018; 15:290-306. [PMID: 31423113 PMCID: PMC6697175 DOI: 10.1109/tase.2016.2635106] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/02/2023]
Abstract
Untethered miniature robotics have recently shown promising results in several scenarios at the microscale, such as targeted drug delivery, microassembly, and biopsy procedures. However, the vast majority of these small-scale robots have very limited manipulation capabilities, and none of the steering systems currently available enable humans to intuitively and effectively control dexterous miniaturized robots in a remote environment. In this paper, we present an innovative micro teleoperation system with haptic assistance for the intuitive steering and control of miniaturized self-folding soft magnetic grippers in 2-D space. The soft grippers can be wirelessly positioned using weak magnetic fields and opened/closed by changing their temperature. An image-guided algorithm tracks the position of the controlled miniaturized gripper in the remote environment. A haptic interface provides the human operator with compelling haptic sensations about the interaction between the gripper and the environment, as well as enables the operator to intuitively control the target position and grasping configuration of the gripper. Finally, magnetic and thermal control systems regulate the position and grasping configuration of the gripper. The viability of the proposed approach is demonstrated through two experiments involving 26 human subjects. Providing haptic stimuli elicited statistically significant improvements in the performance of the considered navigation and micromanipulation tasks. Note to Practitioners-The ability to accurately and intuitively control the motion of miniaturized grippers in remote environments can open new exciting possibilities in the fields of minimally-invasive surgery, micromanipulation, biopsy, and drug delivery. This paper presents a micro teleoperation system with haptic assistance through which a clinician can easily control the motion and open/close capability of miniaturized wireless soft grippers. It introduces the underlying autonomous magnetic and thermal control systems, their interconnection with the master haptic interface, and an extensive evaluation in two real-world scenarios: following of a predetermined trajectory, and pick-and-place of a microscopic object.
Collapse
Affiliation(s)
- C. Pacchierotti
- CNRS at Irisa and Inria Rennes Bretagne Atlantique, Campus de Beaulieu, 35042 Rennes Cedex, France
| | - F. Ongaro
- Surgical Robotics Laboratory, Department of Biomechanical Engineering, MIRA–Institute for Biomedical Technology and Technical Medicine, University of Twente, 7522 NB Enschede, The Netherlands
| | - F. van den Brink
- Surgical Robotics Laboratory, Department of Biomechanical Engineering, MIRA–Institute for Biomedical Technology and Technical Medicine, University of Twente, 7522 NB Enschede, The Netherlands
| | - C. Yoon
- Department of Materials Science and Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
| | - D. Prattichizzo
- Department of Information Engineering and Mathematics, University of Siena, 53100 Siena, Italy, and also with the Department of Advanced Robotics, Istituto Italiano di Tecnologia, 16163 Genoa, Italy
| | - D. H. Gracias
- Department of Materials Science and Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
- Department of Chemical and Biomolecular Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
| | - S. Misra
- Surgical Robotics Laboratory, Department of Biomechanical Engineering, MIRA–Institute for Biomedical Technology and Technical Medicine, University of Twente, 7522 NB Enschede, The Netherlands
- Department of Biomedical Engineering, University Medical Centre Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands
| |
Collapse
|
8
|
Meli L, Pacchierotti C, Prattichizzo D. Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation. Int J Med Robot 2017; 13. [PMID: 28218455 DOI: 10.1002/rcs.1809] [Citation(s) in RCA: 48] [Impact Index Per Article: 6.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2016] [Revised: 12/13/2016] [Accepted: 12/15/2016] [Indexed: 12/12/2022]
Abstract
BACKGROUND Haptic feedback has been proven to play a key role in enhancing the performance of teleoperated medical procedures. However, due to safety issues, commercially-available medical robots do not currently provide the clinician with haptic feedback. METHODS This work presents the experimental evaluation of a teleoperation system for robot-assisted medical procedures able to provide magnified haptic feedback to the clinician. Forces registered at the operating table are magnified and provided to the clinician through a 7-DoF haptic interface. The same interface is also used to control the motion of a 6-DoF slave robotic manipulator. The safety of the system is guaranteed by a time-domain passivity-based control algorithm. RESULTS Two experiments were carried out on stiffness discrimination (during palpation and needle insertion) and one experiment on needle guidance. CONCLUSIONS Our haptic-enabled teleoperation system improved the performance with respect to direct hand interaction of 80%, 306%, and 27% in stiffness discrimination through palpation, stiffness discrimination during needle insertion, and guidance, respectively.
Collapse
Affiliation(s)
- Leonardo Meli
- Dept. of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy.,Dept. of Information Engineering and Mathematics, University of Siena, Siena, Italy
| | | | - Domenico Prattichizzo
- Dept. of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy.,Dept. of Information Engineering and Mathematics, University of Siena, Siena, Italy
| |
Collapse
|
9
|
Ueltzhöffer T, Streubel R, Koch I, Holzinger D, Makarov D, Schmidt OG, Ehresmann A. Magnetically Patterned Rolled-Up Exchange Bias Tubes: A Paternoster for Superparamagnetic Beads. ACS NANO 2016; 10:8491-8498. [PMID: 27529182 DOI: 10.1021/acsnano.6b03566] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
We realized a deterministic transport system for superparamagnetic microbeads through micrometer-sized tubes acting as channels. Beads are moved stepwise in a paternoster-like manner through the tube and back on top of it by weak magnetic field pulses without changing the field pulse polarity and taking advantage of the magnetic stray field emerging from the tubular structures. The microtubes are engineered by rolling up exchange bias layer systems, magnetically patterned into parallel stripe magnetic domains. In this way, the tubes possess distinct azimuthally aligned magnetic domain patterns. This transport mechanism features high step velocities and remote control of not only the direction and trajectory but also the velocity of the transport without the need of fuel or catalytic material. Therefore, this approach has the potential to impact several fields of 3D applications in biotechnology, including particle transport related phenomena in lab-on-a-chip and lab-in-a-tube devices.
Collapse
Affiliation(s)
- Timo Ueltzhöffer
- Institute of Physics and Center for Interdisciplinary Nanostructure Science and Technology (CINSaT), University of Kassel , Heinrich-Plett-Strasse 40, 34132 Kassel, Germany
| | - Robert Streubel
- Institute for Integrative Nanosciences, Leibniz Institute for Solid State and Materials Research Dresden (IFW Dresden) , Helmholtzstraße 20, 01069 Dresden, Germany
| | - Iris Koch
- Institute of Physics and Center for Interdisciplinary Nanostructure Science and Technology (CINSaT), University of Kassel , Heinrich-Plett-Strasse 40, 34132 Kassel, Germany
| | - Dennis Holzinger
- Institute of Physics and Center for Interdisciplinary Nanostructure Science and Technology (CINSaT), University of Kassel , Heinrich-Plett-Strasse 40, 34132 Kassel, Germany
| | - Denys Makarov
- Institute for Integrative Nanosciences, Leibniz Institute for Solid State and Materials Research Dresden (IFW Dresden) , Helmholtzstraße 20, 01069 Dresden, Germany
| | - Oliver G Schmidt
- Institute for Integrative Nanosciences, Leibniz Institute for Solid State and Materials Research Dresden (IFW Dresden) , Helmholtzstraße 20, 01069 Dresden, Germany
| | - Arno Ehresmann
- Institute of Physics and Center for Interdisciplinary Nanostructure Science and Technology (CINSaT), University of Kassel , Heinrich-Plett-Strasse 40, 34132 Kassel, Germany
| |
Collapse
|
10
|
Pacchierotti C, Scheggi S, Prattichizzo D, Misra S. Haptic Feedback for Microrobotics Applications: A Review. Front Robot AI 2016. [DOI: 10.3389/frobt.2016.00053] [Citation(s) in RCA: 23] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/01/2023] Open
|
11
|
Mura M, Abu-Kheil Y, Ciuti G, Visentini-Scarzanella M, Menciassi A, Dario P, Dias J, Seneviratne L. Vision-based haptic feedback for capsule endoscopy navigation: a proof of concept. JOURNAL OF MICRO-BIO ROBOTICS 2016. [DOI: 10.1007/s12213-016-0090-2] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2022]
|