1
|
Souipas S, Nguyen A, Laws SG, Davies BL, Rodriguez y Baena F. Real-time active constraint generation and enforcement for surgical tools using 3D detection and localisation network. Front Robot AI 2024; 11:1365632. [PMID: 38562408 PMCID: PMC10982332 DOI: 10.3389/frobt.2024.1365632] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Accepted: 02/27/2024] [Indexed: 04/04/2024] Open
Abstract
Introduction: Collaborative robots, designed to work alongside humans for manipulating end-effectors, greatly benefit from the implementation of active constraints. This process comprises the definition of a boundary, followed by the enforcement of some control algorithm when the robot tooltip interacts with the generated boundary. Contact with the constraint boundary is communicated to the human operator through various potential forms of feedback. In fields like surgical robotics, where patient safety is paramount, implementing active constraints can prevent the robot from interacting with portions of the patient anatomy that shouldn't be operated on. Despite improvements in orthopaedic surgical robots, however, there exists a gap between bulky systems with haptic feedback capabilities and miniaturised systems that only allow for boundary control, where interaction with the active constraint boundary interrupts robot functions. Generally, active constraint generation relies on optical tracking systems and preoperative imaging techniques. Methods: This paper presents a refined version of the Signature Robot, a three degrees-of-freedom, hands-on collaborative system for orthopaedic surgery. Additionally, it presents a method for generating and enforcing active constraints "on-the-fly" using our previously introduced monocular, RGB, camera-based network, SimPS-Net. The network was deployed in real-time for the purpose of boundary definition. This boundary was subsequently used for constraint enforcement testing. The robot was utilised to test two different active constraints: a safe region and a restricted region. Results: The network success rate, defined as the ratio of correct over total object localisation results, was calculated to be 54.7% ± 5.2%. In the safe region case, haptic feedback resisted tooltip manipulation beyond the active constraint boundary, with a mean distance from the boundary of 2.70 mm ± 0.37 mm and a mean exit duration of 0.76 s ± 0.11 s. For the restricted-zone constraint, the operator was successfully prevented from penetrating the boundary in 100% of attempts. Discussion: This paper showcases the viability of the proposed robotic platform and presents promising results of a versatile constraint generation and enforcement pipeline.
Collapse
Affiliation(s)
- Spyridon Souipas
- Mechatronics in Medicine, Imperial College London, Mechanical Engineering, London, United Kingdom
| | - Anh Nguyen
- Department of Computer Science, University of Liverpool, London, United Kingdom
| | - Stephen G. Laws
- Mechatronics in Medicine, Imperial College London, Mechanical Engineering, London, United Kingdom
| | - Brian L. Davies
- Mechatronics in Medicine, Imperial College London, Mechanical Engineering, London, United Kingdom
| | | |
Collapse
|
2
|
Shi Y, Zhu P, Wang T, Mai H, Yeh X, Yang L, Wang J. Dynamic Virtual Fixture Generation Based on Intra-Operative 3D Image Feedback in Robot-Assisted Minimally Invasive Thoracic Surgery. SENSORS (BASEL, SWITZERLAND) 2024; 24:492. [PMID: 38257585 PMCID: PMC10820968 DOI: 10.3390/s24020492] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/12/2023] [Revised: 01/09/2024] [Accepted: 01/10/2024] [Indexed: 01/24/2024]
Abstract
This paper proposes a method for generating dynamic virtual fixtures with real-time 3D image feedback to facilitate human-robot collaboration in medical robotics. Seamless shared control in a dynamic environment, like that of a surgical field, remains challenging despite extensive research on collaborative control and planning. To address this problem, our method dynamically creates virtual fixtures to guide the manipulation of a trocar-placing robot arm using the force field generated by point cloud data from an RGB-D camera. Additionally, the "view scope" concept selectively determines the region for computational points, thereby reducing computational load. In a phantom experiment for robot-assisted port incision in minimally invasive thoracic surgery, our method demonstrates substantially improved accuracy for port placement, reducing error and completion time by 50% (p=1.06×10-2) and 35% (p=3.23×10-2), respectively. These results suggest that our proposed approach is promising in improving surgical human-robot collaboration.
Collapse
Affiliation(s)
- Yunze Shi
- ZJU-UIUC Institute, International Campus, Zhejiang University, Haining 314400, China; (Y.S.); (T.W.); (H.M.)
- School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
| | - Peizhang Zhu
- Flexiv Ltd., Santa Clara, CA 95054, USA; (P.Z.); (X.Y.)
| | - Tengyue Wang
- ZJU-UIUC Institute, International Campus, Zhejiang University, Haining 314400, China; (Y.S.); (T.W.); (H.M.)
- School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
| | - Haonan Mai
- ZJU-UIUC Institute, International Campus, Zhejiang University, Haining 314400, China; (Y.S.); (T.W.); (H.M.)
| | - Xiyang Yeh
- Flexiv Ltd., Santa Clara, CA 95054, USA; (P.Z.); (X.Y.)
| | - Liangjing Yang
- ZJU-UIUC Institute, International Campus, Zhejiang University, Haining 314400, China; (Y.S.); (T.W.); (H.M.)
- School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
- Department of Mechanical Engineering, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA
| | - Jingfan Wang
- Flexiv Ltd., Santa Clara, CA 95054, USA; (P.Z.); (X.Y.)
| |
Collapse
|
3
|
Marques Marinho M, Oikawa R, Hayashi K, Takazawa S, Harada K, Mitsuishi M. Design and validation of looping assistance methods in robotic-assisted neonatal surgical suturing in a chest model. Int J Med Robot 2023; 19:e2476. [PMID: 36302228 DOI: 10.1002/rcs.2476] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2022] [Revised: 10/17/2022] [Accepted: 10/25/2022] [Indexed: 11/17/2022]
Abstract
BACKGROUND Neonate patients have a reduced thoracic cavity, making thoracoscopic procedures even more challenging than their adult counterparts. METHODS We evaluated five control strategies for robot-assisted thoracoscopic surgical looping in simulations and experiments with a physical robotic system in a neonate surgical phantom. The strategies are composed of state-of-the-art constrained optimization and a novel looping force feedback term. RESULTS All control strategies allowed users to successfully perform looping. A user study in simulation showed that the proposed strategy was superior in terms of Physical demand p < 0.05 $\left(p< 0.05\right)$ and task duration p < 0.05 $\left(p< 0.05\right)$ . The cumulative sum analysis of inexperienced users shows that the proposed looping force feedback can speed up the learning. Results with surgeons did not show a significant difference among control strategies. CONCLUSIONS Assistive strategies in looping show promise and further work is needed to extend these benefits to other subtasks in robot-aided surgical suturing.
Collapse
Affiliation(s)
| | - Risa Oikawa
- Department of Bioengineering, The University of Tokyo, Tokyo, Japan
| | - Kentaro Hayashi
- Department of Pediatric Surgery, the University of Tokyo Hospital, Tokyo, Japan
| | - Shinya Takazawa
- Department of Pediatric Surgery, the University of Tokyo Hospital, Tokyo, Japan
| | - Kanako Harada
- Graduate Schools of Engineering and Medicine, The University of Tokyo, Tokyo, Japan
| | - Mamoru Mitsuishi
- Department of Mechanical Engineering, University of Tokyo, Tokyo, Japan
| |
Collapse
|
4
|
Suture Looping Task Pose Planner in a Constrained Surgical Environment. J INTELL ROBOT SYST 2022. [DOI: 10.1007/s10846-022-01772-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
|
5
|
Pruks V, Ryu JH. Method for generating real-time interactive virtual fixture for shared teleoperation in unknown environments. Int J Rob Res 2022. [DOI: 10.1177/02783649221102980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
A virtual fixture (VF) is a constraint built into software that assists a human operator in moving a remote tool along a preferred path via an augmented guidance force to improve teleoperation performance. However, teleoperation generally applies to unknown or dynamic environments, which are challenging for VF use. Most researchers have assumed that VFs are pre-defined or generated automatically; however, these processes are complicated and unreliable in unknown environments where teleoperation is in high demand. Recently, a few researchers have addressed this issue by introducing a user-interactive method of generating VFs in unknown environments. However, these methods are limited to generating a single type of primitive for a single robot tool. Moreover, the accuracy of the VF generated by these methods depends on the accuracy of the human input. Thus, applications of these methods are limited. To overcome those limitations, this work introduces a novel interactive VF generation method that includes a new method of representing VFs as a composition of components. A feature-based user interface allows the human operator to intuitively specify the VF components. The new VF representation accommodates a variety of robot tools and actions. Using the feature-based interface, the process of VF generation is more intuitive and accurate. In this study, the proposed method is evaluated with human subjects in three teleoperation experiments: peg-in-hole, pipe-sawing, and pipe-welding. The experimental results show that the VFs generated by the proposed approach result in a higher manipulation quality while demonstrating the lowest total workload in all experiments. The peg-in-hole task teleoperation was the safest in terms of failure proportion and exerted force of the robot tool. In the pipe-sawing task, the positioning of the robot tool was the most accurate. In the pipe-welding task, the quality of weld was the best in terms of measured tool-trajectory smoothness and visual weld observation.
Collapse
Affiliation(s)
| | - Jee-Hwan Ryu
- Department of Civil and Environmental Engineering, KAIST, Daejeon, Republic of Korea
| |
Collapse
|
6
|
Rezende AMC, Goncalves VM, Pimenta LCA. Constructive Time-Varying Vector Fields for Robot Navigation. IEEE T ROBOT 2022. [DOI: 10.1109/tro.2021.3093674] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
7
|
Zhao L, Zhao J, Liu Z, Yang D, Liu H. Solving the Real-Time Motion Planning Problem for Non-Holonomic Robots With Collision Avoidance in Dynamic Scenes. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3194313] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
- Liangliang Zhao
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China
| | - Jingdong Zhao
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China
| | - Ziyi Liu
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China
| | - Dapeng Yang
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China
| | - Hong Liu
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China
| |
Collapse
|
8
|
Zhao L, Jiang Z, Sun Y, Zhao J, Liu H. Collision-free kinematics for hyper-redundant manipulators in dynamic scenes using optimal velocity obstacles. INT J ADV ROBOT SYST 2021. [DOI: 10.1177/1729881421996148] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Hyper-redundant manipulators have been widely used in the complex and cluttered environment for achieving various kinds of tasks. In this article, we present two contributions. First, we provide a novel algorithm of relating forward and backward reaching inverse kinematic algorithm to velocity obstacles. Our optimization-based algorithm simultaneously handles the task space constraints, the joint limit constraints, and the collision-free constraints for hyper-redundant manipulators based on the generalized framework. Second, we present an extension of our inverse kinematic algorithm to collision avoidance for the hyper-redundant manipulators, where the workspaces may have different types of obstacles. We highlight the performance of our algorithm on hyper-redundant manipulators with various degrees of freedom. The results show that our algorithm has made full use of dexterity of hyper-redundant manipulators in complex environments, enhancing the performance and increasing the flexibility.
Collapse
Affiliation(s)
- Liangliang Zhao
- School of Mechatronics Engineering, Harbin Institute of Technology, Harbin, Heilongjiang Province, China
| | - Zainan Jiang
- School of Mechatronics Engineering, Harbin Institute of Technology, Harbin, Heilongjiang Province, China
| | - Yongjun Sun
- School of Mechatronics Engineering, Harbin Institute of Technology, Harbin, Heilongjiang Province, China
| | - Jingdong Zhao
- School of Mechatronics Engineering, Harbin Institute of Technology, Harbin, Heilongjiang Province, China
| | - Hong Liu
- School of Mechatronics Engineering, Harbin Institute of Technology, Harbin, Heilongjiang Province, China
| |
Collapse
|
9
|
Cortesao R, Santos L. Noise Effects on Quaternion and Axis-Angle Representations in Robotics. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2020.3030563] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
10
|
Khadem M, O'Neill J, Mitros Z, da Cruz L, Bergeles C. Autonomous Steering of Concentric Tube Robots via Nonlinear Model Predictive Control. IEEE T ROBOT 2020. [DOI: 10.1109/tro.2020.2991651] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
11
|
The effects of different levels of realism on the training of CNNs with only synthetic images for the semantic segmentation of robotic instruments in a head phantom. Int J Comput Assist Radiol Surg 2020; 15:1257-1265. [PMID: 32445129 DOI: 10.1007/s11548-020-02185-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2019] [Accepted: 04/23/2020] [Indexed: 10/24/2022]
Abstract
PURPOSE The manual generation of training data for the semantic segmentation of medical images using deep neural networks is a time-consuming and error-prone task. In this paper, we investigate the effect of different levels of realism on the training of deep neural networks for semantic segmentation of robotic instruments. An interactive virtual-reality environment was developed to generate synthetic images for robot-aided endoscopic surgery. In contrast with earlier works, we use physically based rendering for increased realism. METHODS Using a virtual reality simulator that replicates our robotic setup, three synthetic image databases with an increasing level of realism were generated: flat, basic, and realistic (using the physically-based rendering). Each of those databases was used to train 20 instances of a UNet-based semantic-segmentation deep-learning model. The networks trained with only synthetic images were evaluated on the segmentation of 160 endoscopic images of a phantom. The networks were compared using the Dwass-Steel-Critchlow-Fligner nonparametric test. RESULTS Our results show that the levels of realism increased the mean intersection-over-union (mIoU) of the networks on endoscopic images of a phantom ([Formula: see text]). The median mIoU values were 0.235 for the flat dataset, 0.458 for the basic, and 0.729 for the realistic. All the networks trained with synthetic images outperformed naive classifiers. Moreover, in an ablation study, we show that the mIoU of physically based rendering is superior to texture mapping ([Formula: see text]) of the instrument (0.606), the background (0.685), and the background and instruments combined (0.672). CONCLUSIONS Using physical-based rendering to generate synthetic images is an effective approach to improve the training of neural networks for the semantic segmentation of surgical instruments in endoscopic images. Our results show that this strategy can be an essential step in the broad applicability of deep neural networks in semantic segmentation tasks and help bridge the domain gap in machine learning.
Collapse
|
12
|
Marinho MM, Ishida H, Harada K, Deie K, Mitsuishi M. Virtual Fixture Assistance for Suturing in Robot-Aided Pediatric Endoscopic Surgery. IEEE Robot Autom Lett 2020. [DOI: 10.1109/lra.2019.2963642] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
13
|
Moccia R, Iacono C, Siciliano B, Ficuciello F. Vision-Based Dynamic Virtual Fixtures for Tools Collision Avoidance in Robotic Surgery. IEEE Robot Autom Lett 2020. [DOI: 10.1109/lra.2020.2969941] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
14
|
Marinho MM, Harada K, Morita A, Mitsuishi M. SmartArm: Integration and validation of a versatile surgical robotic system for constrained workspaces. Int J Med Robot 2020; 16:e2053. [PMID: 31677353 DOI: 10.1002/rcs.2053] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2019] [Revised: 10/23/2019] [Accepted: 10/24/2019] [Indexed: 11/09/2022]
Abstract
BACKGROUND With the increasing presence of surgical robots minimally invasive surgery, there is a growing necessity of a versatile surgical system for deep and narrow workspaces. METHODS We developed a versatile system for constrained workspaces called SmartArm. It has two industrial-type robotic arms with flexible tools attached to its distal tip, with a total of nine active degrees-of-freedom. The system has a control algorithm based on constrained optimization that allows the safe generation of task constraints and intuitive teleoperation. RESULTS The SmartArm system is evaluated in a master-slave experiment in which a medically untrained user operates the robot to suture the dura mater membrane at the skull base of a realistic head phantom. Our results show that the user could accomplish the task proficiently, with speed and accuracy comparable to manual suturing by surgeons. Conclusions We demonstrated the integration and validation of the SmartArm.
Collapse
Affiliation(s)
| | - Kanako Harada
- Department of Mechanical Engineering, The University of Tokyo, Tokyo, Japan
| | | | - Mamoru Mitsuishi
- Department of Mechanical Engineering, The University of Tokyo, Tokyo, Japan
| |
Collapse
|